Blog Posts Business Management

Cloud Data Lakes – Four Must-have TCO Optimization Capabilities

Blog: NASSCOM Official Blog

Enterprises leverage cloud providers’ compute and storage services for their ad-hoc data analytics, streaming analytics and ML use cases as cloud data lakes provide significant cost advantages, agility and scale from the get-go. Proof of concepts (POCs) for data-driven initiatives start easily and without any huge upfront bill. But over time as projects mature or ad hoc queries take longer or model iteration cycles increase, the seemingly endless supply of underlying resources leads to wasteful expenditure on compute and resources.

The usage comes with cost unpredictability and lacks financial governance and thus negatively impacts TCO. In the cloud, rising costs are not necessarily bad; it means that the data team is using more services, which theoretically means the team is doing more “good stuff” and hopefully is delivering business value. TCO optimization makes sure that wasteful spending is identified and eventually eliminated. Cloud data lake platforms should be able to help enterprises keep check on this wasteful spending to lower TCO. Admins should be able to do the following for optimizing TCO within their data lake platforms:

  1. Control and design the infrastructure spend at will, override policy, preference, or autonomous self-learning
  2. Leverage built-in capabilities to optimize clusters for lower infrastructure spend based on custom-defined parameters
  3. Monitor total costs at the application, user, account, cluster, cluster-instance level to drive accountability and meaningful discussions across teams
  4. Identify areas of cost optimization to drive maximum performance for the lowest TCO

As platforms provide these core TCO focussed capabilities, it should be autonomous and policy-based TCO optimization without sacrificing service level agreements (SLAs).

With Qubole, the open data lake platform, enterprises address all 4 key above requirements for optimizing TCO by:

  1. Reducing costs continuously in an automated manner based on set or default policy, preference, and autonomous self-learning.
  2. Optimizing the consumption of resources consistently like performance improvements to the underlying engine so that jobs are completed efficiently.
  3. Finding and consuming lower-priced resources on a continual basis with workload-aware autoscaling; admin-defined heterogeneous cluster configurations and only provision resources when needed, whether On-demand or Spot.
  4. Eliminating unnecessary resource consumption with aggressive downscaling, optimized upscaling and at-will shut down.
  5. Throttling queries based on monetary limits based on the budget set by the administrator.
  6. Providing insights for user, job and cluster level cost metrics in a multi-tenant environment to do data-driven show back discussion.

In summary, a cloud data lake platform should be able to understand what is currently happening and build a financial profile of your cloud spending, help put measures in place to control spending and optimize by taking the advantage of cloud data platform facilities to reduce costs and improve overall TCO.

P.S – This blog was first published on

The post Cloud Data Lakes – Four Must-have TCO Optimization Capabilities appeared first on NASSCOM Community |The Official Community of Indian IT Industry.

Leave a Comment

Get the BPI Web Feed

Using the HTML code below, you can display this Business Process Incubator page content with the current filter and sorting inside your web site for FREE.

Copy/Paste this code in your website html code:

<iframe src="" frameborder="0" scrolling="auto" width="100%" height="700">

Customizing your BPI Web Feed

You can click on the Get the BPI Web Feed link on any of our page to create the best possible feed for your site. Here are a few tips to customize your BPI Web Feed.

Customizing the Content Filter
On any page, you can add filter criteria using the MORE FILTERS interface:

Customizing the Content Filter

Customizing the Content Sorting
Clicking on the sorting options will also change the way your BPI Web Feed will be ordered on your site:

Get the BPI Web Feed

Some integration examples