Blog Posts

Big Data Security and its Evolution!

Blog: Indium Software - Big Data

Plugging the Gaps

A research report projects a big data security market growth at 17.1 per cent Compound Annual Growth Rate (CAGR) from USD 12.22 Billion in 2017 to USD 26.85 Billion by 2022.

Some of the key drivers of this growth include the ever evolving regulatory landscape, an increasing volume of business data generated from a variety of sources, and greater threat from cyber-attacks requiring scalable high security solutions.

Big Data is typically stored in the Hadoop Distributed File System, which provides a very basic form of security, not enough to protect business interests.

The name string level of authentication is not enough to deal with the nature of breaches that cloud and networking to the World Wide Web expose one to.

Security Challenges

The challenges to big data storage, retrieval and use come at multiple levels and can be broadly classified as:

To counter these challenges, big data solutions for security can be ensured only through encryption, access control, security intelligence, data governance and data masking.

Cutting edge Big Data Engineering Services at your Finger Tips

Read More

Protecting Data

The Big Data security world is still evolving and expected to mature as newer and newer challenges emerge.

In real world systems, organisations may have their entire solution developed on Big Data, or have apps talking to legacy systems.

This will decide the complexity of the security solution being developed and the levels of security that will have to be built in.

Designing entirely for the Big Data environment is relatively easier as a compact solution can take care of all possible internal and external breaches.

But, with diligence and proper assessment, a hybrid environment too can be effectively protected.


Customers who wanted to build a platform on top of a Big Data ecosystem encounter security concerns even though the architecture is successfully implemented.

Unfortunately, most of the applications that are built on top of Big Data ecosystem components weren’t designed to address this.

This means, many applications lack encryption enablement, policy enablement to address the user level ACL, compliance and risk management to handle in case of emergencies or breach.

To ensure that their data or environment is secured, organisations will have to build those features by themselves using Big Data security components.

Access Control:

One of Indium Software’s clients was developing a mobile app and needed security solution for their Big Data.

They also had legacy systems with which the HDFS had to communicate with.

The security solution needed to provide access controls for internal and external users, as well as assign privileges.

Based on the evaluation of the security of the existing legacy systems, Indium Software developed a blue print to ensure robustness and designed the solution accordingly.

Second, it provided access control with an OpenSource tool called Kerberos to different teams based on their needs.

Through this, it was able to define privileges, thus ensuring authenticated and authorised user access of data at multiple levels.

This was based on the client list of users with privileges at the various levels, ensuring access of data to different teams based on their development goals.

Kerberos limits the assigning of privileges at the group level and not to individuals. Indium Software worked around this limitation by creating groups with single members to be able to grant appropriate authorisations.

Thirdly, it provided security for the web protocols using Knox to restrict external breaches.

This was especially important as the client had provided access to his customers, which needed to be allowed only on authentication.

Hadoop also provides auditing logs, which is being monitored and maintained as part of the SLAs.

Today, various rule based methods and different anomaly detection methods are already being used by many banks.

However, these have their own limitations and are not all that powerful. Fraud detection capabilities are enhanced with the influx of analytics and a whole new dimension to fraud detection techniques can be seen.

Along with this, performance measurement which helps standardize and maintain control for constant improvement is possible with fraud analytics.

The Evolving World

The advantage of being predominantly OpenSource is that there is a community of developers and as and when a patch is developed, it will become accessible to all.

Leverge your Biggest Asset Data

Inquire Now

Inquire Now

Leverge your Biggest Asset Data

This will take care of the limitations be it in the HDFS, Kerberos or any other security solution.

However, the threats are also expected to become just as sophisticated. Therefore, it is essential that businesses have a clear security strategy, define their goals and ensure the implementation of a good security solution to protect not just their data, but their business as well.

The post Big Data Security and its Evolution! appeared first on Indium Software.

Leave a Comment

Get the BPI Web Feed

Using the HTML code below, you can display this Business Process Incubator page content with the current filter and sorting inside your web site for FREE.

Copy/Paste this code in your website html code:

<iframe src="" frameborder="0" scrolling="auto" width="100%" height="700">

Customizing your BPI Web Feed

You can click on the Get the BPI Web Feed link on any of our page to create the best possible feed for your site. Here are a few tips to customize your BPI Web Feed.

Customizing the Content Filter
On any page, you can add filter criteria using the MORE FILTERS interface:

Customizing the Content Filter

Customizing the Content Sorting
Clicking on the sorting options will also change the way your BPI Web Feed will be ordered on your site:

Get the BPI Web Feed

Some integration examples