Read Blog Post

Read Blog Post

Contributed on June 23, 2020

57

345

New TDWI Report Reveals How to Manage Your Data for Advanced Analytics

Blog: The Tibco Blog

Reading Time: 4 minutes

In a recent report, research firm Transforming Data with Intelligence (TDWI) asked a group of IT and Business Intelligence professionals to talk about the specifics of how they are making advanced analytics (AA) work at their companies. One of the first questions they asked is: 

“How important is data management to the success of your organization’s programs and applications for advanced analytics?” 

79% of the participants in the report responded that data management is Extremely Important for the success of AA. (TDWI, Data Management for Advanced Analytics, 2020, Philip Russom)

This means that one of the main, universal challenges to success in AA has been getting the right data to the right platform at the right time. 

One of the main, universal challenges to success in advanced analytics has been getting the right data to the right platform at the right time.
Click To Tweet

Why is Data Management in AA so Difficult and What’s the Best Approach to Take?

For starters, the field of AA has grown exponentially. There are a plethora of different options in use today. According to the report, statistical techniques (78%) continue to be the king of analytics today. But it’s interesting to note that predictive analytics (65%) and machine learning (53%) has arisen aggressively, as organizations seek to predict business outcomes and dynamically adjust their response based on these predictions (TDWI, Data Management for Advanced Analytics, 2020, Philip Russom). Being able to personalize offers, stop fraudsters in their tracks, and prevent equipment breakdowns before they happen are the new norm for many businesses. 

With the diverse set of AA tools out there, there is also a myriad of ways to manage the data that goes into them. The 2020 TDWI Data Management for Advanced Analytics report showed that 75% of their respondents agreed that data management requirements vary across diverse forms of advanced analytics. Given the diversity of analytic approaches, it is inevitable that data requirements are also highly diverse for analytic approaches as a whole. Tailoring data management to the needs of analytics is an opportunity to increase the value of the investment in analytics. 

Top Three Data Management Capabilities Needed for Advanced Analytics

The report found that the top three data management capabilities needed for successful advanced analytics include: 

1. Data integration 85%
2. Data warehouse 85%
3. Data quality 80%

(TDWI, Data Management for Advanced Analytics, 2020, Philip Russom)

Having a flexible data architecture that encompasses these top capabilities allows you to respond to new demands for data while leveraging existing investments. Tools like master data management, data virtualization, and metadata management can help with these challenges and meet many of the other capabilities listed on the TDWI report. 

What’s also interesting is that traditional data warehousing is found to be a great fit for reporting, but not so much for AA. What’s the difference between reporting and AA you may ask? 

Respondents in the 2020 TDWI Data Management for Advanced Analytics report agreed that reporting and analytics are different practices (79% true, 9% maybe). 

Reporting is oriented towards regularly tracking well-known entities and simple quantifications. Analytics is more of a discovery-oriented exploration of data to find previously unknown correlations or trends. Even when a single user consumes both reports and analytics, they tend to “wear different hats” as they do.

But back to why traditional data warehousing works for reporting but is not such a great fit for AA. Standard business reports depend on relational data (found in a typical relational database which most are today), while some modern forms of operational reporting (e.g., those that count website clicks, parse enterprise server logs, count entities in unstructured sources like IoT) rely more on data mining or natural language processing than the relational paradigm. It’s important to know where your company is headed so you can be prepared for any potential future needs.

Almost Half of Today’s Organizations Still Rely Exclusively on On-Premises Solutions

It’s amazing to see that half (49%) of organizations still have exclusively on-premises systems, with 28% listed as “increasingly hybrid”, and only 12% hosting their solutions in equal doses on-premises and in the cloud (TDWI, Data Management for Advanced Analytics, 2020, Philip Russom). It makes you not feel so bad if your cloud journey is stalled, right? However, they did ask respondents to say where they would be in three years, and only 8% responded still on-premises so it’s best to be on your way to the cloud now to stay competitive.

They found that users perform data management for AA with a wide range of data platforms and tools, both on-premises and cloud. These include:

1. Data warehouses (81% on-premises, 43% cloud), 
2. Data integration platforms (68% on-premises, 32% cloud), 
3. Data lakes (43% on-premises, 29% cloud)
4. Analytics tools (81% on-premises, 42% cloud)

(TDWI, Data Management for Advanced Analytics, 2020, Philip Russom)

Again, more evidence that cloud-based data, data platforms, and DM tools are established and growing. 

DM4AA – Data Management for Advanced Analytics

In the report, they asked participants “If your organization were to tailor data management specifically for advanced analytics, what would the leading barriers be?” 

1. Data Governance at 60%, modernized to cover additional data platforms and AA use cases 
2. Maintaining a single version of the truth that is current and accurate at 53%
3. Modernizing a data warehouse (originally built for reporting) to handle AA data at 48% 
4. Self-service data practices including business-friendly data semantics, easy-to-use self-service data tools (58%) and data prep for simplified integration (58%)

(TDWI, Data Management for Advanced Analytics, 2020, Philip Russom)

So, how do you scale these barriers and find the right data management solution for your particular AA use case? In the report, TDWI introduces its “Data Management for Advanced Analytics” (DM4AA) methodology to help you pair your AA solution with the right data management solution. Top-ranked by survey respondents, they agreed that the DM4AA approach has the potential to yield better operational decisions, due to better data access (68%). Respondents commented about the importance of data management to a successful AA program:

• “Good data management means good data quality that is 100% related to AA quality and success.”
• “Data management increases the value of our data assets”
• “Data management needs to provide quality foundations to reduce the risk of AA returning incomplete or inaccurate results.”  

(TDWI, Data Management for Advanced Analytics, 2020, Philip Russom)

Download this report to see how DM4AA can help your organization. There’s also a handy list at the end of the report with 12 top priorities for achieving successful DM4AA.