Are You Just Digging in Your Data to Make Decisions?
Blog: Jim Sinur
It is very tempting to want to dig in your data to find opportunities to find business gold, because it really appeals to get busy to mine patterns. It’s a bit like wanting to run dirt into a sluice to find gold. The problem is that not all dirt has pay in it and hoping to accidentally find some seems a bad bet. There are a number of factors that make this a low probability approach.
The Amount of Data to Consider is Ever Increasing
Finding gold is easier than finding great decisions, because gold is usually found near water or in gold bearing land. There are limited areas known for gold. Data however is ever increasing by the second and most of what we do is store it for future refinement. While we can point to past successful sources, there is more data created than we can possibly refine. As Quantum computing and other forms of speed processing come on board, the chances will get greater for finding nuggets, but it seems like a solution looking for a problem. Its just pouring through more data.
The Speed of the Data is Ever Increasing
Having the Right Algorithm is Also a Challenge
Not only is the data coming faster than we can deal with it and the amount is over whelming, matching the data up to the proper algorithms or combinations of algorithms seems a difficult approach. It seems like trying to A/B test these algorithms alone or in combination a near impossible task. Of course, new forms of technology will help, but again this seems backwards to me. It seems too inside out of an approach and trial and error in nature.
The odds are stacked against coming up with brilliant business decisions by just digging into data even with faster computing. While we hear of examples from time to time, but finding great nuggets that way is a lot of work and does not always pan out. While I wouldn’t abandon looking at data for opportunities, especially with machine learning, deep learning within the bounds of explainable AI, but starting with great decisions and working to the data and algorithms makes more sense to me. Successful miners narrow down where the gold is likely to be for the goals they are seeking.