Many of my clients and contacts are extremely enthusiastic about the potential of Big Data combined with advanced analytics, and not at all interested in my reservations about things like how to decide what data to include and who or what (algorithms that only very few understand) determines the outcome of an analysis. I agree that insight can trigger good ideas, but still think that truly great ideas, the ones that create a real advantage, need the human imagination. After all, collected data predominantly describe the past and we are far worse at predicting the future than we want to believe.
In the meantime it seems to me that the prevailing mindset of “collect anything, all you can, find correlations and thus new opportunities” is spreading like a bushfire in strong wind. ( i.e. the rapidly descending cost of data storage)
My main worry is that we’re letting new technology run away with us, instead of using it cleverly to create real improvements. The fire might over shout human vision and imagination as a source for ideas. I haven’t seen to many people being motivated or inspired by a plot of a strong correlation.
Could this focus all energy on the shorter term improvements and push long term vision out of sight?
This could lead to companies trying to steer on increasingly shorter term by analyzing more and more data, thus spiraling into increasing short term stress.
But, I might not have been looking in the right places, so let’s talk to some insiders to see where we stand in real life, before I turn too cynical.
Good news! It’s very reassuring to hear a well-respected, top CIO ensure me that the key requirement for any Big Date type initiative in his organization is end-customer benefit.
Better still, he agreed strongly that technology and human vision, creativity should work hand in hand.
What’s more: after years and years of process improvements, reorganizations and cost-cutting, this relatively new field supports further improvements. The advantages achieved that were mentioned to me are largely in the area of efficiency and effectiveness.
The cost of a marketing campaign decimated while the return is much better. Improving the service experience of the customer by continuous monitoring of his or her use of the product and then pro-actively arranging maintenance. That sort of thing.
6 Sigma and Lean had introduced data analysis as improvement tool to manufacturing and other processes, the next wave is in the area of marketing , sales and service. I worked on a green belt 6 Sigma project in the field of marketing and sales while at GE in the 1990’s, and experienced how difficult it was then to get sensible data. That’s completely changed. The down side might be for the marketeers: they were already struggling to get away from behind a screen, now the chance of meeting a real customer will be even further reduced. But that’s fine: no need to talk to customers anymore, the data tells you more about their needs than the customers know themselves…..
Concluding that there are plenty of process improvement and enhancement opportunities left, means that there seems to be no need to get panicky about data analysis setting our course completely. (For a while at least.)
However, not all worries have disappeared; in several companies I met a very limited number of specialists (with illusive descriptions like econometrists) determine how and where to search and what to look for. That’s a lot of responsibility concentrated in a small place. Especially so if the entire team consists solely of bright youngsters fresh out of university.
So, some obvious questions seem to arise; are the new searchers / analysts close enough to people with fresh ideas at multiple levels of the business to create a very fruitful cooperation?
Are we growing a new profession in the form of data analysts or will these tools soon be made usable for the people running and building their business? (In other words; will these specialist be able to make themselves redundant by developing tools so the rest of us can do the magic? Would you?)
Can big data and the subsequent analytics be a strong feeder of great vision? After all the latter is what inspires people and makes great, sustainable, companies and great products. I sincerely hope that the exiting new technical possibilities don’t make us lose sight of the role of our human vision.
In these times of increasing scarcity of resourses the enthusiasm for Big Data is understandable.
There is no shortage of data; in fact the supply is growing exponentially. Processing and storage capacity is getting cheaper as well. Great! Where else can we find this?
On top of that, and far more important, analyzing vast amounts of data can generate real benefits.
No one argues the use of consumer and system data to increase the efficiency of power grids, which operate at a shameful 20 to 40% level today.
Concluding from phone usage and location data that someone might be depressed is up for more debate. Include that person’s purchase history and you might discover he or she started painting as a new hobby. Where do you stop?
Learning from the analysis of data is a good development, as long as it benefits real people and not just corporations. My biggest concern is with the way we handle this new opportunity, specifically a golden link in the system: the data processing by algorithms that very few people understand.
Where have we seen that before? Didn’t the financial world embrace a couple of whiz kids that built algorithms to trade electronically in derivatives and other complex products?
That lead to a situation where financial institutions were (and still are) making enormous profits with product and mechanisms they don’t understand. Nobody cared as long as it went well.
Blind faith leading to disaster.
The outcome of computer analysis and processing models look very scientific and thus seems to provide some certainty in an increasingly complex world. The danger is that marketers and other business developers start to confuse the model for the real world.
Why talk to customers when you can learn from data that someone is pregnant before she knows that herself?
To tackle the bigger issues in our world we need breakthroughs in the use of resources far beyond the 2 to 3% productivity improvement that our current way of running businesses produces.
As analyzing data is essentially looking in the rear view mirror, it’s potential to create the breakthroughs we need is limited. Let’s hope we don’t waste billions again before we find out that we should use technology and human creativity plus common sense in synergy to build a better world.