Data vision?

Many of my clients and contacts are extremely enthusiastic about the potential of Big Data combined with advanced analytics, and not at all interested in my reservations about things like how to decide what data to include and who or what (algorithms that only very few understand) determines the outcome of an analysis. I agree that insight can trigger good ideas, but still think that truly great ideas, the ones that create a real advantage, need the human imagination. After all, collected data predominantly describe the past and we are far worse at predicting the future than we want to believe.

In the meantime it seems to me that the prevailing mindset of “collect anything, all you can, find correlations and thus new opportunities” is spreading like a bushfire in strong wind. ( i.e. the rapidly descending cost of data storage)

My main worry is that we’re letting new technology run away with us, instead of using it cleverly to create real improvements. The fire might over shout human vision and imagination as a source for ideas. I haven’t seen to many people being motivated or inspired by a plot of a strong correlation.

Could this focus all energy on the shorter term improvements and push long term vision out of sight?
This could lead to companies trying to steer on increasingly shorter term by analyzing more and more data, thus spiraling into increasing short term stress.

In front of me loomed a scenario of robotizing people, lost if the data doesn’t tell them what to do.

But, I might not have been looking in the right places, so let’s talk to some insiders to see where we stand in real life, before I turn too cynical.

Good news! It’s very reassuring to hear a well-respected, top CIO ensure me that the key requirement for any Big Date type initiative in his organization is end-customer benefit.
Better still, he agreed strongly that technology and human vision, creativity should work hand in hand.

What’s more: after years and years of process improvements, reorganizations and cost-cutting, this relatively new field supports further improvements. The advantages achieved that were mentioned to me are largely in the area of efficiency and effectiveness.
The cost of a marketing campaign decimated while the return is much better. Improving the service experience of the customer by continuous monitoring of his or her use of the product and then pro-actively arranging maintenance. That sort of thing.

6 Sigma and Lean had introduced data analysis as improvement tool to manufacturing and other processes, the next wave is in the area of marketing , sales and service. I worked on a green belt 6 Sigma project in the field of marketing and sales while at GE in the 1990’s, and experienced how difficult it was then to get sensible data. That’s completely changed. The down side might be for the marketeers: they were already struggling to get away from behind a screen, now the chance of meeting a real customer will be even further reduced. But that’s fine: no need to talk to customers anymore, the data tells you more about their needs than the customers know themselves…..

Concluding that there are plenty of process improvement and enhancement opportunities left, means that there seems to be no need to get panicky about data analysis setting our course completely. (For a while at least.)
However, not all worries have disappeared; in several companies I met a very limited number of specialists (with illusive descriptions like econometrists) determine how and where to search and what to look for. That’s a lot of responsibility concentrated in a small place. Especially so if the entire team consists solely of bright youngsters fresh out of university.

So, some obvious questions seem to arise; are the new searchers / analysts close enough to people with fresh ideas at multiple levels of the business to create a very fruitful cooperation?
Are we growing a new profession in the form of data analysts or will these tools soon be made usable for the people running and building their business? (In other words; will these specialist be able to make themselves redundant by developing tools so the rest of us can do the magic? Would you?)
Can big data and the subsequent analytics be a strong feeder of great vision? After all the latter is what inspires people and makes great, sustainable, companies and great products. I sincerely hope that the exiting new technical possibilities don’t make us lose sight of the role of our human vision.