Big Data, blind faith?


In these times of increasing scarcity of resourses the enthusiasm for Big Data is understandable.
There is no shortage of data; in fact the supply is growing exponentially. Processing and storage capacity is getting cheaper as well. Great! Where else can we find this?

On top of that, and far more important, analyzing vast amounts of data can generate real benefits.
No one argues the use of consumer and system data to increase the efficiency of power grids, which operate at a shameful 20 to 40% level today.
Concluding from phone usage and location data that someone might be depressed is up for more debate. Include that person’s purchase history and you might discover he or she started painting as a new hobby. Where do you stop?

Learning from the analysis of data is a good development, as long as it benefits real people and not just corporations. My biggest concern is with the way we handle this new opportunity, specifically a golden link in the system: the data processing by algorithms that very few people understand.
Where have we seen that before? Didn’t the financial world embrace a couple of whiz kids that built algorithms to trade electronically in derivatives and other complex products?
That lead to a situation where financial institutions were (and still are) making enormous profits with product and mechanisms they don’t understand. Nobody cared as long as it went well.
Blind faith leading to disaster.

The outcome of computer analysis and processing models look very scientific and thus seems to provide some certainty in an increasingly complex world. The danger is that marketers and other business developers start to confuse the model for the real world.
Why talk to customers when you can learn from data that someone is pregnant before she knows that herself?
To tackle the bigger issues in our world we need breakthroughs in the use of resources far beyond the 2 to 3% productivity improvement that our current way of running businesses produces.

As analyzing data is essentially looking in the rear view mirror, it’s potential to create the breakthroughs we need is limited. Let’s hope we don’t waste billions again before we find out that we should use technology and human creativity plus common sense in synergy to build a better world.