Monthly Archives: May 2012

As storage costs plummet, NoSQL databases mature, and processing power continues to follow Moore’s Law into the stratosphere, companies everywhere  are talking about the potential of Big Data to revolutionize business, unlocking hidden value and allowing companies to gain a competitive edge.  If you ask the vendors and consultants involved in this space, every company is sitting on a gold mine of untapped data, just waiting for a clever statistician with the right tools to dig out its value.  But is the whole big data push just hype?  An analyst at Seeking Alpha seems to think so.  First, he breaks down the six sub-hypotheses underpinning the big data impetus:

A) This data collection is a recent phenomenon B) Value is not already being extracted from whatever data is being collected C) Companies will need outside help to extract insights D) Outsiders can help companies extract insights without having deep industry knowledge E) The insights gathered from ever larger data sets have more value and are more accurate than insights gathered from smaller data sets F) Unstructured and cross functional data have huge value waiting to be extracted.

His final conclusion after examining each one?

My personal opinion is that the potential market size and value proposition are both over-hyped. I’m reminded of the hype over biotechnology in late eighties and early nineties. For example, biotechnology was supposed to have brought us juicy red, delicious and stackably cubical tomatoes that would yield square slices that fit a slice of bread. I’m still waiting for my square tomato slices.

…

In fact, when you look at the “success” stories of companies such as IBM, the ballyhooed results are almost comical (“Reduced Report Generation Time by 63%!”). Another common theme, found on all the data analytics companies websites, is that their focus areas seem to be internet, banking, healthcare, industrial, life sciences, CPG, insurance – the very areas which have traditionally crunched large amounts of data before it became “sexy”.  So where are the big inroads into new markets? One example of such an inroad is law. Law firms’ revenues have beendestroyed by revolutionary data analytics technology, which has replaced law associates with software, leading savvy clients to pay very much less for the same work.

The law firm example is especially apt for this blog, and hopefully I’ll have time to dive into the world of big data in law another time, but for now – I’m even more interested in his point about magic sandwich tomatoes from the early 90s.  The more I hear consultants touting big data as the next big thing, the more it sounds like the exact same language they’ve used to describe any sort of deep analytics over the past ten years.  Big data isn’t new, it’s a repackaging of the same old.

Consider this McKinsey report, widely cited in the industry, which may as well have been written in 1999 and updated to change gigabytes to terabytes or petabytes.  Big data to revolutionize retail?  Wal-mart, Target, and other big-box stores have been doing it for decades – that’s the whole reason grocery stores have loyalty cards.  The big data techniques for marketing have been used for decades in order to improve the effectiveness of direct mail.  In many corporate functions, like supply chain and HR, the promises of Big Data are the same as the promises that enterprise resource planning providers like Oracle and SAP have been touting for years.

So if Big Data is all hype, is there any gold there?  Underneath the surface, there’s still some opportunity.  First, the returns to rudimentary analytical skills continue to grow.  We’re reaching a point where innumeracy in the business world is becoming an even greater handicap, and the ability to analyze and interpret data will take a professional a long way.  Second, analytics is democratizing.  Many sources of big data are being made public, or available at low cost, on cloud computing servers, through services like Twitter, or by academics.  The more eyes looking at data, the greater the likelihood someone will find something revolutionary, which is why Kaggle competitions offer so much potential.    Third, the de-anonymization of the internet has the potential to allow marketers to link together a wide variety of consumer data (of course, with worrying privacy implications), which could open new opportunities.

So while I don’t think Big Data is the Next Big Thing, it’s still the continuation of an exciting trend that’s helping businesses become more effective and more efficient.   And when consultants find a new label for “crunching your numbers to help you find insights,” touting it as The Next Revolution For Business, I’ll probably use this same post again, and update my petabytes to exabytes.