Have we witnessed a Big Data bust, or is this just a temporary lull in excitement?
With the addition of “Big Data” to the Oxford English Dictionary this past June, the term has officially gone mainstream. Nearly all technology hype cycles of late, be they cloud computing, social media, or smart phones, have had periods of intense media coverage and interest. Big Data is no different. For the better part of three years, it has had a flurry of content and speculation filling the airwaves — often with minimal coherence. This lack of focus has temporarily plagued every tech hype cycle, which means Big Data is in for a rough period ahead.
Gartner’s General Hype Cycle for Technology
At this point, we have probably hit and passed the Peak of Inflated Expectations, set by such articles as “Big Data Won the US Election. Now Can It Save the World?” in Technology Review last year and the hyperbole of such headlines as Dataversity’s "How Big Data Will Save the World." In fact, the backlash is beginning to set in, as Econsultancy points out in “Big Data: the Backlash Begins” and the recent posts on PandoDaily and ZDNet.
Slipping into the Trough of Disillusionment
Where we are exactly in the Trough of Disillusionment is impossible to say, but one thing’s for sure: before we can get out of it, we need to determine what is causing this Big Data backlash. One of the primary causes is its definition, or rather, its many definitions. “Big Data” is still a term that confuses most people, particularly in the enterprise. Depending on the messenger, Big Data solutions have been described as next-generation BI visualization tools, more advanced data warehousing/lake infrastructure, cloud-based storage, social media monitoring, advanced data processing and ETL, and DIY, pre-baked, or advanced analytics. With so many definitions, pinning down true ROI is difficult (see here, here, and here). It’s no wonder that business executives we’ve met with recently have begun to express the frustrations indicative of early disillusionment in the hype cycle.
The long, hard climb up the Slope of Enlightenment
Inevitably, to address the general interest in Big Data and skepticism around ROI, a clearer, more cohesive market definition will emerge, and a subsequent separation of the contenders from the pretenders will occur. This clarity and separation will be the Slope of Enlightenment and get us on track to real productivity. The first signs of this can be seen from the efforts of the biggest IT vendors on the market:
- SAP’s HANA campaign, which delivers In-Memory storage, processing, and next-generation BI applications as a replacement to Oracle or traditional DBMS
- Oracle’s announcement of a comprehensive suite of Big Data–specialized appliances that complement existing traditional DBMS, with a specific nod to SAP’s HANA
- Workday’s recent launch of Big Data analytics in its latest release, Workday 2.0, which adds more advanced BI to its existing offerings in HCM and financial management
- IBM’s Smarter Planet campaign, and Watson in particular, demonstrating the value of natural language processing within unstructured data
- EMC Greenplum, HP Vertica, IBM Netezza, Microsoft, Teradata Aster, and Teradata Aster for data warehouse/analytics appliances round out this list of big companies' Big Data moves
Finally reaching the Plateau of Productivity
Clearly, there is interest from incumbent ERP and BI vendors to continue to control the data center through the transition to Big Data while also clarifying what Big Data truly represents. The competing models of In-Memory only vs. In-Memory as a supplement, as well as variations on SQL and R, MapReduce vs. Spark/Shark, cloud-based vs. on-premise, and the evolution of Hadoop will likely be cleared up through the battles of the colossal vendors mentioned above.
For the smaller players, adapting to the newfound skepticism of customers and sophistication of incumbents will mark a significant shift. There will certainly be a separation of the contenders and pretenders, as we’ve seen in prior hype cycles, with a fair amount of consolidation, as well as a handful of proper standalone companies created. The key to winning — and ultimately surviving until the Plateau of Productivity — will be differentiation, flexibility, and ability to deliver tangible value to the customer.
Differentiation — Several vendors are focused on the visualization of Big Data (or BI 2.0), namely the old BI vendors (IBM, Oracle, Qlikview, SAP, etc.) and newcomers such as Datameer, Palantir, and Splunk. Customers have chosen among these newer vendors largely by vertical industry, but that is likely to change as they try to expand rapidly from their roots. Ultimately, the winners in this market will have a unique enough proposition, either on interface, TCO, or service.
On the predictive/prescriptive analytics side, vendors will need to build a differentiated platform that enables rapid use case deployment and improved ROI over time. The traditional time and material model may work in the near term, but ultimately empowering the customer to rapidly build and deploy complex models will be the fashion.
Flexibility — In the early days of Big Data, early adopters were willing to leverage generic hardware and live with standalone appliances and applications that could improve productivity and drive an interesting result. Today, smaller vendors will need to have more flexible frameworks, applicable to data environments with increased variation, as the majority of enterprises want their Big Data appliances to tie neatly into their data warehouse construction and existing infrastructure. Partnerships with the largest vendors will accelerate, and enterprises will largely choose vendors that do not require complex configurations to work with their environment.
Value to customer — The early days of Big Data involved many business models of questionable value, as early adopters were drawn to new bold ideas. As the market matures and early adopters give way to the larger mainstream masses, a larger emphasis will be on solutions that actually drive top-line and bottom-line expansion. This starts from solving a specific, sizable customer need, often in domain-specific areas.
ROI will continue to loom large, particularly in the vendor-selection process, just as TCO for data center and enterprise software has been for the last few decades. In addition, emerging freemium and SaaS-like models, enabling companies to scale up their capacity as needs increase, will become more mainstream.
The Plateau of Productivity is within reach
While the market will undoubtedly go through some consolidation over the coming 18 to 24 months, there is reason to be optimistic. The Big Data phenomenon is something that is affecting and will continue to impact enterprise customers across nearly all industries. Vendors who emerge victorious from this period of disillusionment will find an enormous growth market for the next ten years or more as enterprises begin to embrace and push the envelope of Big Data analytics in improving their businesses. Ultimately, the most prosperous will stand alone as this generation’s IBM, Oracle, or Workday heir apparent.
Ben Weiss is a director in the Office of the Chairman at Opera Solutions.