Enterprise practitioners believe the potential value of Big Data is significant, but many are struggling to derive maximum value from their investments in related technology.
While a majority a Fortune 500 companies have Big Data deployments in production, and a significant percentage of mid-sized enterprises have proof-of-concept and pilot projects underway, Wikibon estimates that close to half have not realized the level of value anticipated at their onset.
In a recent survey, 46% of Big Data practitioners report that they have only realized partial value from their Big Data deployments. An unfortunate 2% declared their Big Data deployments total failures, with no value achieved.
The three compelling reasons for this struggle to achieve maximum business value form Big Data are:
– A lack of skilled Big Data practitioners.
– “Raw” and relatively immature technology.
– A lack of compelling business use case.
These findings are based on analysis of a number of sources of information, including feedback from the Wikibon community, conversations with Big Data vendors and services providers, and a first look at a Wikibon survey of close to 100 Big Data practitioners.
Based on these sources of information, two common scenarios emerge among those enterprises that have yet to achieve the full value of their Big Data investments.
Common Scenarios of Failure
The first common scenario is that enterprises invest in Big Data technology such as Hadoop without specific and measurable business applications tied to the projects. In such cases, largely driven by IT departments, enterprises begin amassing large volumes of data in Hadoop, which is sometimes made available to Data Scientists and business analysts for exploratory analysis, but that otherwise sits underutilized.
The second scenario is that enterprises successfully test Big Data pilot projects, only to find they do not have enough skilled practitioners – including administrators, developers and data scientists – to support large scale production deployments. As a result, technology performance degrades over time as stressed Hadoop and database administrators struggle to keep systems running optimally. Data Scientists and business analysts consequently are unable to perform the quantity or quality of Big Data analysis desired.
Even in cases where initial production uses are successful, subsequent projects often don’t achieve similar levels of value as the lack of skilled practitioners and/or performance issues begin to take a their toll.
Expectations vs Reality
These findings are all the more striking when considering the level of value Big Data practitioners expected to achieve. According to feedback from the Wikibon community and early results from the recent survey of Big Data practitioners, enterprises expect a return of between $3 and $4 for every $1 invested in Big Data technology over the next three to five years. The realty, however, is that most enterprises have to date achieved a return of just $.55 on the dollar. (See Figure 1)
Wikibon’s analysis indicates that enterprises that have achieved significant value from Big Data are those that address both these issues at the onset of new Big Data projects. These projects are generally not initiated by IT but driven by line-of-business departments, often marketing, and focus on small but strategic use cases. Practitioners involved in successful projects also indicate they realistically assess the level of skilled practitioners inside the enterprise and often engage outside professional services organizations to fill talent gaps and keep systems running optimally.
These findings are in line with Wikibon’s Big Data market forecast, which estimates Big Data services make up 44% of overall Big Data-related revenue. Services make up an even large percentage of revenue when considering just the Hadoop and NoSQL markets.
Action Item: Enterprises interested in leveraging Big Data to drive impactful business outcomes must be prepared to deal with the double threat of too few skilled Big Data practitioners and raw, immature technology. In order to overcome these obstacles and derive significant value from Big Data, enterprises should consider engaging professional services organizations and/or cloud services. The cost of engaging Big Data services will most often be more than offset by the increase in value achieved by successful Big Data projects. Professional services organizations are also useful at identifying initial Big Data use cases to that result in value in-and-of-themselves but also serve as proof-points for driving future projects.
