Big Data Adoption Progress Across Industries

Premise

The rate of Big Data analytics investment and adoption will differ between enterprises based on the perception of the potential business value it can deliver in terms of operational efficiency and competitive advantage.  The latter are a function of the critical business processes that can benefit from Big Data Analytics and these in turn point to where adoption will be relatively rapid or slow.  Industries are defined by the collection of business processes that must be delivered – so an industry comparison is a good point of departure for considering the progress of adoption of Big Data Analytics.

In this study, we found that Banking, Telco and Media, and Retail were the leaders in Big Data adoption, while Government (non-Defense), IT and Business Services, and Transportation were laggards overall.  Compared to our Spring 2014 survey, Health Care took the biggest strides forward in adoption over the past 18 months.   Manufacturing adoption tended to be average compared to the other industries.

Users should use this data to evaluate where they stand relative to their peers and act accordingly.  Vendors should be gearing product offerings and marketing effort to meet customers where they truly are in their relative Big Data readiness and maturity.

Wikibon’s Fall 2015 Big Data Survey

In the Fall of 2015, Wikibon completed research on the adoption status, attitudes, workloads, tool and deployment methods among 303 U.S. Big Data practitioners (http://wikibon.com/wikibon-big-data-analytics-survey-fall-2015/).   This report is part of Wikibon’s continuing coverage of Big Data practitioners’ attitudes, experiences, and tools use which began with our initial Big Data survey report in Spring 2014 report (http://wikibon.com/wikibon-big-data-analytics-adoption-survey-2014-2015-frequency-analysis/).

In this new report we focus on Big Data Analytics maturity and practice by industry based on our Fall 2015 survey, exploring the differences between practitioners in each sector on a variety of metrics.  This research will help both vendors and users to gauge where each industry is now in terms of its Big Data Big Data strategy and practices and where they will be heading next.

In addition, besides exploring new material in our Fall 2015 report, we also polled practitioners on a series of questions in both time periods that has enabled us to view the progress of this rapidly evolving market over the past 18 months.  Both the Spring 2014 and Fall 2015 samples were drawn from the same list sources and the same screener qualifiers were used for inclusion in the report, enabling us to fairly assess progress on a number of metrics (See section below – Big Data Progressing to Acceptance and More Aggressive Deployment).

Differences Between Industries on Key Big Data Metrics

Table 1 is a summary of the relative readiness for Big Data deployment for each industry across 10 indicators of Big Data Maturity.  These include reported success rates, attitudes, advanced usage, and deployment tool use.  We used these factors to create a Big Data Analytics Maturity Index that yields a relative ranking for where the Big Data action is moving along strongly, where it appears to be percolating, and where it is lagging.  The industries are listed in relative rank order in terms of their being above average more frequently on each indicator.

Table 1: Relative Ranking on Big Data Adoption Metrics

Industries were credited with being Above Average (blue) if their mean response was >5% of the average across the entire sample.   They were credited with being Below Average (red) if their mean response was <5% of the average across the entire sample.  Blanks in the above table indicate that the mean for that industry was within 5% of the average for the entire sample.  Actual data for each question are in the figures in the Metrics Comprising the Big Data Maturity Index section below.

The differences between industries are a function of:

• The perception of the value of Big Data analytics for the enterprise today
• Vendor and user readiness (tools, investment levels, processes) to deploy Big Data for value today
• Business process affinity for Big Data solutions for that industry
• General levels of IT maturity, skills, and risk tolerance
• Experience with analytics as a useful tool for business value

Industry Ranking Summary

More specifically, we found that Banking, Telco and Media, and Retail were the leaders in Big Data adoption, while Government (non-Defense), IT and Business Services, and Transportation were laggards overall.  Compared to our Spring 2014 survey, Health Care took the biggest strides forward in adoption over the past 18 months.   Manufacturing adoption tended to be average compared to the other industries.

• Banking and Finance – Banking and Finance is the most mature and forward thinking industry when it comes to Big Data Analytics. Fraud detection is one area of high value:high return usage, but retail banks also use Big Data for customer segmentation, churn analysis, and marketing campaign analytics with higher than average frequency. The Banking and finance industry was a leader in 6 of the 10 indicators we used in our maturity index.
• Telco & Media – Telco and Media had higher rates of Big Data in Production, but lower rates of “Success” to date. This suggests that, in the case of telco, their scale and their legacy regulatory environments may be inhibiting effective deployments because of complexity.  They tend to focus more on customer churn and recommendation engines than do other industries, as well as fraud detection and IT operations equipment management.  They are higher on investing more in Hadoop vs. traditional solutions and on Spark.  They tend to be low on Public Cloud usage – perhaps a function of scale and complexity.
• Retail/Wholesale – Customer segmentation and marketing campaign analysis show up for Retail/Wholesale enterprises at a higher rate than average. They are higher on “Success” of Big Data Production deployments, expectation that they will invest more in Hadoop than traditional technologies, and the view that Big Data is a new source of competitive advantage.  They are lower in Spark usage.   The mixed picture in this sector is a function perhaps of retail/wholesale being divided into e-commerce native businesses with a high degree of commitment to Big Data, larger retailers with omni-channel challenges, as well smaller brick and mortar operations with limited resource to invest.
• Health Care – Health Care had a very significant gain in Big Data usage between Spring 2014 and Fall 2015.  They report a relatively high rate of Big Data deployment success, paid Hadoop, investment in Hadoop vs. traditional data warehouses.  On the other hand, they are less satisfied with Hadoop than others, and also less interested in Public Cloud (HIPPA concerns) and Spark.   This is a profile of an industry in the early days of Big Data deployment as well as the smaller enterprises (relative to other industries) structurally and in our sample.  It appears that there is a willingness to deploy Big Data – especially for patient analytics and risk management – but that skills may be lacking.
• Manufacturing – Manufacturing shows up in the middle of the pack in terms of Big Data maturity. Supply Chain management is a key Big Data application for them.
• IT and Business Services – This industry is comprised of a diverse set of enterprises offering professional, business process, consulting, as well as IT-related services. They were more frequently using Big Data for IT Equipment Management and Network Analytics – so their focus is on optimizing operational performance to meet SLAs for their customers as well as controlling their own costs.   The IT skill levels here are likely to be higher than other industries, but aside from service providers focusing on Big Data, it’s likely that they are fairly average in their Big Data maturity beyond their operational considerations.  For instance, they probably do not use customer-facing applications compared to Retail, Banking, and Telco/Media, for instance.
• Transportation – Transportation companies appear to be at the early stage of Big Data investment, reporting more investment in Hadoop and Spark, but less Success and belief that Big Data is a New source of competitive advantage.   Supply Chain management is a key Big Data workload for them.  They are less inclined to Public Cloud use.  The picture is one of their being in early stages of deployment where success is yet to be realized.  Transportation is a likely candidate for high value Big Data analytics related to the Internet of Things – sensor-based analytics for vehicle tracking, capital asset preventive and predictive maintenance, alternative route analysis, weather-related analytics, etc.  Internet of Things is coming, but perhaps its maturity is insufficient to encourage large scale investments to date in large capital goods-centric industries.
• Government –Government was the least mature in Big Data deployment in our sample. Our sample included local and state governments where costs and budgets for exploring new ways to provide efficient services may be low.  It very likely excluded respondents from the Federal defense and intelligence-related arms of government where Big Data investment is very high.

Some caveats:

• Obviously each major industry category includes different sub-industries where Big Data practice may vary depending on specific analytic objectives, e.g., Banking and Finance includes retail banking, investment banking, trading, insurance, real estate, credit card infrastructure, etc. each of which would be expected to have unique business processes with different relative business value in each.
• While the collection of business processes within each industry are roughly similar and provide a good baseline, there are many potential project entry points for Big Data analytics within an industry – so each respondent’s experience may be a function of specifically what their particular enterprise decided to spend it’s early Big Data investments on.
• Enterprise size is another important factor that dictates the scale of business process and taste for risk and investment in new technologies.
• Wikibon excluded some industries from our analysis because of low sample counts.
  • Energy and Utilities – These large capital intensive industries participate in the Big Data market, but perhaps (like Transportation) they will surge in Big Data use when the Internet of Things becomes more mature and the value equation clearer.
  • Education – This is a special case with little Big Data commercial interest (e.g., K-12) and lots of free labor (at the university level) and low budgets.
  • Travel and Hospitality – There were only a handful of respondents to the Fall 2015 survey from the Travel and Hospitality industry, so we did not include them in our analysis above. It should be noted that they were a leading adopter of Big Data Analytics in our Spring 2014 survey – suggesting they would be among the leaders in 2015 as well.

Big Data Progressing to Acceptance and More Aggressive Deployment

Over the past 18 months, most industries showed increases in the proportion of their Big Data projects that had yielded at least one production deployment (Figure 1) and higher levels of satisfaction with the outcomes (Figure 2), suggesting the Big Data has evolved from the experimental stage to a more valuable component of business processes.

Figure 1:  Change in Proportion of Enterprises where there are Big Data Projects in Mission Critical Production Workloads by Industry

Telco & Media (63%) and Healthcare (56%) had the highest proportion of Big Data deployments in the Production stage (vs.  Proof of Concept or Evaluation).   Both industries showed marked gains in Production deployment vs. their Spring 2014 status.  The Transportation industry (albeit a relatively small sample) also showed a larger than average Production deployment gain in Fall 2015 vs. Spring 2014.

Banking and Finance (46%), Manufacturing (43%), and IT Technology and Business Services (38%) practitioners reported steady increases in having at least one Production deployment (vs.  Proof of Concept or Evaluation).  The Transportation data was from a very small sample in 2015 and should be ignored for the purpose of comparison.

Figure 2 shows that enterprises across all industries are experiencing higher rates of Success with their Big Data Production deployments since Spring 2014.

Figure 2 – Change in Rates of Success for Big Data Projects in Mission Critical Production Workloads by Industry

Metrics Comprising the Big Data Maturity Index

The Big Data Maturity Index was constructed from the following survey responses.

• % of respondents reporting having at least one Production Big Data deployment in use in their enterprise (Figure 1 – 2015 data) – A higher percentage indicates that practitioners within the industry are down the path with Production usage and the required investment to realize value from Big Data. This is compared to those who are still Evaluating – so they are not yet “on the mat” in any meaningful way with Big Data or those who are in a Proof of Concept still trying to establish whether there is value or not.  The specific survey question was:
  • Which of the following best describes the state of Big Data Analytics deployments in your organization? (N=294)
    • Evaluating – We are currently evaluating Big Data Analytics use cases and vendors/technology
    • Pilot – We have at least one Big Data Analytics pilot/Proof-of-Concept project underway
    • Production – We have at least one Big Data Analytics in production supporting mission critical business processes
  • Outcomes of deployed projects to date (Figure 2 above – 2015 data) – The rate of “Successful” deployments reported by a respondent suggests that practitioners have derived value and are therefore more likely to move forward with new deployments. Compared to our Spring 2014 survey (21%), the overall % of respondents indicating that their Production Big Data deployments were a “Success” rose dramatically in Spring 2015 to 48%.  It’s possible that industries where “So far so good” was a more frequent response, could be deeply committed, but are working more complex deployment problems.   This suggests that Big Data deployment overall is more likely to progress more slowly for those respondents.  The specific survey question was:
    • Which of the following statements best describes the results of your organization’s Big Data Analytics projects? (n=127 – those with at least 1 Production Big Data workload)
      • Success! We have realized the full value of our financial investment in Big Data technologies and services
      • So far so good. We have realized only partial value of our investment, but we are headed in the right direction
      • Not so good. We have realized only partial value of our investment and have hit an impasse
      • We haven’t realized any value from our investment in Big Data technologies and services
    • Big Data is a new source of competitive advantage (Figure 3) – Respondents whose attitude is that Big Data is a new source of competitive advantage are realizing the benefits and are likely to invest more aggressively to compete on a new analytics playing field than others who view Big Data more as a “complement” to traditional data warehouse analytics. The specific survey question was:
      • Which of the following best describes your attitude towards Big Data Analytics? (N=294)
        • Big Data Analytics is the new source of competitive advantage and is/will be fundamental to our business
        • Big Data Analytics is/will be an important complement to our existing Data Warehouse and Business Intelligence practice
        • Big Data Analytics is “nice to have” set of technologies/capabilities but is not a top priority
        • Big Data Analytics is a buzzword with unclear meaning or application within my enterprise

Figure 3:  Big Data Attitude by Industry

• Status of realtime operational applications (Figure 4) – The rate of actual or planned deployment of Big Data for real-time Systems of Intelligence (http://wikibon.com/systems-of-intelligence-are-driving-database-proliferation-part-1-big-fast-semi-structured/) suggests that the industry is on track for achieving maximum value of Big Data by deploying analytics for advantage in business processes in real time.  The specific survey question was:
  • Has your organization deployed an operational/transactional Big Data Analytics application(s) in production? An operational/transactional Big Data Analytics application executes near real-time operational/transactions based on insights derived from Big Data Analytics. An example is a recommendation engine that serves relevant content to end-users.
    • Yes
    • No, but we are planning to deploy an operational Big Data Analytics application within the next six months
    • No, and we do not intend to deploy an operational Big Data Analytics application in the next 6 months

Figure 4: Production Deployment of Operational/Transactional Big Data

• “Paying for Hadoop” (Figure 5) – The proportion of respondents paying for Hadoop support grew dramatically overall between Spring 2014 and Fall 2015, suggesting that enterprises were bearing down on using Big Data effectively by employing the sorts of support from vendors that enable effective use of the tools. Industries with higher levels of paid Hadoop (vs. free Hadoop) are investing more and are more likely to be further down the path of realizing significant business value than users going it alone.  The specific survey question was:
  • Regarding your organization’s Hadoop deployment, is your organization using a free/open source distribution or is your organization a paying subscription customer of one or more of the commercial Hadoop vendors? (n=179, only Hadoop users)
    • Pay – We use two or more for-pay Hadoop distributions from commercial vendors
    • Pay – We are a paying subscription customer of a commercial vendor (i.e. Cloudera, Pivotal, Hortonworks, MapR, IBM)
    • Both – We use both Apache Hadoop and subscription-supported commercial Hadoop distribution(s)
    • Free – We are using a free Hadoop distribution from a vendor (i.e. CDH, Pivotal HD, Hortonworks HDP)
    • Free – We are using open source Hadoop (i.e. Roll-your-own Apache Hadoop)

Figure 5:  Paid vs. Free Hadoop by Industry

• # of Hadoop Clusters (Figure 6) – The number of Hadoop clusters reflects the levels of maturity, commitment, and investment in Big Data tools. At the same time it is also a function of enterprise size within our sample or a structural factor within an industry.  However, this appears to only be a significant factor in our Health Care sample.  The specific survey question was:
  • How many Hadoop clusters do you have in your organization? (n=179, only Hadoop users)
    • 1
    • 2
    • >2 (Note for computational purposes, we assumed that >2 =4 on average)

Figure 6: Average # of Hadoop Clusters by Industry

• Hadoop Satisfaction (Figure 7) – High satisfaction relates to industry Big Data maturity in terms of the mastery and effective use of Hadoop as a tool for realizing business value. Of course, in some cases, some advanced users may find Hadoop less satisfactory based on their extensive use of the tool and their frustration with its shortcomings.  The specific survey question was:
  • Overall, how satisfied are you with Hadoop? (n=179, only Hadoop users)
    • Very Satisfied (equate to 5)
    • Mostly Satisfied (equate to 4)
    • Somewhat Satisfied (equate to 3)
    • Somewhat Dissatisfied (equate to 2)
    • Mostly Dissatisfied (equate to 1)
    • Very Dissatisfied (equate to 0)

Figure 7: Overall Hadoop Satisfaction by Industry

• Investing more in Hadoop vs. Traditional Analytics (Figure 8)– As opposed to viewing Hadoop as a static tool within their analytics environment, respondents whose view was that they would be investing more in Hadoop vs, Traditional technologies are further along the curve of driving their Big Data initiatives. The specific survey question was:
  • Which of the following best describes your mid-to-long term strategy (2-5 years) with Hadoop at your organization? (n=179, only Hadoop users)
    • Equal roles – Hadoop and traditional data warehousing technologies play equally important roles and we will invest in both
    • Hadoop for Less Critical – We will continue to invest in traditional enterprise data warehouse technologies for our most mission critical analytical and business intelligence workloads, while Hadoop will be used for less mission-critical analytic workloads
    • Tradtional Flat, Hadoop gets more – Our traditional data warehouse investments will be flat and Hadoop will capture the incremental investments that would have gone to the traditional data warehouse
    • Traditional decreases, Hadoop gets More – We will be decreasing our investment in traditional enterprise data warehousing technologies, and we will strongly invest in Hadoop

Figure 8: Hadoop Investment vs. Tradition Data Warehouse Plans

• Spark Status (Figure 9) – Interest in Spark, in terms of use in Production or Proofs of Concept, suggests that the enterprise is actively exploring new tools and solutions to advance the effectiveness of their Big Data deployments. The specific survey question was:
  • What statement best describes the status of your workloads involving Spark? (n=294)
    • Production Deployment – We have at least one Spark deployment in production
    • Spark Pilot – We have at least one Spark pilot/proof-of-concept project underway
    • Evaluating – We are currently evaluating Spark use cases and vendors/technology
    • Not evaluating – Not evaluating nor planning to deploy Spark at this time

Figure 9: Spark Status by Industry

• Big Data in the Public Cloud (Figure 10) – Overall, Public cloud was considered as a deployment solution more frequently in Fall 2015 than it was in Spring 2014. Enterprises that are looking at Public Cloud for Big Data are indicating an interest in and/or mastery of tools to further their Big Data agenda.  In terms of industry, solution or business process complexity, and security and regulatory concerns are an important factor in using Public Cloud for Big Data analytics.  So, we might expect regulated industries (Health Care and Banking) to be more modest users of Public Cloud.  However, there are many workloads (customer segmentation, marketing campaign analytics, social media analytics, scientific/engineering analytics, HR, et alia) where these constraints do not apply.  The specific survey question was:
  • Do you use or plan to use Public Cloud Services for Big Data? (n=294)
    • Yes, we use Hadoop in the Public Cloud
    • Yes, we use native services from Amazon (e.g. Data Pipeline, Kinesis, DynamoDB, Redshift), Google, Microsoft
    • No, but we’re planning to implement in the future
    • No

Figure 10: Use of Public Cloud for Big Data

Conclusions

As we expected, there are significant differences between industries in the maturity of their adoption of Big Data Analytics.  In this study, we found that Banking, Telco and Media, and Retail were the leaders in Big Data adoption, while Government (non-Defense), IT and Business Services, and Transportation were laggards overall.  Compared to our Spring 2014 survey, Health Care took the biggest strides forward in adoption over the past 18 months.   These differences are a function of key factors that enterprises are considering today – such as the potential for high value in high value business processes, enterprise skills and tool use, the analytics culture withni the enterprise, and risk factors such as security and regulatory compliance.

Action Item

It is still early days for Big Data Analytics.  While a greater number of enterprises have Big Data in production workloads and are spending more on tools, there is still a long way to go before Big Data becomes a mainstream technology embedded into high value business processes throughout the enterprise.

That being said, all enterprises need to be on the journey at an appropriate place on the path to obtaining optimal value from Big Data Analytics.  Users should evaluate where they stand relative to their peers and act accordingly.  Vendors should be gearing product offerings and marketing effort to meet customers where they truly are in their relative Big Data readiness and maturity.

Appendix

The industry distribution for the Fall 2015 survey is as follows: