When I began covering the Big Data market for Wikibon back in early 2011, it was early days. Big Data technologies such as Hadoop and NoSQL, while still early in their development, were becoming better known by enterprise practitioners. The market for commercial Big Data technologies and services was small but on the verge of rapid growth. But from a business use case perspective, Big Data was long on promise but short on specifics.
A lot has changed over the last four years. Without question, Big Data technologies have developed at breakneck speed due in large part to a vibrant open source community of developers. Hadoop in particular has taken great strides. What was once a batch processing, unsecure and somewhat finicky framework is now a much more comprehensive, enterprise-grade, multi-application supporting Big Data platform.
The market has likewise matured significantly since 2011, when total Big Data-related revenue stood at just over $7 billion. As of 2014, the market topped $27 billion in Big Data hardware, software and professional services revenue. The number of vendors playing in the space increased dramatically as well. Start-ups have emerged at each layer of the Big Data stack, from Hadoop distribution vendors to data transformation specialists and everything in between.
The most dramatic change, however, has been the application of Big Data to solve real-world business problems. As we predicted in Wikibon’s seminal Big Data Manifesto four long years ago, Big Data has established itself as one of the key competitive differentiators in the emerging digital economy. Companies that have embraced Big Data and put insights into action are revolutionizing entire industries – just look at Uber, AirBnB and Facebook. But the benefits of Big Data aren’t limited to emerging companies. Many tried and true enterprises such as GE, UPS and Bank of America are likewise using Big Data to stay ahead of the competition.
Over the last four years I’ve been fortunate to speak with hundreds of Big Data practitioners across a variety of vertical markets, in both large enterprises and small businesses. Many of these conversations took place on theCUBE. Below are just a handful of my favorites.
TrueCar is using Big Data to bring transparency to a notoriously translucent market – the used car market. John Williams, Senior Vice President of Technology at TrueCar, joined my co-host, SiliconANGLE’s John Furrier, and me on theCUBE at Hadoop Summit 2013. Williams discusses how TrueCar leverages Hadoop among other Big Data technologies “in order to make car buying fun and fair to everybody.”
At nearly 100 years old, Halliburton is the furthest thing from a Silicon Valley start-up. Nevertheless, the oil and gas giant is using Big Data technology to reinvent the energy business. Dr. Satyam Priyadarshy, Chief Data Scientist at Halliburton, stopped by theCUBE at #BigDataSV 2015 to discuss the ways his company uses Big Data in the energy exploration and drilling process, and why data is the key to the future of the energy and utilities markets.
Spotify relies on Big Data as much as any company in the world. Founded in 2008, the company today has over 60 million active users, all of whom produce tons of data. Spotify uses all that data to improve its service and serve up personalized music recommendations to users. Speaking on theCUBE at #BigDataSV 2014 in New York City, Wouter de Bie, Big Data Architect at Spotify, said, “From the beginning, we really knew data was going to drive this company.” de Bie talks to Furrier and Wikibon CEO Dave Vellante about building a scalable Big Data infrastructure.
It’s hard to argue with the recent success of the San Francisco Giants. The club has won three of the last five World Series. Part of its winning formula, according to the club’s CIO Bill Schlough, is Big Data. Speaking on theCUBE at #SportsDataSV 2014, Schlough discussed how “data is everywhere here at the ballpark (AT&T Park)” and how analytics plays a crucial role in the club’s success both on and off the field.
The Congressional Budget Office is practically synonymous with data. The CBO has been crunching numbers for members of the House of Representatives and the Senate since 1974. Since then, data volumes have skyrocketed while the tools used at the CBO to analyze data have matured. Jonathan Schwabish, Principal Analyst at the CBO, joins Vellante and me on theCUBE at Tableau Conference 2014 to discuss how he and his team use Big Data to keep America’s lawmakers informed.
Big Data is not just hype anymore. Real companies, from large, established firms to small, nimble start-ups – are using Big Data technology to reinvent themselves and upend established markets. Most importantly, they are delivering real value for real people. As I leave Wikibon to tackle new opportunities, I’m excited to see so many organizations taking advantage of Big Data and can’t wait to see the innovation to come. Big Data is no longer just promise, but reality.
