Amazon Web Services didn’t make any Big Data-related announcements at its annual customer conference last week, choosing instead to focus on a slew of developer-targeted services and a new relational database offering. But the topic of Big Data was none-the-less top of mind for many AWS enterprise customers.
This goes for direct enterprise customers — Philips Healthcare took the re:Invent keynote stage to discuss how it is using data analytics in the cloud to better diagnose and coordinate treatment in cancer patients – as well as enterprise Big Data and analytics vendors. Splunk and SAP, for example, both said they are seeing customers move their deployments from on-premises to the AWS cloud with increasing frequency.
But another interesting group of enterprise customers that is finding a home on AWS is the data service providers. These are companies whose business it is to collect, store, process and analyze data from multiple open and proprietary sources around the world (usually focused on a specific vertical market) and deliver the resulting data sets to clients in any of a number of forms. In some cases, data service providers make streams of data available via APIs to enterprise developers, who then integrate the data and insights into existing data warehouses and other analytics systems. In other cases, data service providers are developing prepackaged analytic applications that enterprise business users consume from the cloud.
The benefits of running this kind of business in a highly scalable cloud environment are numerous. The most obvious benefit is that AWS removes the IT-related burden associated with procuring, maintaining, and scaling the infrastructure required to support these data-intensive workloads. Many data service providers have built up vast data stores over the years and are adding new sources of data all the time. The cloud, generally, and AWS, specifically, allow data service providers to focus on adding value to the data in the form of industry-specific analytics and applications rather than on the “undifferentiated heavy lifting,” as AWS calls it, of maintaining and growing the underlying IT infrastructure.
One data service provider taking advantage of this benefit is the Weather Company. While most know the company for its flagship cable television station, the Weather Channel, the Weather Company has also developed a growing data services business. As part of its business-to-business data platform, the company collects, processes, and analyzes vast volumes of weather-related data from around the world and makes it available via APIs and applications to clients in the insurance, energy, and aviation industries, including 48 of the top 100 commercial airlines worldwide. In many cases, enterprise developers integrate data and analysis from the Weather Company with internal customer and financial data to help business users better understand the impact of weather on customer behavior and to determine next-best-actions to capitalize on these insights.
The Weather Company’s data API’s, which handle between 100,000 to 150,000 transactions a second, are also used by more than 100,000 application developers worldwide to serve up weather data via mobile applications on more than a billion mobile devices. Its own weather forecasting mobile applications have been downloaded more than 170 million times.
Bryson Koehler, EVP, CTO and CIO at the Weather Company, said serving as a data platform and a provider of value-add data services is “totally at the core of our business.” Speaking specifically about the decision to move its data service infrastructure to AWS, Koehler said the Weather Company needed a platform that would enable it to scale as data volumes increase and that “I didn’t want to run 13 data centers. That’s not the business I want to be in.”
The flip side benefit to offloading infrastructure procurement, support, and maintenance to AWS is that it provides data service providers increased speed and agility. Big Data is very much an iterative discipline. The job of analyzing data for competitive advantage is never “complete.” In order to maintain a leg up on the competition, enterprises need to continuously experiment with and operationalize new ways to deliver value with data, and this goes double for data service providers. Data scientists and analysts at data service providers are always interrogating new and existing data sources to discover insights that can be packaged into differentiated services for enterprise customers.
AWS lowers the barriers to spinning up new analytic environments to support this kind of data experimentation. Namely, data scientists and analysts can quickly spin-up analytics environments via AWS’s native Big Data services – Amazon Redshift for data warehousing, Amazon Elastic MapReduce for Hadoop, and Amazon Kinesis for streaming data – or bring tooling of their choice to the platform. The former option will often provide quicker time-to-value, but some data service providers have invested in developing their own analytics engines and tools. In many cases, data service providers will take a hybrid approach, bringing existing tools to the platform but adding the use of native AWS Big Data services when appropriate. This is the case at the Weather Company, which uses a number of Big Data tools, including its own analytics capabilities and Amazon Redshift to support its data services business.
Koehler points out that iterative Big Data analytics are key to maintaining the Weather Company’s competitive advantage and status as the most accurate weather forecaster on the market. Specifically, the company’s data scientists and analysts must continue to innovate and experiment with new data sources and data types, including those brought about by the emerging Internet of Things. “Amazon Web Services allows me to focus in on those innovations around the algorithmic science of understanding these [new] data sets and how they equate to a better forecast and allows me to not focus on racking and stacking servers, and running cables … so I can take my engineering talent and shift it to running great platforms and doing things that are scientifically advancing the company forward and not doing things that are commodity table-stakes.”
The Weather Company is just one of many data service providers leveraging AWS to deliver analytic insight to clients. Others include IMS Health, which has built up a 10+ petabyte proprietary database of healthcare data such as anonymized patient data, drug sales data, and medical research findings. IMS Health, which recently shifted its platform and applications to AWS, analyzes the data for insights into prescription trends, drug interactions in the real world, and marketing campaign effectiveness. It delivers the insights to clients in government and the pharmaceuticals industry through APIs and five packaged analytic and reporting applications all through the AWS cloud.
Kabir Shahani, a vice president at IMS Health who runs the company’s packaged software business, put it succinctly: “A lot of [the benefit of AWS] is being able to focus on the things that matter … AWS let’s us focus on building great software and it also lets us scale really quickly.”
Action Item: AWS is proving an ideal platform for data service providers. It allows them to offload infrastructure-related “undifferentiated heavy lifting” needed to support such data intensive workloads to AWS, leaving the data service providers more time to focus on advanced data science and analytics delivery that creates real competitive differentiation. Data service providers not leveraging AWS for underlying infrastructure should weigh the costs associated with moving existing data stores and analytics systems to the cloud with the benefits of more time to develop differentiated value for customers.
