Premise: Big data is targeting increasingly complex business problems, including one of the biggest problems of all: How does a business optimize interactions across an entire ecosystem? Operating big data at this scale is the “grand unification theory” of big data. While still early, some software and services companies are making important strides worth studying.
Systems of intelligence (SoI), the most complex class of big data applications, anticipate, influence, and personalize human interactions using machine learning. In the absence of end-to-end horizontal packaged applications such as ERP, Wikibon is reviewing applications that can serve as archetypes for customers looking to solve similar problems. Previously we have looked at big data micro apps (BDMA) such as money laundering and big data horizontal applications (BDHA) such as cybersecurity. In this report we look at ad-tech as an example SoI application. Using ad-tech applications, combinations of advertisers, ad agencies, consumers, and publishers — the advertiser’s marketing ecosystem — collaborate in real time to personalize an optimal customer interactions with the brand. One software company pioneering video ad-tech, Altitude Digital, pulls all these different pieces together, and highlights the types of function big data ecosystem apps must support, including:
- Buyers and sellers of digital ad inventory engage directly. While standards make it possible for many middlemen to participate, publishers capture more revenue when they control their distribution channels. They pay fewer fees and get better “signals” about consumer interests.
- Buying and selling systems choose prices in milliseconds. The ad selection and pricing process works seamlessly in real-time across the ecosystem. These are not applications that operate completely in batch mode using conventional Hadoop tools.
- APIs feed the systems of record (SoR) in participating companies. Big data ecosystem need to integrate with participants’ SoR’s to manage administrative processes such as order-to-cash. Important integration investments are involved, but a digital business platform approach to managing these interfaces keeps these costs manageable.
In Ad-Tech, and Other Big Data Ecosystem Apps, Simpler Is Much Better
In a market where knowledge about consumer interest and ad performance determines pricing, buyers and sellers both have to control their digital channels or risk being at a material negotiating disadvantage. Direct connectivity greatly improves results. With software available for both buyers, which includes advertisers and ad agencies, and sellers, or publishers, which includes ad-funded apps and websites, both sides can track and collect rich information about a consumer’s profile from cookies. Publishers can collect profile information from visitor cookies and additional syndicated marketing data. With both buyers and sellers having rich visitor profile information, both sides can participate in anticipating and influencing the consumer-facing interaction. In addition, buyers’ and sellers’ operational systems coordinate across several enterprises.
Ad-tech apps depend heavily on standards and conventions in order to operate across multiple enterprises in an ecosystem. Advertisers were the first to invest heavily in automating their channels, sellers typically outsourced sales of their ads to a sell-side platform (SSP) belonging to other vendors. SSP’s relied on services that put additional middlemen between publishers and consumers. That disintermediation cost publishers fees for the middlemen that could absorb as much as 2/3 of the price of each ad (see Figure 1). Just as bad was the loss of direct access to information about site visitors. As profile data passed through the middlemen, each was able to pass along only a portion of the profile. Each middleman designed their processing pipeline to track different chunks of profile information. Altitude Digital’s software cuts out the middlemen and lets publishers directly control their own digital advertising inventory across lines of business and distribution channels (see Figure 2).
“ACT NOW” Is More Than A Barker’s Call
Publishers and advertisers have only a hundred milliseconds to agree what ad to show to a visitor while a page is loading. Anticipating and optimizing prospect interactions is a problem for big data and data feedback loops: inventory, channel, and visitors are matched based on huge volumes of diverse data and machine learning algorithms that can do much of the complex targeting that can’t be completely determined ahead of time. In addition, ad targeting improves automatically over time with feedback data on user engagement.
Influencing and optimizing the consumer interaction requires selecting the ad most engaging to the visitor with commensurate pricing. The seller uses machine learning ahead of time to classify ideal visitor profiles for each ad slot in their inventory. When invited to the auction, the advertiser matches the seller’s profile to the advertiser’s ideal profile. The quality of the match and the desirability of the seller’s profile determines the bid. The whole bidding process across demand-side platforms (DSP) and open exchanges happens in milliseconds using a standard protocol called “open real-time bidding” (OpenRTB). It is this speed and standards-based coordination across operational systems that makes ad-tech a prototype for other big data B2B applications that span an ecosystem.
Real-time integration of SoR’s across vendors in an ecosystem is necessary
The automated auction also automates the pricing process. But ecosystem apps would break down if participants couldn’t manage the operational processes beyond pricing. An eCommerce app, for example, has to manage directly or integrate with systems of record for billing and other accounting functions. Publishers have to be able to manage their full ad inventory across all digital distribution channels, allocating the premium space directly to particular advertisers and open the rest for auction.
At the highest level, the ad-tech industry is demonstrating how intelligent systems of engagement and systems of record can integrate, automate, and optimize human interactions and business processes across multi-companies in an ecosystem. For example, the RTB protocol does more than just facilitate agreement on price. It’s a protocol for integrating systems of record. RTB allows publishers to provide visibility into their premium ad inventory. It also makes it possible for advertisers to fulfill winning bids with information on how to deliver or place the ad.
Action Item: There are important lessons for Doer’s in industries far beyond advertising. If your pricing power depends on direct access to and knowledge of your customer, you need to control your digital channels. Not every company can or will build their own. But as ecosystem participants start to establish conventions on interoperability, Doer’s can start to make intelligent choices along the build vs. buy spectrum.
