Gen AI enthusiasm has lately turned to AI skepticism. Lack of clarity on tangible ROI for mainstream businesses, a narrow list of early winners and relentless vendor marketing around AI has caused cynicism and media backlash. But the reality remains that we have entered a new era in technology innovation that has a high probability of transforming industries, public policy, company leadership and the related fortunes of individuals and organizations. Moreover, 97% of leading Gen AI adopters report that they’re achieving tangible benefits from their deployments. Adoption of Gen AI is relentlessly on the rise and nearly two years in, is poised to begin throwing off enough value that it will heighten a mandate to apply AI to drive business results. As such, we believe that as we exit Q4 into 2025, the demand for AI solutions will continue to occupy the headspace of business technology pros and AI momentum will maintain its accelerated pace.
In this Breaking Analysis, we highlight fresh data from the latest ETR drilldown on Generative AI to look at adoption rates, production use cases, ROI expectations, benefits realized and current spending patterns.
The Media Piles on the Lack of AI ROI
Barely a day goes by where you don’t see some type of negative article about AI ROI, references to Gartner’s trough of disillusionment and the like. From the Wall Street Journal to the Register to trade publications and TV programs, the narrative is decidedly downbeat when it comes to AI ROI.
But the last headline in the lower left is instructive. While AI ROI may be elusive for many organizations, enthusiasm that the bubble will continue persists.
New ETR Drilldown Survey: Gen AI Adoption and Business Impact
Today we unpack the most recent Gen AI survey from ETRs latest drilldown.
This is fresh data from nearly 1,800 ITDMS across industries and company sizes. It’s weighted toward larger spendings in North America with a nice mix of C-level executives, middle managers and practitioners.
The Number of Gen AI Holdouts is Dwindling
The first set of data we’ll look at shows that while there are still some holdouts, the vast majority of respondents (84%) are leaning into Gen AI.
Notice the steep decline in those firms that have not evaluated Gen AI. I recall in the summer of last year being struck that 1/3rd of the survey base was not evaluating Gen AI. And the reasons they gave were that it was too risky and/or moving too fast. Well that figure is down to 13% of respondents after a steep downward trajectory. A full 84% of respondents have clarity on at least one use case they’re contemplating.
This is relevant because organizations told us a year ago that they were inundated with use cases and had to prioritize and pare down the many ideas flowing in from lines of business. The point is today we’re seeing much more clarity from organizations and that bodes well for ROI, which we’ll explore later.
Firms Have Prioritized Use Cases
Let’s double click on use cases and zoom in on those dominating the landscape.
Many pundits have called 2023 the year of experimentation and that was largely true. You can see above on the leftmost bars – that dark blue bar is the July survey – and only 25% of the respondents weren’t yet in production.
So to review – from the previous chart, about 85% of customers are leaning in to Gen AI and this data shows nearly 75% have at least one use case in production.
In the red text we show which use cases are most popular. Text and data summarization at 31%, collaboration is 28%, sales and marketing content development and code generation next etc. With very few respondents saying they don’t know. Now these use cases are pretty straightforward and some of the other higher NPV use cases are in that small “other” category in the low single digits.
While these are not earth shattering use cases, they’re becoming widespread.
Practitioners are More Conservative About ROI
Let’s explore the ROI question. Does it matter that these examples are what we sometimes call “chatty,” meaning they’re ChatGPT-like? Well from an ROI perspective, no, not in our view anyway.
The chart above shows how the ROI expectations have pushed to the right – with now 21% saying the breakeven will be more than 1 year out, up from 13% in the April survey. So while here we see an indication of uncertainty with 21% (1 year+) + 23% (not sure) = 44% saying more than a year or not sure. Nonetheless, a full 56% say they saw or expect to see ROI inside of 12 months, which is encouraging, despite the fact that customers are being more cautious about ROI.
That’s the nature of these technology waves. When the surfs up like it is now, you grab the board and dive in. Because your competitors are out there stealing the best rides. As Pat Gelsinger famously said on theCUBE, if you’re not riding the waves you’ll end up as driftwood.
If you’re not riding the waves you’ll end up as driftwood…
Pat Gelsinger, CEO of VMware, 2012
Productivity Gains Dominate the Gen AI Benefit Scoreboard
Let’s keep digging into the ROI question and look at how ITDMs see the benefits of deploying Gen AI.
Ninety-eight percent of the respondents in the previous chart answered the question posed in the chart above – i.e. what benefits are you seeing specifically from Gen AI deployments? Seventy-seven percent (77%) said increased productivity and better efficiency. Thirty-nine percent (39%) cited better customers support – contact centers is a big initial use case. And you can see the other benefits customers cite like better engagement with customers, product innovation, better data analysis, headcount savings, etc. Notably, 33% said cost savings on personnel, which is a big percentage of customers and we felt it was worth highlighting in the red. We don’t know the degree of headcount savings but the fact that a large proportion of customers cites that benefit directly is instructive.
Note that only 3% of respondents said “none of the above” so clearly most customers deploying Gen AI are seeing tangible benefits.
Financial Services Firms are Even More Optimistic
Financial services is a bellwether industry. When you drill further into the data and look at a leading cohort like financial services you see even better results.
Above we show 152 respondents in FS and the percent citing productivity improvements jumps to 84% with 50% seeing improvements in customer support. Only 1% saying “none of the above.” The reason we feel this is important is that over the course of IT history, financial services has often been the leading adopter of the latest and greatest technologies. Now they were slower with the cloud but that was a matter of both trust and cloud maturity. Whereas financial firms can exercise better control in this AI cycle, whether doing so in the cloud with VPCs or working on-prem.
Interpreting AI ROI
It’s important to ground this data in reality. Remember, ROI, technically, is a ratio of benefit over cost. Breakeven period refers to the time it takes to recoup your initial investment. Net present value or NPV is the discounted cash flow of an initiative over a time period and reflects the actual dollar value of an investment. In surveys like this, these concepts are often bundled into a single vague metrics called ROI, which is the case here.
The point is when you look at an investment, it’s important to evaluate the amount of money invested as we show below. From a strict ROI definition (benefit / cost), a lower denominator will increase the ROI percentage. It will generally lower risk and often result in smaller NPVs.
ETR asked respondents to quantify their Gen AI spend and you can see above that most firms are spending less than $100K annually. About half the survey spends less than $500K. The reason this is relevant is because lowering the initial costs will naturally increase the percent ROI. But this gives no indication of the timeframe to get to cash flow positive and no indication of size of benefit.
As you can see above, about 30% spend more than $500K per annum and 23% spend north of $1M annually with nearly 10% spending more than 8 figures this year.
The point is firms are spending. We know that around 45% of customers are stealing from other budgets to fund Gen AI. In financial services, around 20% of respondents are spending more than $5M this year. The big spenders are getting really serious. For example when we cut the data on the Global 2000, we see that 26% of this cohort are spending more than $5M this year. And these big spender firms might see a smaller ROI percentage but they’ll likely see bigger NPVs. And they will set the pace for future investments. Especially as and if these investments start to throw of cash and enable reinvestment.
The other thing to remember is that Gen AI specifically and AI generally will become ubiquitous. The tech industry is embedding AI throughout the stack so as AI becomes less visible it will begin to drive new levels of productivity.
The Promise of AI as a Macro Tailwind to Productivity Growth
It’s been a while since we’ve seen a long sustained period of above average productivity growth.
Above is a chart from the Bureau of Labor Statistics. The data shows the five-year moving average of annual productivity growth over 6-7 decades. The 50s and 60s saw consumer spending become a dominant driver of productivity growth coming as the post WW II industrial economy kicked in. But for the better part of the 70’s and 80’s we saw generally below average productivity growth until the PC revolution kicked in and put a computer at everyone’s fingertips. Then after the dotcom boom and bust, we saw productivity growth bottom at the start of the financial crisis of 2008/2009; and then a rebound to just above the post WWII average.
But that wave was largely false momentum driven by the recovery versus a major technological wave. The promise of Gen AI specifically and AI overall is it will boost productivity growth by a meaningful amount. Erik Brynjolfsson said last year he’d be disappointed if we don’t see a 3-4% of improvement in productivity growth from AI, which would be noticeable on the above diagram. That’s the hope for the technology industry to address many of the world’s challenges including massive debt, climate change, poverty, overpopulation, terminal diseases and the like.
Now our industry has made many promises in the past. Some have failed to deliver but in the grand scheme of things, the technology industry has a pretty good track record of delivering societal value.
Things to Watch in AI ROI
In the near term, below we cite a few of the items we’re watching as indicators of progress in the AI ROI discussion.
Private Data. Watch for the use of private data in training and tuning LLMs and deploying Gen AI specifically. We discuss this frequently, citing theCUBE Research Gen AI Power Law. That concept refers to the long tail of use cases and applications that will leverage smaller, domain specific models. We’re starting to see a lot more discussion around smaller models, which have hit the market recently. Along with open source models. We see proprietary data becoming a new form of competitive advantage as companies leverage their own data.
RAG Adoption. In the July drilldown, only 7% of the customers ETR surveyed were deploying RAG. That was a surprisingly low figure. Now RAG maybe doesn’t set the world on fire, but it does start to set the base. A lot of the experimentation that is being done is accomplished with RAG and those systems provide learnings and will go into production in the near future to drive additional productivity.
Spending Levels on Gen AI. We’re also watching spending levels particularly watch the ROI denominator. When the denominator is small, you have big ROIs, but we’re expecting bigger denominators which will lower the percent return but make the value bigger. It also might take longer to deliver break even, but higher NPVs could serve as a beacon of the potential for Gen AI value in the future. If these initiatives throw off positive value/cash flow it will trigger gain sharing that we talked about earlier.
Use Cases Beyond ChatGPT. Let’s think about use cases beyond those that are “chatty.” Drug discovery, novel climate solutions. Fore example, maybe reducing carbon emissions is not the answer. Maybe the AI will help us figure out ways to take carbon out of the atmosphere. New energy solutions to power AI. Maybe let AI bloom and help figure out how to identify better energy solutions, cures for terminal diseases, solutions for world hunger and poverty. Also, expect new military technologies in the form of drones and other intelligent devices. Not to mention cyber security.
Agentic AI. We’re also watching the rise of agents as a productivity booster. Agents has become the hot new buzzword beyond single co-pilots. Single agents really aren’t as interesting, but when you start to apply “swarms” of agents that can learn from reasoning traces of humans and connect to data from back end systems and be guided by top-down metrics of an organization, this is going to be a signal for real productivity boosts. We’re talking here about true end-to-end automation and the emergence of what we sometimes call AI native companies that dramatically change the way in which organizations operate with new workflows that require one 10th of the people to do today’s work. And the expectation that outcomes will be achieved much faster over a dramatically compressed elapsed time.
What about it? Are you an optimist or a skeptic on AI ROI? What are you seeing in your organization and how do you think 2025 will play out?
As always, let us know.
