As AI Moves into Production, Infrastructure Success Depends on More Than GPUs
Artificial intelligence is entering a new phase. Across nearly every industry, organizations are moving beyond experimentation and pilot projects toward production deployments that embed AI directly into business operations. Large language models are increasingly powering intelligent assistants, while agentic AI promises to automate increasingly complex workflows through systems capable of reasoning, planning, and taking action.
This evolution is fundamentally reshaping enterprise infrastructure. While much of the industry’s attention has understandably focused on advances in GPUs and accelerated computing, one of the most significant shifts occurring beneath the surface is the growing importance of networking. What was once viewed as the “plumbing” connecting servers has become a foundational component of the AI computing platform itself.
These themes formed the basis of a recent theCUBE Research executive discussion hosted by Dave Vellante, Chief Analyst at theCUBE Research, and me with Gilad Shainer, Senior Vice President of Networking at NVIDIA. The conversation explored how networking is evolving to support AI factories, why Ethernet has become increasingly important for enterprise AI, and how organizations should rethink the design of AI infrastructure for long-term success. To view the video, click below.
Networking Has Become a Strategic Business Priority
The role of networking within the enterprise has changed dramatically over the past several years. Historically, networks were evaluated based on bandwidth, availability, and connectivity. Today, they have become strategic business assets that directly influence application performance, operational agility, and increasingly, AI outcomes.
Our recent theCUBE Research Networking for AI study reflects this shift. Nearly 95 percent of organizations reported that networking is more important to achieving business objectives today than it was just two years ago. That is a remarkable change in perception and reflects how AI is reshaping infrastructure priorities. As Dave Vellante observed during our discussion, “Networking has evolved from a simple infrastructure that moves packets. It’s now an essential part of the AI computing platform.”
Gilad Shainer expanded on this transformation by explaining that networking has become a defining element of AI infrastructure itself. “The network has become a key element. The way that you connect compute engines will determine how that infrastructure, how that data center, how that AI factory is going to work.”
That statement captures perhaps the most significant architectural change occurring across enterprise IT. AI infrastructure is no longer simply about deploying faster processors. It is about enabling thousands of compute resources to function as one coordinated system.
From Server Farms to AI Factories
One of the clearest distinctions Shainer made during our conversation was between traditional server farms and modern AI factories. Conventional enterprise applications typically execute on individual servers that operate independently from one another. AI workloads are fundamentally different. Training, distributed inference, retrieval-augmented generation (RAG), and agentic AI all require thousands of GPUs, CPUs, DPUs, and storage resources to exchange information continuously while remaining tightly synchronized.
As Shainer explained: “If you have a network that cannot enable the collection of compute engines to work together, then you just have a server farm. If you want to build an AI supercomputer, you need a network that makes those compute engines work as a single unit.”
That distinction is more than semantics. The success of an AI factory depends on how efficiently distributed resources collaborate. The network is no longer simply transporting information between servers; it is coordinating the operation of an entire distributed computing environment.
This represents a fundamental change in how enterprise architects should evaluate infrastructure. Individual component specifications remain important, but system-level performance increasingly determines business outcomes.
Inference Is Harder Than Training
One of the more surprising insights from the discussion challenged a widely held assumption regarding AI workloads. Many organizations initially viewed inference as less demanding than model training. NVIDIA’s experience suggests otherwise.
According to Shainer: “People thought inferencing would be easier. Actually, it’s wrong. Training was complicated. Inferencing is much harder than that.” The reason is straightforward. Training largely involves moving massive amounts of data between GPU clusters. Inference introduces far greater diversity. Modern AI factories must coordinate GPUs, CPUs, DPUs, storage infrastructure, user requests, retrieval systems, and increasingly multiple AI agents collaborating in real time.
Agentic AI further amplifies these demands. Rather than executing isolated prompts, autonomous agents continuously exchange context, retrieve information, invoke tools, coordinate with other agents, and make decisions. These interactions generate significantly more east-west traffic and place greater emphasis on latency, synchronization, and predictable network behavior.
This shift reinforces why networking has become central to enterprise AI architectures rather than an afterthought.
Extreme Co-Design Changes the Rules
Perhaps the defining architectural philosophy discussed throughout the interview was NVIDIA’s concept of extreme co-design. Rather than designing networking, compute, storage, and software independently, NVIDIA develops these technologies as an integrated system. As Shainer explained, “The entire machine, the GPUs, the CPUs, the DPUs, the storage, needs to work as a single unit. This is where we’re describing an extreme co-design.”
Shainer explained NVIDIA’s philosophy of extreme co-design using the analogy of making a pizza. Rather than simply adding more ovens to increase output, he argued that improving performance requires redesigning the entire process. He then likened an AI workload to a pizza so large that it must be divided among multiple ovens. Every oven must bake its slice at exactly the same rate so the finished pizza comes together perfectly. If one slice is delayed, the entire pizza, and ultimately the customer experience, is compromised. The analogy effectively illustrates why networking has become so critical inside AI factories. Just as the ovens must operate in lockstep, the network must keep GPUs, CPUs, DPUs, and storage synchronized so they function as a single distributed computing platform rather than a collection of independent systems.
This philosophy extends beyond hardware integration. Within architectures such as NVLink, networking participates directly in portions of the computing process through in-network computing. Rather than simply forwarding packets, networking helps accelerate collective operations that traditionally occurred exclusively inside GPUs.
The broader lesson for enterprise organizations is significant. Evaluating AI infrastructure component by component risks overlooking one of the largest determinants of overall system performance. Increasingly, success depends on how effectively networking, compute, storage, software frameworks, cooling, and system architecture operate together as an integrated platform. This systems-level thinking is becoming increasingly important as AI factories continue growing in size and complexity.
Ethernet Continues to Evolve for AI
Ethernet has emerged as one of the industry’s most closely watched topics as organizations evaluate networking options for AI deployments.
The introduction of NVIDIA’s Spectrum-X Ethernet platform sparked considerable discussion around openness and standards, with some questioning whether AI-optimized Ethernet remains truly open.
Shainer addressed that perception directly. “Ethernet is open by definition. We’re not using any proprietary protocols. Spectrum-X runs standard Ethernet protocols, and customers can run any operating system they choose.” He also emphasized NVIDIA’s continued participation in standards organizations including RoCE with the InfiniBand Trade Association (IBTA), the Ultra Ethernet Consortium (UEC), SONiC, and the Switch Abstraction Interface (SAI).
The distinction NVIDIA makes is not about replacing Ethernet but optimizing how networking functions are implemented to address AI-specific challenges such as eliminating jitter, synchronizing distributed compute resources, and maximizing GPU utilization.
Whether every vendor agrees with that perspective remains part of the ongoing industry discussion. What is increasingly clear, however, is that Ethernet continues gaining momentum as enterprises seek scalable, standards-based networking capable of supporting AI workloads at unprecedented scale.
AI Economics Depend on Infrastructure Efficiency
As AI deployments move from experimentation into production, infrastructure conversations increasingly revolve around economics rather than specifications.
Organizations are asking new questions.
How efficiently are GPUs utilized?
What is the cost per generated token?
How much power is consumed?
How resilient is the infrastructure?
Can operations scale without dramatically increasing complexity?
Networking increasingly influences every one of those answers. As Shainer explained, “The network actually pays for itself. It dramatically impacts the number of tokens you can generate, which means it dramatically impacts the cost per token.” That observation represents an important shift in enterprise thinking.
Historically, networking was viewed primarily as a cost center. Within AI factories, networking has become a business accelerator because higher GPU utilization, lower latency, improved synchronization, and predictable performance directly translate into greater productivity and improved return on AI investments.
Ultimately, organizations are no longer optimizing infrastructure simply to move packets faster. They are optimizing for business outcomes.
Building AI Factories That Scale
As AI factories expand from thousands toward hundreds of thousands of accelerators, resiliency becomes equally important. Infrastructure failures that reduce GPU utilization or interrupt inference directly affect business operations and revenue generation.
Shainer emphasized that resiliency has become foundational to AI infrastructure design. “Resiliency is an integral part of the entire AI factory design.” In fact, Shainer highlighted several innovations intended to support that objective, including co-packaged optics, new multi-plane network topologies, improved synchronization, and architectural enhancements designed to reduce jitter while maintaining predictable performance.
Collectively, these innovations illustrate an important industry trend. Networking is evolving beyond raw bandwidth toward intelligent infrastructure capable of maintaining operational continuity even as AI workloads become increasingly distributed and dynamic. For enterprise organizations, resiliency is no longer simply a networking feature. It has become a business requirement.
Why It Matters
The organizations that derive the greatest value from AI over the next several years will likely be those that view infrastructure as an integrated platform rather than a collection of independent technologies.
Networking now sits at the center of that platform. It determines how effectively distributed compute resources collaborate, how efficiently inference executes, how resilient AI factories remain under changing conditions, and ultimately how much business value organizations generate from their AI investments.
Perhaps the strongest message from our discussion came as Shainer summarized why infrastructure design matters. “You cannot just bring some connectivity that is not fully designed with the compute. If it isn’t co-designed, then you didn’t build an AI factory, you built a data center that will consume money instead of generating it.” That observation captures the broader transition taking place across enterprise IT.
AI infrastructure is no longer simply about deploying faster processors or larger clusters. Success increasingly depends on designing systems where networking, compute, storage, software, and operations function together as a coordinated platform.
As enterprises continue scaling AI in production, networking will play an increasingly important role, not merely connecting infrastructure, but enabling the performance, resiliency, and operational efficiency required to transform AI investments into measurable business outcomes.
