Special Breaking Analysis: NVIDIA’s AI networking moat is real – But the lock-in debate continues

In a special editorial discussion hosted by Dave Vellante and Bob Laliberte, NVIDIA networking chief Gilad Shainer explains why agentic inference turns the network into part of the computer. We believe NVIDIA is materially ahead of the field, but in this special Breaking Analysis we evaluate Nvidia’s claims of openness, which must be analyzed at the system level – not merely at the Ethernet protocol layer.

Editor’s note: Performance figures attributed to NVIDIA below come from company-supplied materials. They should be treated as vendor claims unless independently validated against a disclosed workload, topology, software stack and competitive baseline.

Artificial intelligence infrastructure has reached an important architectural milestone.

In the first phase of generative AI, the industry focused primarily on training ever-larger models. The network was critical, especially as flash storage shifted the I/O bottleneck and pushed it to networking. But the network’s role was still generally described as connecting accelerators and moving data. In the emerging era of real-time inference and agentic AI, that concept is no longer valid.

The network is now a first class participant in the computational process. It synchronizes distributed processors, moves context between memory tiers, influences operations synchronously, manages congestion, isolates failures and determines how efficiently a fixed power envelope can be converted into useful tokens.

This was the central theme of our special editorial discussion with Gilad Shainer, senior vice president of networking at NVIDIA. Shainer’s contention is that without a network engineered as part of the computing system, an organization has not built an AI factory. It has assembled what he frequently referred to as a server farm.

[Listen to Gilad Shainer explain the difference between an AI Factory and a Server Farm].

We believe that distinction is strategically important, but nuanced. While NVIDIA has built the most complete AI networking architecture in the market and is materially ahead of its competitors at the system level, its advantage is not attributable to a single switch, smart NIC or protocol. It comes from the deep integration of GPUs, CPUs, DPUs, NVLink, Spectrum-X, ConnectX, BlueField, storage services, software frameworks, rack design, cooling and orchestration.

But the same extreme co-design that gives NVIDIA its advantage creates legitimate concerns about customer dependency. NVIDIA’s response is that Spectrum-X uses standard Ethernet protocols, supports RoCE and can run multiple network operating systems. The claim has substance, but it does not fully resolve the lock-in question.

Our bottom line is that NVIDIA can be open at its interfaces and is proprietary in its implementation at the same time. Those positions are not mutually exclusive.

The network has become a strategic business priority

The increased importance of networking is evident in theCUBE Research data.

In a survey of 330 respondents, 94.6% said the network had become either “more important” or “much more important” to meeting business goals than it was two years earlier. Nearly two-thirds, 65.2%, selected “much more important,” while 29.4% chose “more important.” Only 4.8% reported no change, and 0.6% said the network had become less important.

As we’ve previously written, AI is not a routine infrastructure refresh. It represents a fundamental change in how enterprises view infrastructure generally and the network specifically. The network used to be a background utility and is now becoming a determinant of application performance, AI economics and business resilience. While this was always the case in marketing messages it is increasingly an architectural reality.

The data below, based on survey research, provides an important counterweight. In a multi-select question about the biggest challenges in aligning the network with AI enablement, respondents cited:

The data above exposes the tension at the center of the market. Enterprises want the performance benefits of specialized AI infrastructure, but they do not want to create an operational island that is difficult to integrate, secure, observe and staff.

This is why the openness discussion is gaining so much attention and there’s so much buzz around AI sovereignty. It is also why deep integration, what Nvidia calls “Extreme Co-design,” has real economic value. We’ve seen the benefits of deep integration with Oracle Exadata but Nvidia has taken it to a whole new level. A highly optimized system will most typically reduce deployment complexity, accelerate time to value, while at the same time increasing switching costs over the longer term. In our experience, in the fullness of time, customers choose the risk of lock-in to get greater function and business value. Whether that trend holds with sovereign AI is an open question. Regardless, it’s highly likely Nvidia will be at the center of that discussion and almost a certainty in the case of western countries.

Why agentic inference changes the network

Shainer made a notable statement during the discussion: Training was complicated, but inference is harder.

Let’s qualify that assertion. Frontier model training remains extraordinarily demanding computationally. But production inference – particularly long-context and agentic inference – can be more heterogeneous, less predictable and more complex.

Training jobs are often bulk and synchronous in nature. Each accelerator performs a block of computation and then participates in a collective operation like “gather all” then “reduce all.” Often steps are linear and the next step cannot begin until the required participants have completed the current one.

Inference introduces a more diverse AI pipeline:

• Prompt processing, or prefill, can be compute- and memory-intensive.
• Token decoding is sequential and latency-sensitive.
• Mixture-of-experts models may generate substantial all-to-all traffic.
• Long contexts expand the key-value, or KV cache; creating efficiency challenges, especially important given the increased costs of memories.
• Agentic workflows can fan out across models, tools, databases and other agents.
• CPUs may perform scheduling, control-plane work and other functions.
• DPUs may handle isolation, security, storage and other services.
• Context may flow among GPU memory, host memory and external storage.
• Multiple users and applications require different service levels that must be managed.

As Shainer described it, the AI pod has evolved from a comparatively straightforward collection of GPU racks into a heterogeneous machine containing GPU racks, CPU racks, DPU infrastructure, context storage, an access network and connectivity across AI factories. NVIDIA’s supporting diagram below depicts scale-up, scale-out, scale-across and context-memory infrastructure as distinct but co-designed elements.

A useful way to decompose the architecture is as follows:

The point is, this underscores a much broader definition of networking than the industry has traditionally used.

Jitter is a tax on synchronization

Conventional measures of bandwidth are not enough to characterize today’s AI network challenges. Jitter becomes a key to enabling the development of AI factories vs. simply creating large server farms with GPUs and CPUs.

[Listen to Gilad Shainer explain why jitter becomes important in today’s AI networks].

For distributed workloads, the relevant question is whether every participant receives the necessary data within a tightly bounded interval. In a synchronized operation, the fastest processor does not determine completion time. It’s the slowest participant that determines the overall cycle time.

A simplified way to express effective accelerator utilization is:

Effective Utilization = Compute Time / (Compute Time + Communications Time + Wait Time).

Because network jitter increases wait time, as the number of accelerators in a cluster grows, the probability that at least one endpoint or link in the chain becomes problematic, increases substantially. A small delay can therefore leave a large number of expensive GPUs stitting idle.

This is why tail latency – not just average latency – becomes so important. Tail latency is the response time of the slowest percentage of requests (e.g. the 99th percentile). Average latency hides outliers whereas tail latency exposes worst case delays.

The point is, if congestion creates one slow path, the end-to-end process will complete at the speed of the slowest path. The economic effect is heightened because the stalled asset is an expensive GPU.

Shainer repeatedly emphasized this point. In the discussion. When processors stop operating in a synchronized fashion, the AI factory begins to behave like a server farm.

That is an intentionally provocative statement designed to de-position the competition. But it captures an important architectural reality that operators should pay attention to in that the efficacy of distributed AI infrastructure depends on coordinated utilization, not merely installed GPU count or GPU:CPU ratios.

What Spectrum-X is actually doing

NVIDIA’s core argument in defense of its approach is that conventional Ethernet implementations were optimized for other workload classes.

Enterprise Ethernet prioritizes feature richness, policy and virtualization. Hyperscale networks prioritize port density and large volumes of comparatively independent server traffic, optimized for multi-tenancy and low cost. Service-provider networks may use techniques to accommodate distance. NVIDIA’s own slide below separates these designs from Ethernet built for high-performance distributed computing.

Spectrum-X is NVIDIA’s attempt to preserve Ethernet compatibility while changing how work is divided between the switch and the endpoint.

Shainer described the design in two parts:

1. The switch distributes traffic across the available paths using awareness of surrounding network conditions.
2. The SuperNIC controls the rate at which traffic enters the network and restores packets to the required order in GPU memory.

The salient point is that the switch and the NIC are not acting as unrelated components. They are participating in a coordinated control loop designed around the behavior of GPU workloads.

Perhaps Shainer’s strongest statement from the previous clip: “This is the IP. This is the implementation of NVIDIA.”

That statement gets to the heart of the openness debate. Spectrum-X uses standard Ethernet protocols, but the algorithms, endpoint coordination and system behavior that differentiate it are proprietary NVIDIA engineering.

The NVIDIA-supplied slide below claims that the switch-to-SuperNIC architecture provides:

• 1.6 times higher effective bandwidth through improved load balancing;
• 1.3 times higher collective bandwidth through reduced tail latency;
• 2.2 times higher all-reduce bandwidth through noise isolation; and
• 1.3 times higher all-to-all bandwidth through flow rebalancing around failed links.

The above figures support NVIDIA’s architectural argument. Practitioners should dig deeper and get a better understanding of the complete baseline, network topology, workload, congestion pattern, software version and test methodology before making definitive comparisons and internal decisions.

It’s important to remember, these numbers are NVIDIA claims, not universal outcomes for all customers and workloads. Enterprise buyers should require comparable measurements on their own models and traffic patterns.

Open, proprietary or both?

[Listen to Gilad Shainer respond to the criticism that Spectrum-X is not an open architecture].

Shainer offered several concrete arguments for Spectrum-X openness:

• It uses Ethernet and standard network protocols.
• NVIDIA’s NICs can connect to other Ethernet switches.
• NVIDIA switches can connect to other Ethernet NICs.
• NVIDIA contributes to the Switch Abstraction Interface, or SAI.
• NVIDIA contributes to SONiC.
• Spectrum hardware can support multiple network operating systems.
• NVIDIA works with Meta’s FBOSS and Cisco’s NX-OS, in addition to Cumulus and SONiC.

The supporting slide below (supplied by Nvidia) explicitly presents Spectrum-X as a platform capable of supporting FBOSS, SONiC, Cumulus and NX-OS.

These are meaningful proof points of openness. In our view it would be inaccurate to characterize Spectrum-X as a closed, non-interoperable network protocol.

But it would also be inaccurate to extend that argument to the entire NVIDIA system.

We think a fair description of Spectrum-X would therefore be:

Protocol-open, implementation-differentiated and system-integrated.

That is not a criticism. It’s our intent to provide a balanced view that doesn’t simply parrot vendor marketing. We stress that meaningful technology differentiation occurs in proprietary implementations rather than in the existence of a standard packet format. Standards are critical. Incremental value comes from deep engineering investments that firms must rightly protect and charge for.

So the lock-in question arises when basic interoperability is confused with performance portability.

Performance portability means that a customer can substitute a standards-compliant component and retain not only connectivity, but also acceptable throughput, latency, resilience, observability, automation and support.

A third-party switch may successfully forward Ethernet frames. But if replacing the Spectrum-X switch causes a material decline in collective performance, or creates disputed support boundaries, the customer remains practically dependent on the integrated stack.

That dependency may ofen be the economically rational decision. Our point is that decision should be explicitly understood and a business case made around it.

NVLink is not merely a faster network

[Listen to Gilad Shainer explain NVLink in detail].

Scale-up networking has a different mission from scale-out Ethernet.

Shainer described NVLink as an extension of the GPU. Its purpose is not just to connect many accelerators, but to make multiple GPUs behave as one larger logical accelerator.

That requires much greater bandwidth, lower latency and a higher message rate than a conventional scale-out fabric. It also requires the network to participate directly in computation.

In Shainer’s example, rather than having every GPU send its partial result to every other GPU and requiring each receiver to perform the reduction, the NVLink switch can combine partial results in the network and send the reduced result back to the GPUs. Shainer referred to the use of SHARP-like in-network computing to accelerate this process.

Based on our understanding, the technical significance is data movement.

Distributed AI performance is frequently constrained not by arithmetic alone, but by how quickly data can be moved, combined and made available to the next stage. In-network reduction can reduce redundant transfers, memory traffic and synchronization overhead.

An NVIDIA-supplied NVLink slide lists the following:

• NVLink 6 at 3.6 terabytes per second;
• a 72-GPU NVL domain;
• 260 terabytes per second of aggregate all-to-all bandwidth; and
• 520 terabytes per second of aggregate all-reduce bandwidth.

Again, these are NVIDIA specifications and should be labeled as such. But the broader architectural point is that scale-up is not simply scale-out with faster ports. It is part of the construction of the accelerator itself.

NVLink Fusion broadens participation – but changes the lock-in vector

Shainer positioned NVLink Fusion as another expression of openness. The concept allows customers or partners developing custom CPUs or XPUs to connect those processors to NVIDIA’s scale-up fabric and potentially use other elements of NVIDIA’s rack, cooling, networking and storage infrastructure.

Last year, when Nvidia announced Fusion, we saw it as a recognition by Nvidia that the world is diverse and by enabling more open connectivity it expands the overall TAM. This is strategically significant and forward-thinking. It acknowledges that the future will be heterogeneous and that not every component of the system will be supplied by NVIDIA.

It also broadens NVIDIA’s addressable market. A customer can bring a non-NVIDIA accelerator while still adopting NVIDIA’s fabric and system architecture.

We view Fusion as a legitimate opening of the ecosystem, but not as evidence that NVLink has become vendor-neutral. It changes the source of dependency from an exclusively NVIDIA compute stack to an NVIDIA-centered platform in which third-party compute can participate.

That may be an attractive trade. But it remains a trade.

Context memory becomes a new infrastructure tier

One of the most consequential parts of the discussion involved context storage.

During autoregressive inference, a model retains keys and values representing prior tokens so it does not have to recompute the full attention history for each newly generated token. As context length and user concurrency increase, this KV cache can consume substantial memory, which has spiked by 400% over the past year.

At scale, context can no longer be treated solely as local GPU state. It must be placed, shared, moved, reused and deleted across a hierarchy.

NVIDIA’s STX slide describes an integrated context-memory system composed of:

• BlueField as a storage processor;
• DOCA-based storage services;
• Spectrum-X as the storage fabric; and
• Dynamo as the orchestration layer.

The stated goal is to turn Ethernet-attached flash into pod-level cache, expose a key-value interface, provide low-latency RDMA for KV-cache sharing and orchestrate inference context across storage tiers.

We believe this is an important architectural direction. It effectively inserts context memory between accelerator memory and conventional storage, creating a new tier in the inference hierarchy.

[Read our analysis of STX and the implications to the storage ecosystem from GTC 2026].

It may improve GPU utilization and enable context reuse. But it also expands the scope of the platform dependency.

Questions buyers should consider:

• Is the KV representation portable across serving systems?
• Are placement and lookup APIs documented and independently implementable?
• What happens to performance when a third-party storage platform is introduced?
  • Which third party storage platforms are best aligned with STX and why?
• How are consistency, eviction and failure semantics handled?
• Can telemetry be exported into existing operations platforms?
• Can context be migrated without rehydration or recomputation?
• Which functions require specialized hardware or softwsare (e.g. BlueField, DOCA, etc.)?

The context layer may become as strategically important as the scale-out network itself.

The economics: Measure cost per accepted token, not the price of the switch

Shainer argued that the right network pays for itself because it increases the number of tokens generated from a fixed amount of compute and power.

[Read our analysis that explains Jensen’s New Law and our take on the claim: ‘the network is free’].

We agree with that reasoning conceptually. We would not accept it as universally true without specificity around current utilization assumptions, workload, other details and more evidence of individual operator validation.

The relevant economic equation is approximately:

Cost per Accepted Token = Total Infra Cost / Accepted Tokens

Where Total Yearly Infra Cost includes annualized infrastructure and operating costs.

A quick aside on “Accepted Tokens.” Speculative decoding is a technique that meaningfully speeds up text generation in LLMs. Instead of waiting for tokens, it pairs a small, fast, less expensive model to guess on upcoming words and a large model checks them in parallel.

Accepted tokens: Makes guesses from the draft (small model) that match what the main model would have written. These save processing time.

The goal of this metric is to determine the true cost-effectiveness of the speculative process. If you achieve a high acceptance rate, you save on computational costs and get faster results.

Going back to the above equation, a TCO calculation for the numerator includes:

• annualized compute, networking and storage capital;
• power and cooling;
• software and support;
• facilities;
• operations personnel;
• downtime;
• failed or retried work; and
• underutilized capacity.

The network adds to the numerator. But in theory, a well-designed network can increase the denominator by improving accelerator utilization, reducing stall outs, recovering faster and improving overall availability. The higher the denominator, the lower accepted token costs.

This is why selecting a network solely on purchase price can be a false calculation. It is also why a premium network must prove that its incremental cost produces a larger improvement in useful output.

The appropriate comparison is not ports per dollar. It is:

NVIDIA’s greatest economic advantage may be its ability to optimize across these categories simultaneously. Its greatest commercial vulnerability is perhaps complexity in understanding the value. In other words, customers may struggle to separate the value of the system from the cost of long-term dependency on Nvidia.

Resilience, multi-plane fabrics and co-packaged optics

As AI factories grow, failure is increasingly likely. At sufficiently large scale, the probability increases that some component will be degraded, out of service or performing outside of an ideal range.

Shainer described two relevant innovations.

The first is a move from a single large multi-level fat-tree fabric toward a multi-plane architecture composed of parallel network planes. A failure within one plane can be isolated while the remaining planes continue carrying traffic.

Think of this as a form of graceful degradation. The objective is not to pretend failures will disappear, but to reduce their blast radius and preserve token generation while remediation occurs.

The second is co-packaged optics, in which optical components are brought closer to the switch silicon rather than being implemented entirely through conventional pluggable modules.

NVIDIA’s Spectrum-X Photonics slide claims:

• five times lower power;
• 10 times higher mean time between interruptions, expressed on the slide as MTBI; and
• four times fewer lasers.

Shainer said NVIDIA had a system operating internally and that Spectrum-X Photonics was moving into production. He positioned the technology as a way to improve both power efficiency and resilience.

The claims are potentially important because power and optical reliability are major scaling constraints for operators.

But remember, co-packaged optics also change the service model. Buyers should evaluate field replaceability, failure isolation, fiber attachment, repair procedures, spare-parts strategy, thermal behavior and the consequences of moving optical failures closer to the switch package.

A lower component failure rate is valuable. So is the ability to repair a failure without replacing a larger and more expensive assembly. The lifecycle tradeoff needs to be carefully assessed and tradeoffs understood before moving in this direction.

Where NVIDIA’s critics are right – and where they are not

The debate is often framed too simplistically in our view.

NVIDIA’s strongest argument in our view is that the system has to be co-designed to solve diverse workload challenges.

Its weakest argument is the tendency to use “open” as though it describes every layer of the architecture equally.

The fairest interpretation in our view is that NVIDIA opens interfaces where ecosystem participation expands the market, while protecting the implementation details that create performance differentiation. That is a rational strategy. It is also a strategy that can increase customer dependence.

Both things can be true at the same time.

Not every enterprise needs a gigascale AI factory

There is another important qualification we’re compelled to highlight.

NVIDIA’s architecture is designed around demanding distributed workloads and very large AI systems. Many enterprise AI applications will not operate at that scale.

A company running retrieval-augmented generation, department-level copilots or moderate-volume inference may not require an NVL72 domain, a specialized context-memory tier or a multi-plane fabric spanning hundreds of thousands of accelerators.

For these organizations, conventional Ethernet, cloud services or a smaller integrated appliance may deliver better economics and lower operational risk.

The fact that networking has become strategic does not mean every enterprise should reproduce hyperscale infrastructure.

Customers should size the architecture to the workload. Overbuilding creates its own form of lock-in by committing capital and skills to an operating model the business may not need.

What customers should test before committing

We believe buyers evaluating NVIDIA or any competing AI networking architecture should require five forms of evidence.

The mixed-vendor test is especially important and will test Nvidia’s claims. Our bet is the performance of an all-Nvidia stack, while not necessarily practical for all customers, will deliver substantially better results than a mixed back of heterogenous components.

Regardless, this approach converts an abstract argument about openness into a measurable number that allows operators to truly understand the tradeoffs. If replacing one component reduces performance by 2%, the dependency may be modest. If it reduces effective throughput by 30%, compromises latency or creates an unsupported configuration, the customer has quantified the practical cost of disaggregation.

In our view, that is far more useful than debating the definition of “open.”

Our conclusion

We believe NVIDIA deserves substantial credit for recognizing earlier than most of the industry that networking would become part of the AI computer. It’s acquisition of Mellanox will go down as one of the great visionary M&A moves of all time.

Nvidia’s lead spans scale-up connectivity, AI-optimized Ethernet, endpoint coordination, in-network computing, DPUs, context memory, rack-scale engineering, cooling, photonics and software orchestration. Competitors may match individual components, but NVIDIA’s advantage comes from making the components behave as a system.

That is why the company is ahead and in our view will maintain market share in enterprise AI and emerging markets like physical AI.

It is also why the lock-in criticism will persist and be a battle that Nvidia must continue to address.

NVIDIA’s openness defense is credible at the Ethernet, protocol and network operating system layers. It is less persuasive when applied to the complete AI factory. The system’s best performance derives from proprietary implementation choices and tight coordination across NVIDIA-controlled technologies.

Customers should not demand an architecture with no proprietary intellectual property. Such a system would likely surrender much of the innovation they are trying to acquire.

They should demand transparent boundaries, independently reproducible benchmarks, standards-based interfaces, exportable telemetry, data portability and a credible migration path.

The broader industry will not displace Nvidia by repeating the word “open.” It must deliver an open architecture that performs, recovers, scales and operates like a coherent system.

Until that happens, we believe NVIDIA will continue to set the pace in AI networking – while carrying the burden of proving that customers can benefit from its integration without surrendering an unacceptable degree of control.

Action Item

We believe data center operators should establish an AI-factory readiness program that treats networking, compute, storage, power, cooling and facilities as one operating system – not as separate infrastructure domains. Gilad Shainer’s core point is significant – i.e. an AI environment creates max value when GPUs, CPUs, DPUs and storage remain synchronized; otherwise, expensive accelerated infrastructure can behave like an inefficient server farm.

The objective should not be to purchase the lowest-cost network or to avoid integrated systems reflexively. It should be to determine which architecture produces the highest sustained AI output within the available power envelope, while preserving acceptable interoperability and exit options. Operators should require vendors to demonstrate these outcomes using the organization’s models, context lengths, concurrency levels and failure scenarios – not vendor-selected benchmarks alone.

Watch the full deep dive with Gilad Shainer: