Contents |
Introduction
In the networking world, moving the industry from one generation of technology to the next is a complicated process. While at face value a bump in speed seems simple enough, many dependencies in technology and the broader ecosystem can take years to sort out (as discussed in this video with Emulex’s Shaun Walsh). 10Gb Ethernet (10GbE) was ratified as a standard in 2002, and nine years later only 10% of servers shipped with 10GbE. With a new generation of servers hitting the market in Q1 2012 – see HP’s Gen8,Dell’s 12G, and of course updates from IBM, Cisco, and every other x86 server OEM/ODM – we should see an acceleration of 10GbE adoption beyond blade servers, into the rack and tower environments. UPDATE: For more on the Romley/Sandy Bridge release, seeIntel Xeon Processor E5-2600.
PCIe Gen 3 and 10G LOMs
The new generation of servers will have the PCIe 3.0 bus (see Demartek’s Data Rate table), that can fully drive 10GbE environments. Many of these new servers will have FlexLOM technology, which supports the ability to change the chip, allowing for greater flexibility and choice on supplier (such as from Intel, Broadcom, Emulex, QLogic, Brocade and Mellanox). The Competitive Positioning of Network Adapters now factors into design wins, default configurations, and end-user choice.
The Cabling Conundrum
The physical layer of networking isn’t the most exciting, but it is critical for deployments. See Network Cabling for a discussion of the various cabling options for 1Gb and 10Gb Ethernet. The trade-offs between copper and optical cabling is a multi-dimensional choice. Despite the high cost of optics, fiber (optical) options are growing in popularity for high speed Ethernet. As recognition of the trend, networking market leader Cisco announced the intent to acquire optical transceiver company Lightwire.
Over the last couple of years, SFP+ copper cabling has been useful for in-rack deployments to supplement optical out of the rack. While the optical and SFP+ options have grown, there are billions of ports of UTP copper in legacy structured cabling environments. While 10GBase-T brings the promise of extending the life of copper cabling, 10GBase-T solutions currently have high power draw (this has decreased with every generation, but optical options currently draw half the power) and deployments need to be on Cat 6/6a cabling, not Cat 5.
FCoE and 10GBase-T
Fibre Channel over Ethernet (FCoE) is currently supported in both optical and SFP+ copper environments and will be supported by 10GBase-T in the future. The support of FCoE with 10GBase-T is a mixture of qualification testing and software support. In general, FCoE is not a driving force for companies to adopt 10GbE, rather the networking group deploys 10GbE first and somewhere down the line FCoE can be considered (see further discussion on FCoE adoption). Vendors that have announced 10GbE adapters with 10GBase-T have FCoE as a “future release”. While both FCoE and 10GBase-T options will grow in adoption through 2012, it is not surprising that support of FCoE+10GBase-T will sit on the back burner until later in the year. Some users are concerned that the bit error rate (BER) of 10GBase-T will be too high for FCoE. I have heard from many vendors that the BER will not be an issue in supporting the solution; the team at Panduit was nice enough to allow me to share the slide below that shows testing of a BER less than 10^-16 (that’s in the range of FC today and the definition for FC is 10^-12).
High Speed Cabling
While 10GbE is still working through adoption at the server, networking core environments are starting to look at 40Gb and 100Gb Ethernet options. The physics of cabling becomes a lot harder at these higher speeds (these challenges will also face 32Gb FC environments). Even multi-mode fiber optic cabling has limitations at these speeds. Panduit has a technology called “signature core” that it is working with the standards bodies to turn into OM5 cabling. Single mode fiber is an option to replace multi-mode but at a price premium. Speaking of pricing, the single-mode transceiver for 100GbE is currently running over $90,000! For short distances, QSFP+ copper options will be able to support configurations similar to SFP+ for 10GbE today.
Action Item: The general rule of thumb is that customers are willing to spend 4x cost for 10x performance. 10Gb Ethernet has reached the point of financial viability; it will be a while before 40 Gb and 100 Gb Ethernet reach this point. If the cabling story for high-speed networking is complicated, it is another barrier that will delay adoption. As companies look to deploy next generation servers and network, users should take into consideration the impact of high-speed cabling.
