Micron’s SolidScale – De-concocting the Stack

The tech industry is beginning to confront the fact that we are creating new system imbalances. Advances in persistent storage technologies are outpacing CPU and server bus advances. A handful of today’s SSDs can swamp a PCIe bus. Moreover, as IOPs increase traditional host software can contribute up to 95% of the total I/O latency.

Users are used to installing SSDs internally within servers to handle performance-hungry applications. Good. But they also are applying habits developed when configuring disk-based systems, namely dedicating storage devices to performance-demanding applications. This not only wastes IOPS and storage capacity, it also is limited by the number of PCIe bus slots, thereby limiting the benefits of the highest performing — and most costly – SSDs. Storage vendors and users have to establish new habits to maximize returns from SSD technologies – vendors, in the form of product designs; users, in the form of configuration practices.

SolidScale is a collaboration among storage (Micron), network (Mellanox) and software (Excelero) vendors that takes aim at fixing this imbalance. A review of the product reveals that SolidScale has:

• The right combination of ingredients. SolidScale clearly was conceived by marrying together two perspectives: flash out and best-tech “designed in.”
• Ecosystem implications. Component companies have often been flirting with the idea of transforming into systems providers, but rarely succeed at making the jump. SolidScale suggests we’re about ready to see more of that thinking in the storage world.
• Pointers toward what is yet to come. What isn’t in SolidScale is almost as interesting as what is, for it suggests exciting new directions in system architectures.

SolidScale Has the Right Combination of Ingredients

The fundamental challenge is to network and share NVMe SSDs across multiple servers yet still provide near-local speeds. However, most storage networks were designed to work with SCSI protocols and not NVMe protocols, which lack SCSI’s reserve and lock features.

The collaboration’s goal was to add no more than 5% additional latency and increase the number of SSDs that can be shared versus internal SSDs. How did they do this? By combining leading technology from Micron, Mellanox, and Excelero to create a “modern” high-performance storage system (see Table 1):

Table 1: SolidScale Ingredients

Currently in beta testing, SolidScale leverages several nifty technologies. The first is Excelero’s recently patented NVMesh Server SAN software to pool and share SSDs. The second is Excelero’s Remote Data Drive Access (RDDA) which leverages RDMA. The third is Mellanox’s low-latency 100-Gigabit RoCE fabric. RoCE networks are the latest high-performance Ethernet interconnect technology specifically designed for memory and storage data transfer between independent server resources. Testing to date has shown that they have, in practice, achieved their target of less than 5% additional latency for any given workload (105 microseconds vs 100).

Ecosystem Implications

Micron was the main “announcer” for SolidScale, but until now it has been a component provider of DRAM, flash chips, and SSDs. That presages an interesting twist in hardware ecosystems: As systems become semiconductor memory and bandwidth bound, the locus of system value shifts away from the CPU. Companies that have the best visibility into flash, DRAM, and SSD technologies, road maps, and inventories have newfound market advantages in many hardware domains. With SolidScale, Micron becomes a systems integrator that will market, sell, and support this subsystem through Micron’s channel partners. While this is somewhat disruptive to Micron’s ecosystem, we expected this shift and see it as manageable – and replicable by others. We expect to see big players in flash and DRAM leverage their component advantages and make similar moves.

A second ecosystem implication is this: a new round of architecture wars. To date the RDMA/RoCE protocol standards lack a method to “commit” data – to ensure that data has actually arrived. Excelero has developed a unique ‘doorbell’ method to overcome this limitation. The result is that data can securely move from remote SSDs directly to an applications I/O buffer without the NVMe host’s CPU involvement eliminating many buffer copy operations and freeing the CPU to do other work (see Figure 1). This also addresses the “noisy neighbor” problem – at least on the host side. We expect to see more of these efforts by infrastructure suppliers: extending “standard” technologies and protocols in new directions as rapid technology changes reveal new requirements. What’s the challenge? Users may also, again, find themselves addressing the burdens of final integration and testing.

Figure 1: Micron SolidScale (simplified)

Source: Wikibon

What’s missing?

While there is a lot in SolidScale, there is a lot that is not. Micron claims this was by design, providing Micron the opportunity to collect customer & industry feedback on targeted workloads. Additionally, Micron claims that SolidScale is a platform architecture that will grow over time with the first instantiation targeting latency & performance oriented workloads where data services aren’t paramount. To wit, Wikibon believes that vendors are thrashing to develop the right combination of data services and hardware implementations. Over the next 5 years, our expectation is that we’ll see a lot of experimenting with “minimum viable models” for data services that can deliver the greatest business bang for the least operational buck. A look at what isn’t yet in SolidScale reveals that a few data services, in particular, are going to receive even greater scrutiny, including:

• Deduplication and compression
• Thin provisioning
• Copy services including replication and snapshots
• Quality-of-service features
• Erasure coding
• Atomic writes
• Dual ported SSDs
• NVMe-MI (management)
• An affordable backup platform
• Pricing

Action Item. With this advancement, the light of an all-flash datacenter is clearly visible and users should prepare for the inevitable, but recognize that the network will become the bottleneck followed by PCIe bus speeds. In addition, even faster technologies such as Gen-Z are on the near horizon. As storage technologies advance, system imbalances will just get worse. A new round of architectural wars is likely.