CTO & Co-founder, Wikibon
The origin of this research is in comments made by well established, well-meaning and talented practitioners and senior executives when faced with thinking about all-flash strategies. Typical comments include:
- “It is key that the important data is stored on more expensive, higher performing storage systems while less critical data is stored on less expensive, lower performing storage”
- “I don’t think tiering will go away anytime soon but flash changes something for sure”
These statements looks banal, and sound very reasonable. They were reasonable. However, Wikibon believes this mindset is out of date and unreasonable. This mindset will lead to wasted IT budget in the short term, and threatens the survival of any organization in the long term. The key challenge is “how can such mindsets be changed?”
The “old-school” traditional way of viewing storage is to put mission critical systems on Tier 1 expensive tiered storage (e.g., flash + magnetic disk), giving great performance and availability. Copies of that data are moved to other applications, e.g., a copy to load into the data warehouse, another copy to put onto a web-front-end, etc. These copies are taken because of the bandwidth and IOPS limitations of magnetic disk storage.
Functionality such as space-efficient snapshots have been available for a long time on traditional storage arrays, and allow the same data to be shared across different applications. The main use of these snapshots is the ability to revert to a previous copy of the data should there be any hardware or software corruption or failure. These snapshots cannot be used for sharing data between applications because the IO bandwidth and IOPS performance limitations of traditional magnetic disks. The result of trying to share magnetic data between transactional and data warehouse applications would result in melted disk heads.
The result of this constraint in magnetic disk performance is that tiering is used to migrate data, and that there are 10 to 15 copies of the same data in a typical data center.
Enter capacity flash. It costs more, so it can and should be de-duplicated and compressed. Even more important, it has so much IO performance it can and must be shared. The same physical storage can be shared by all of the copies normally made, and many more. The marginal cost of extra copies is like the marginal cost of extra copies of software, racing to zero.
What is the outcome of doing things this way? Tiering becomes irrelevant! Making copies is not necessary! Early all-flash users are sharing one copy of databases and code between 120 developers all with full read/write access, with no performance problems. Flash gives you better performing data, instantly available to any other application. Less expensive storage is not needed! The less active data will reside in flash in the same place as the copies more actively accessed. The result is the potential of creating many more logical copies.
New function is required. Great QoS is necessary to manage any performance conflicts. Catalogs are required to manage logical snapshots and physical placement and backups. Scale-out with 8 or more processors gives more compute and more sharing than dual-processor controllers. Automated management will require its own system of intelligence. The journey is tough; the endpoint inevitable.
Action Item: IT will sound like yesterday’s news if it continues to advocate tiering, advocate putting the most important data on high performance storage, and declaring that less expensive lower performing storage is key and necessary. Today’s flash reality is that a single compressed/de-duplicated physical version of data must be shared between as many different applications as possible, because the incremental cost of creating and sharing another copy is close to zero.
Links
The challenge is not technology. The CIO challenge is leadership to change the mindsets of practitioners and adapt the current processes that have built up over decades. In the video below, David Floyer, CTO & Co-founder of Wikibon, presents Storage Directions 2015-2020, and concludes that all-flash datacenters are a strategic imperative by 2016. They will reduce IT budget and lay the foundation for a migration to Systems of Intelligence.
Wikibon Storage Projections to an All-flash Datacenter in 2016
Additional Wikibon Flash Research
Top Ten Reasons why CIOs Should Migrate to All-flash Datacenters by 2016
Flash as Memory Extension is the Dominant Architecture for Enterprise Performance Computing
