As a systems integrator we provide services, support and consultancy in addition to our hardware and software solutions. One of the questions we’re often asked by customers – both enterprise and research organizations - is around what storage technologies they should be using.
The answer, invariably, is as complex as the technologies we’re talking about. There’s no right or wrong answer in choosing a storage solution, but your choices will have an impact on how well it will perform.
Killer combo
For the purposes of this article, I’d like to focus on storage hardware at a high level; Solid State Drives (SSDs), Hard Disk Drives (HDDs) and Tape. I’m going to make some generalizations here as these technologies do have a lot more written about them individually.
If you need a large amount of performance and less capacity, the cheapest way to achieve that is with SSD’s – the cost per Input/Output Operations Per Second (IOPS) is much lower than disk or tape. It is, of course, much more expensive per terabyte of capacity and that’s when things start to get a little more complicated.
Filling your rack with SSD’s for capacity would be very expensive and, similarly, filling racks with HDD’s to achieve performance wouldn’t necessarily be the most cost-effective route either (although it can be, and is, done – when it matches a customer’s mix of performance and capacity). Whilst entirely dependent on applications running on the system and the types of workloads that would be exposed to the storage system, we’d often suggest a mix of both SSD’s (for performance) and HDDs (for the capacity).
In HPC storage systems for example we often use SSD’s for metadata, yet still use HDD’s for capacity. This is sometimes also combined with an SSD tier for very small or recently used files. Additionally, most of our Storage Area Network (SAN) storage array partners have technologies that allow SSD’s and HDD’s to be added into the same disk pool automatically, moving ‘hot blocks’ of data to the faster (but more expensive) SSD tier, which is great for more traditional enterprise storage requirements.
With SSD’s getting larger and more cost effective, further down the line we may see ‘spinning rust’ HDD’s fall by the wayside. With 12TB drives now available, capacities have effectively doubled in the last 3 years but have not got any faster (essentially halving the performance per TB).
More often than not we are still using 6TB HDD’s to get the combination of capacity and performance that our customer’s desire but in the near future I see our customers using SSD/Flash cache devices in front of large HDD’s to get the best of both worlds. With SSD’s getting larger and cheaper it does beg the questions of whether or not we could get rid of the disk tier altogether and operate a flash + tape model?
There are, today, some very specific applications where using flash and tape is by far the most cost-effective storage setup. And, with IDC predicting that the price difference of flash compared to disk drives will drop from its current 6.6x to 2.2x by 2021, it could mean the flash + tape mix is a viable, cost effective alternative to the traditional three-tiered storage infrastructure. This is something I have written a few blogs about recently.
We’re certainly not at the tipping point where this kind of solution would be right for everyone but the number looking into these solutions in both enterprise and research organizations is growing.
Many organizations have performance and capacity use cases for all of the technologies I’ve described above, the challenge is bringing all of those technologies together in a unified way so you don’t end up with inefficiency through poor utilization of many disparate storage systems. There are a number of file systems, visualization technologies and even appliances that mix a number of these technologies together to help minimize the management and efficiency costs of using multiple technologies today.
Unfortunately, there is no one-size-fits-all answer on what the perfect storage media mix would look like, but as we see prices of SSD’s come down, capacities of HDD’s go up and the sheer cost effectiveness of tape, the way we build our storage solutions will continue to evolve.
Andy Dean is HPC Business Development Manager at OCF