While there is a multitude of startups pitching software-defined data center solutions, the big vendors all have a software-defined play of their own. This is certainly true in the world of storage. Both EMC and NetApp, the two giants of the data center storage world, had no choice but to stake claims in the buzzing software-defined market. And stake claims they have.

While NetApp’s play in the space built on its existing storage operating system – Data OnTap, – EMC rolled out a whole new approach earlier this year. It is hard not to notice how similar the philosophies of the two companies are on what software-defined storage should do, and perhaps the biggest similarity is the acknowledgment that whatever software solution a customer goes with, they should be able to use it across all hardware, whether it is made by EMC, NetApp, IBM or any other vendor.

EMC unleashes ViPR in Vegas

EMC made a huge deal out of its software- defined storage announcement at the EMC World conference in Las Vegas in May. ViPR, the product, virtualizes storage, abstracting physical hardware underneath. It creates virtual storage pools that can be assigned to applications. Management functions are automated and provisioning is done through a self-service portal.

The idea is to have ViPR eventually support as many vendors’ hardware as possible, but when the solution becomes generally available, sometime in the second half of 2013, it will only support EMC and NetApp gear, says Manuvir Das, VP of engineering at EMC’s Advanced Software division. Das was one of the key brains behind ViPR.

Beyond NetApp, what ViPR will be integrated with will depend on market need. “ViPR itself does not have to be customized in any way,” Das says. If a customer has a particular array they want to use ViPR with, an adapter can be written for that particular array. “That’s the extent of customization we’re talking about.” The adapters can be written by EMC or by others. “You can think of it as analogous to a device-driver model,” he says.

Every storage array today typically has a publicly available API (application programming interface), which is what the ViPR adapter uses to control it. So an open API is the only criterion for a storage device to be compatible with ViPR.

The solution was built with new application workloads in mind. These are apps built on Node JS, Ruby on Rails, Python or Hadoop. While the amount of these next-gen workloads will grow, traditional workloads are not going anywhere any time soon. To accommodate both types, ViPR has separate data plane and control plane. The control plane creates and provisions virtual storage pools to applications, which is all the traditional workloads need, while the data plane manages data for next-gen applications that use things like object storage and the HDFS protocol for analytics.

Work on what eventually became ViPR began about two years ago, Das recalls. Most customers EMC has worked with have storage environments with multiple solutions, each with its own management software. This complexity makes the life of storage admins very difficult. “Every array [vendor] is innovating on its own management stack, so you actually get further and further into the siloed world,” Das says.

Like any IT vendor, EMC’s ultimate job is to make life better for IT staff. “We know that the right thing to do on behalf of the customer here is to build a software layer that will bring everything together,” Das says. What has pushed these ideas to fruition over the past couple of years is the general realization that a lot of this management functionality can live in software, outside of the storage box. There has traditionally been a prevailing attitude in the storage world that because it is important not to lose data, the device that stores it needs to be “rock-solid, like a car”, complete with all the management features. But software is software, regardless of where it sits. “The reality is every hardware device has software running on it anyway,” Das says.

Is commodity hardware a threat?

This begs the question whether companies like EMC, who have for many years innovated to compete on not only storage capacity and performance but also management features, are shooting themselves in the foot. Isn’t hardware-agnostic software making more room for commodity hardware vendors, who make simple and low-cost boxes? Das does not think so.

“We do see commodity hardware coming more into play than in the past,” he says. “As a company, we don’t shy away from that. We’re perfectly comfortable with that.” Functionality like deduplication and data protection are not easy for an end user to develop in-house, so EMC does not foresee a decline in demand for devices that just work once they are installed.

And Das knows how hard it is to develop those advanced features from experience. He was a member of the team at Microsoft that built and operated the infrastructure that supports Windows Azure, Microsoft’s public-cloud services. There is no book for building resilient systems at scale from scratch. It is “an art that you have to pick up through repeated experience,” he says.

This means only companies operating at mega-scale – like Google or Microsoft – will be the market for commodity hardware. The rest of the IT-using world will want the all-included boxes for a while, he says.

NetApp: software defined OnTap

While EMC is not worried about commodity hardware manufacturers eating too much into its market share, its arch rival NetApp is a different story. The two companies essentially play in the same market space. According to NetApp’s company philosophy, which, as already mentioned, is quite similar to EMC’s, software-defined data center technology has to allow customers to provision services themselves through software, outside of any hardware boundaries. Brendon Howe, VP of product and solutions marketing, says Data OnTap fits nicely within this framework’s three pillars.

First, it enables customers to provision virtual storage services from a pool of infrastructure. That pool may consist of a multitude of different technologies, with storage arrays of different sizes, different quality of service characteristics and made by different vendors. The abstraction layer “that isolates that storage virtual machine from the physical system itself is a really important starting point in how this is accomplished,” Howe says. “Without that type of capability, there really is no software defined framework whatsoever.”

The compatibility with hardware by multiple vendors is the second fundamental capability. Customers use NetApp’s V-Series controllers, running Data OnTap, to control EMC, HDS, HP, IBM and Fujitsu storage arrays. V-Series is a physical appliance, however, but there is a software-only version designed for branch offices. The OS can also be used in the public cloud. Users can extend it to manage virtual infrastructure in Amazon Web Services’ cloud. “You can run it on anything you want,” Howe says.

The third pillar of NetApp’s software defined philosophy is the self-service layer customers can use, and the vendor provides a variety of options in this department. Data OnTap works with Microsoft’s Hyper-V, VMware’s vSphere and Citrix’ XenServer virtualization platforms, all of which can be used to provision storage via the storage OS. It can also be plugged into open-source cloud architectures OpenStack and CloudStack.

NetApp enhanced a lot of this software defined functionality in the latest release of the OS (8.2). One of the biggest additions is the concept of a storage virtual machine, or SVM. Much like a virtual server created through Hyper-V or vSphere, the SVM is assigned to a certain application but can be moved from one piece of hardware to another, regardless of the hardware’s physical attributes (or, according to Howe, its manufacturer). Clients can relocate SVMs without taking them down, and this kind of non-disruptiveness was a major focus for this release, Howe says. “We provide customers the ability to re-balance workloads, introduce larger applications that aren’t part of the cluster yet,” he says. “We give them freedom to change the environment without disrupting the application that’s running.”

In the latest release, NetApp has also expanded the operating system’s ability to scale. A single instance can now support up to 69 petabyes of storage and 24 controller nodes. It can manage 12,000 network attached storage (NAS) volumes for more than 100,000 clients. The vendor did a lot of work with the new version’s ability to support different quality-of-service levels for different applications running in multitenant and multiworkload environmemts for storage area networks (SAN) and NAS.

In real life

One of the first companies to use NetApp’s new OS is Cirrity, a provider of cloud infrastructure services. The infrastructure that supports its services consists entirely of FlexPods, the pre-designed full-package IT solution that combines Cisco’s Unified Computing System servers and Nexus network switches with NetApp FAS (fabric attached storage).Dan Timko, Cirrity CEO, says the company has been using Data OnTap in cluster mode for about three months. One of the biggest features for Cirrity, as a provider, is the new non-disruptive operation capabilities. “The real benefit is in cluster mode we can buy right-sized equipment, so we don’t have to go out and buy their biggest filer,” he says. Cirrity can scale capacity as the need arises without disruptions to customer applications.

Another important feature for Cirrity is workload balancing. The software allows them to limit IO resources available to each storage volume, so a single volume cannot kill an entire filer at the expense of its neighbor volumes if it suddenly starts using too much IO. This makes it easier for the provider to control and deliver quality of service for each customer volume. OnTap’s SVM capability enabled Cirrity to give each customer a certain level of management of the storage capacity they buy from the provider. A customer receives an SVM with a certain amount of storage and can use NetApp software features to manage it. Cirrity has also used this feature for its disaster recovery services. If a customer has a NetApp filer in their private data center, the provider can use an SVM to replicate it in one of its two data centers.

On to stand-alone software sales

Storage vendors are entering a new world where competing on storage density of the media and energy efficiency and management features like deduplication and tiering is not enough. With EMC introducing data management features for applications like Hadoop, across multiple types of storage, the game will be changing quite a bit. Neither EMC nor NetApp are denying the prospect of selling stand-alone software very soon.

And they are building software to be compatible with each other’s hardware and other competitors’ boxes. All of this represents a huge shift in thinking about IT. Whether it will eventually lead to a much easier life for IT staff remains to be seen.

This piece originally ran in the 30^th edition of the DatacenterDynamics FOCUS magazine. Subscribe for free on the DCD website.