Extensible NetApp: Thinking outside of the box

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

FAS the new DAS

When the Forrester report came out with its report on DAS, the response from the storage industry was predictable:

1. Same old, same old
2. Immature, unreliable, inefficient technology
3. Whole product has value

And when I read those responses, I could not but feel a little bit like every disruptee feels when a new disruptive technology shows up.

The funny thing with the Forrester guys is, that they are right. DAS is disruptive to the traditional storage array, in a way that the storage industry has not owned up to. And that unless the storage industry responds, DAS may win out.

The problem with the report is that they knew there was fish in one of the ponds, they just went to the wrong pond, and their report was dismissed.

So instead, I am going to offer a different point of view about the disruption of storage by DAS.

John Fulbright, has an excellent analysis where he shows how the DAS disruption is tied to RAID-5 SAN and how NetApp RAID-DP is different.

This disruption, because it is a storage disruption will take years to shake out. Storage infrastructures change at glacial rates. Once a trend, however, sticks it sticks for a long long time.

This is going to be a long series, but the short version is below.

The disruption in a nutshell

In the late 90’s and early part of this decade, Data Center Architects, reduced costs in their data center by moving away from expensive reliable server architectures to cheap and unreliable server architectures. To address application availability, the Data Center Architects moved to applications that achieved high availability by leveraging clustering for CPU and Memory availability and highly available storage for data availability.

The new DAS disruption is about providing data availability by moving availability out of the storage infrastructure. The new DAS disruption is made possible because traditional legacy arrays are wasting vast amounts of capacity because they were built to optimize the performance of servers that were disrupted last decade. The waste is so significant that the cost advantage of DAS outweighs the value of improved utilization through centralization.

What does this mean for NetApp?

Ultimately shared storage must always compete with DAS. If you can’t deliver superior value at a reasonable cost, then you don’t belong in this industry. Arguing for total customer value is a quick way to being a disrupted technology.

NetApp storage systems were optimized, interestingly enough, for the server infrastructures that are being built today. Which means, that uniquely in the storage industry, we are able to deliver performance and efficient use of capacity.

If you compare the performance, capacity and utilization of our low-end storage systems against a DAS system, it turns out that we are cost competitive and offer superior utilization, performance and availability. This is exactly what John shows in his blog.

More prosaically it means that you can offer external storage in a DAS configuration and deliver value, a surprising conclusion.

And if you can deliver value for DAS, then you can deliver the value of centralized storage if you can combine that DAS into shared storage. And that’s where our ONTAP 8 product architecture fits in.

So unlike the rest of the storage industry, my reaction to DAS is: we can do that too, and we can do that better.

The assault on RAID

Earlier this week I wrote about how the core architecture of Traditional Legacy Arrays limits the performance of RAID-6. Just to recap, RAID-6 requires an additional read and an additional write over and above RAID-5, limiting it’s performance and applicability.

What I didn’t do is explore the consequences of such limitations.

The first, and most obvious consequence is the proliferation of RAID-10. SATA disks are too big to be used with RAID-5. The only alternative is to use RAID-10 when you need both resiliency and performance!

This singular fact turns the economics of RAID on it’s head.

Let’s be clear, the reason RAID became so popular is because it solved two generic problems: performance and availability at a lower cost than the alternatives. The problem with disk drives is that they have a limited number of IOPS per spindle. The only way to get more IOPS is to combine more of them and spread the IOPS over the spindles. The downside is that the MTBF (mean-time-between-failures) of the disk drives makes the group of disk drives increasingly less reliable.

So if you’re a customer that has business critical application that requires availability and performance, you’re more or less screwed. To get the availability and the performance you need to 2-3x the storage. As an example, consider GFS, that uses mirroring and no RAID. GFS will have at will have at least 3 full copies of the data.

Enter RAID.

RAID is so common place now-a-days that we forget how miraculous it must have appeared to folks when it first showed up. RAID allowed the computer industry to provide resiliency and performance at a lower cost! So for example, using RAID-5, you only need 1/5 (4+1) additional storage to get terrific data resiliency compared to the 3x that GFS requires.

Enter Big Drives

The problem with RAID-5 and RAID-4 is that the probability of various kinds of failures increases as the disk drive capacity increases. No surprise.

NetApp pioneered the use of RAID-6 with our FAS systems, and because of WAFL we are able to preserve the RAID value proposition of increasing capacity and performance with modest increase in physical capacity.

But we’re not the market.

As much as I would love to be the top-dog in the storage industry, the Traditional Legacy Array vendors, and in particular EMC, don’t have a very good RAID-6 implementation. Or rather, as I said earlier, their core architecture precludes a RAID-6 implementation that they can recommend for all workloads.

And because of their share of the market, they define the technology expectations for everyone who intends to use storage.

The problem is that we don’t live in a static universe. Software companies, like Microsoft, are continuously trying to reduce the total cost of their solutions. And although it’s nice to be able to bill customers for RAID-10, the reality is that for every dollar that goes into the hardware, Microsoft makes less money.

So Microsoft and the rest of the software industry, faced with the reality that the dominant Traditional Legacy Array vendors, were pushing RAID-10, or basically 2x the physical capacity for each byte of data stored, looked elsewhere for a solution.

It must have been particularly galling to software vendors that disk drive capacities were increasing, disk drive cost per GB per decreasing, but the storage infrastructure was increasing in cost reducing the total available pie.

Enter host based replication etc…

The cool thing about technology trends and curves, is that everyone is staring at them.

If I was a software application architect here’s what I would observe. The biggest cost for an application from a hardware perspective is the storage, and biggest cost factor is the disk drives.

Today with RAID-5, the picture looks like this:

The red controllers are provided by Traditional Legacy Array vendors, and there is a small increase in the total capacity to get excellent resiliency (the pink area).

But with bigger SATA drives, and the absence of a high performance RAID-6, the world looks like this:

Ouch. The total cost of the solution just jumped up! You get cheaper disk drives, but you need more of them!

Now if I am an application architect, I think to myself, you know that RAID stuff can be done in software, and I can do the mirroring at the application level, and the server vendors sell really cheap disk drives, so maybe the picture can really be made to look like this:

Each server has a pool of disk drives, and mirroring is done in the host to obtain some degree of storage resiliency. Yes the costs are greater than RAID-5, but it’s not clear they are that much higher than RAID-10 … And you might even believe they were lower than RAID-10 because you’re saving on the dollars you had to give to the storage company.

So what does this have to do with NetApp?

I really believe that RAID-10 mirroring or DAS is an inefficient way to manage disk drives.

I also believe that the popularity of DAS/Host mirroring solutions is a response to the failure of the market leaders to build an array that can preserve the RAID value proposition.

Since RAID-10 and DAS/Host mirroring are a response but the wrong fundamental approach, I believe that architectures, like NetApp’s, that make RAID-6 work will ultimately win out.

And oftentimes in the debate about DAS vs shared storage, we confuse the specific problems of the Traditional Legacy Array vendors, with the right long term technology trend.

MAD Blog: HP not letting it go ...

There is a specter haunting HP's Karl Dohm, the specter of NetApp's sale force snookering customers about the real value of NetApp storage.

Patrick mentions the true test, i.e. that there are many happy NetApp customers who are running Exchange.  There is truth to this of course, but it isn't a good basis of comparison because every major array vendor has happy Exchange customers.  However, its reasonable to say that these installations can't know what they don't know. 

Karl can not accept the notion that the value NetApp brings to the market is real. After first trying to prove that NetApp can not run on Exchange:

The problems we are talking about here are the core of WAFL, and are clearly not easy to fix - or they would be already fixed.  NetApp is not unique is having problems of course, all array vendors have their strong and weak points.  But to assert that WAFL has no weaknesses around fragmentation, performance, and capacity utilization defies common sense.  The old wounds are there for a reason.

Karl had to back peddle and admit:

I'm not saying you can't run Exchange successfully with NetApp.  In fact I'm sure you can.  The question looking for an answer is whether the user gets good value in choosing NetApp to run Exchange.

So reading Pat Cimprich's latest response was particularly gratifying.

Look, the FUD that HP is currently throwing out there is age old FUD that EMC has been throwing out for a while.

After 10 years of watching an artiste like EMC spread FUD, watching this ham-fisted attempt at spreading the same FUD is, well, embarrassing. And having to respond to it, is well irritating.

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

Using Protection Manager, SnapDrive for Windows and SnapManager for Exchange

In my previous post I described at a high level the power of the recently released SnapManager, SnapDrive and Protection Manager products.

To give you a flavor for the workflow of the tools let me walk you through the process of how they are used. *

First the storage architect or disk to disk backup architect defines a set of policies that can be used using Protection Manager.

To create a new policy the storage architect first selects one of the predefined policies

and then modifies that policy to meet the specific requirements of his organization.

Now the policy is ready to be used by an Exchange administrator.

Before you can use SnapManager for Exchange you need to configure SnapDrive for Windows to talk to a DFM Server that has Protection Manager license.

To use the policy the Exchange administrator brings up SnapManager for Exchange

and where before he could only configure local Snapshot and SnapMirror schedules, now the administrator has additional data protection policies that include disk-to-disk backup.

The Exchange administrator  then simply assigns policies to Exchange storage groups. As each policy gets assigned to a storage group, SnapManager for Exchange through SnapDrive for Windows creates a new dataset  in Protection Manager that contains the LUNs that contain the Exchange storage group.

Each dataset that is created does not, however, have any secondary storage resources assigned to it. That has to be done by the storage administrator.

To assign the physical resources, the storage architect would use Protection Manager to find the new datasets. This is made easy by Protection Manager, because the datasets would appear as a dataset that was out of conformance with respect to the specified policy.

The storage architect then assigns the right resource pool to the storage group. Once that is done, Protection Manager will  then automatically create the backup relationships, run the schedules, and configure any additional secondary storage objects.

Once that is done the Exchange storage group is protected.

* For more info on datasets, policies and resource pools check this paper we published in LISA 2007.

* For more info on how SnapManager and SnapDrive work check this article from Tech ONTAP.

Why are we integrating SnapManager for Exchange, SnapDrive for Windows, and Protection Manager?

SnapDrive for Windows 6.0 and Protection Manager, part of the DFM 3.7 release, and SnapManager for Exchange 5.0  are the first tangible artifacts that demonstrate the power of NetApp's Integrated Data Management Vision.

Three years ago in a meeting at NetApp, a bunch of us had the radical idea that if you could enable server and application administrators to manage their own data you could unlock tremendous amounts of productivity by eliminating processes and procedures in the data center that existed to overcome limitations in storage management systems. 

From that observation came two significant technological breakthroughs. The first was the introduction of our policy based management infrastructure as part of the Protection Manager product and the second is the integration of our very successful SnapDrive for Windows and SnapManager for Exchange products with that policy based infrastructure.

Superficially Protection Manager is a GUI for managing NetApp replication. But that, as they say, completely misses the point. If you think of Protection Manager as just a GUI, then you'll probably wonder why we ever bothered with the integration at all. And you would be right in wondering.

Protection Manager, in reality, is a policy automation layer coupled with a very capable delegation model. These two technologies allow storage architects to define policies for how storage should be used and in parallel allow other people to create storage based on those very same policies. In effect, the storage architect does the hard work of figuring out how he wants people to use his storage once, and leaves the relatively simple task of provisioning storage to other people secure in the knowledge that they can not make a mistake.