Extensible NetApp: Thinking outside of the box

The assault on RAID

Earlier this week I wrote about how the core architecture of Traditional Legacy Arrays limits the performance of RAID-6. Just to recap, RAID-6 requires an additional read and an additional write over and above RAID-5, limiting it’s performance and applicability.

What I didn’t do is explore the consequences of such limitations.

The first, and most obvious consequence is the proliferation of RAID-10. SATA disks are too big to be used with RAID-5. The only alternative is to use RAID-10 when you need both resiliency and performance!

This singular fact turns the economics of RAID on it’s head.

Let’s be clear, the reason RAID became so popular is because it solved two generic problems: performance and availability at a lower cost than the alternatives. The problem with disk drives is that they have a limited number of IOPS per spindle. The only way to get more IOPS is to combine more of them and spread the IOPS over the spindles. The downside is that the MTBF (mean-time-between-failures) of the disk drives makes the group of disk drives increasingly less reliable.

So if you’re a customer that has business critical application that requires availability and performance, you’re more or less screwed. To get the availability and the performance you need to 2-3x the storage. As an example, consider GFS, that uses mirroring and no RAID. GFS will have at will have at least 3 full copies of the data.

Enter RAID.

RAID is so common place now-a-days that we forget how miraculous it must have appeared to folks when it first showed up. RAID allowed the computer industry to provide resiliency and performance at a lower cost! So for example, using RAID-5, you only need 1/5 (4+1) additional storage to get terrific data resiliency compared to the 3x that GFS requires.

Enter Big Drives

The problem with RAID-5 and RAID-4 is that the probability of various kinds of failures increases as the disk drive capacity increases. No surprise.

NetApp pioneered the use of RAID-6 with our FAS systems, and because of WAFL we are able to preserve the RAID value proposition of increasing capacity and performance with modest increase in physical capacity.

But we’re not the market.

As much as I would love to be the top-dog in the storage industry, the Traditional Legacy Array vendors, and in particular EMC, don’t have a very good RAID-6 implementation. Or rather, as I said earlier, their core architecture precludes a RAID-6 implementation that they can recommend for all workloads.

And because of their share of the market, they define the technology expectations for everyone who intends to use storage.

The problem is that we don’t live in a static universe. Software companies, like Microsoft, are continuously trying to reduce the total cost of their solutions. And although it’s nice to be able to bill customers for RAID-10, the reality is that for every dollar that goes into the hardware, Microsoft makes less money.

So Microsoft and the rest of the software industry, faced with the reality that the dominant Traditional Legacy Array vendors, were pushing RAID-10, or basically 2x the physical capacity for each byte of data stored, looked elsewhere for a solution.

It must have been particularly galling to software vendors that disk drive capacities were increasing, disk drive cost per GB per decreasing, but the storage infrastructure was increasing in cost reducing the total available pie.

Enter host based replication etc…

The cool thing about technology trends and curves, is that everyone is staring at them.

If I was a software application architect here’s what I would observe. The biggest cost for an application from a hardware perspective is the storage, and biggest cost factor is the disk drives.

Today with RAID-5, the picture looks like this:

The red controllers are provided by Traditional Legacy Array vendors, and there is a small increase in the total capacity to get excellent resiliency (the pink area).

But with bigger SATA drives, and the absence of a high performance RAID-6, the world looks like this:

Ouch. The total cost of the solution just jumped up! You get cheaper disk drives, but you need more of them!

Now if I am an application architect, I think to myself, you know that RAID stuff can be done in software, and I can do the mirroring at the application level, and the server vendors sell really cheap disk drives, so maybe the picture can really be made to look like this:

Each server has a pool of disk drives, and mirroring is done in the host to obtain some degree of storage resiliency. Yes the costs are greater than RAID-5, but it’s not clear they are that much higher than RAID-10 … And you might even believe they were lower than RAID-10 because you’re saving on the dollars you had to give to the storage company.

So what does this have to do with NetApp?

I really believe that RAID-10 mirroring or DAS is an inefficient way to manage disk drives.

I also believe that the popularity of DAS/Host mirroring solutions is a response to the failure of the market leaders to build an array that can preserve the RAID value proposition.

Since RAID-10 and DAS/Host mirroring are a response but the wrong fundamental approach, I believe that architectures, like NetApp’s, that make RAID-6 work will ultimately win out.

And oftentimes in the debate about DAS vs shared storage, we confuse the specific problems of the Traditional Legacy Array vendors, with the right long term technology trend.

The importance of architecture

This is a more technical/philosophical post than my usual commentary, forewarned is forearmed.

Earlier I wrote about the notion that software is like group theory. What I said was that there exist abstractions that form in a software system over time, and that the key to understanding and improving a software system is to understand those abstractions.

I also argued, that those abstractions were not accidental, but were actual manifestations in software of how the world really worked. By making the case that these were real abstractions and not artificial constructs, I argued that changing or fighting them was done at your peril.

What's important, of course, is that any mathematical, and therefore by extension, software abstraction is an approximation of the world, not the real thing. And that approximation may help you solve some problems better than others ...

When we argue about software and hardware systems in the corporate storage blog-o-sphere we are arguing about specific notions of how the world is actually constructed. I believe part of the acrimony and bitterness is that we're not just puffing our chests and cheering our team on, but we actually have different wold views.

Enough with the blah, blah, architecture, blah, group theory, blah

My recent posts about WAFL reminded me of a theory Prof. David Cheriton repeated in his Stanford course on distributed systems. What Prof. Cheriton observed was that many times innovation happens when, by exploiting a computer trend, you relax a previously hard constraint making a previously hard thing easy.

The example he used, if my increasingly faulty memory serves me right, was the use of HTML/XML and ASCII to serialize or marshall and then deserialize or unmarshall parameters.

A canonical problem in distributed system design is how to transport data over the wire. The problem is that data is represented as a stream of binary digits in main memory. The only thing that gives it context is the structure software imposes on the data.

The problem in a distributed system is that the software can be written in two different programming languages running on different hardware meaning that there has to be well understood language and hardware independent rules about what the stream of bits mean.

For example, suppose I have remote procedure call int foo(int b). I need to convert the input parameter b into some format that can be transported over the wire, and interpreted at the other side as an integer with the same value.  This transformation is what is known as marshalling and unmarshalling, I'll call it translation.

The traditional approach to this problem was to use a binary protocol that encoded the integer into a system independent format as you went over the wire and then translated the binary data into the system dependent format at the point where it needed to be interpreted. Software on either side of the wire using the translator could send and receive data.

Building such and infrastructure that is robust is a relatively hard but straightforward problem.

Building such an infrastructure that is widely used by everyone was impossible.

The problem was that there was no industry standard for the translators. For computer programs to talk to each other over the wire they needed proprietary and expensive binary translators. CORBA, for example, was a failed attempt at creating such a standard.

Enter HTML/XML

HTML/XML are horribly inefficient compared to a binary translator. HTML/XML are inefficient  with their use of bandwidth and their use of compute resources. For example a four byte integer quantity can turn into an 11 byte string, plus the meta-data required to tag the value.

But HTML/XML do not require a proprietary binary translator on either side of the wire.

So the idea was to relax your requirements for performance to get ease of use. And that was okay because CPUs  were becoming faster, memory was becoming cheaper and bandwidth was becoming more plentiful.

The net result was that HTTP+HTML/XML became the defacto mechanism for programs to communicate to each other over the internet and things like ONC RPC/CORBA/JAVA RMI/DCOM died or are dying a slow and increasingly painful death.

But if that's too abstract and confused...

Computer programming languages are another good example.

Programs written in C can be faster and more efficient than programs written in Java. But the time to develop, debug and qa those programs is vastly greater than an equivalent program written in Java. 

By trading off computational efficiency (memory and CPU resources), programmers are made more efficient, resulting in more software being developed.

I am storage architect not a computer science historian...

When Dave Hitz and James Lau architected WAFL they relaxed the fundamental constraint that file systems had preserved for a long time, namely that the client notion of disk layout actually corresponded to the on-disk-layout.

This was a profound decision. By relaxing that constraint, WAFL had taken on the job of providing the illusion to clients that the disk-layout was sequential even if it wasn't.

At the time that might have appeared to be a ridiculous thing to do. And given the reaction of the other guys (vendors of Traditional Legacy Arrays) still seams to be a ridiculous thing to do.

But...

If you looked at the computer trends, maybe it wasn't. By exploiting the increasing performance of CPU's, the increasing density of memory, it became possible to write smarter and smarter algorithms that were more and more capable of providing the illusion that the disk layout was sequential even if it wasn't (for more on this topic see an earlier post)

And by relaxing that constraint, it became possible to deliver innovation like thin provisioning, Real Snapshots and deduplication without compromising performance.

Why?

All of these features, primary dedup, Real Snapshots, high performance RAID-6, require an on-disk layout that is different from the client layout. For our, NetApp, internal abstractions, these are easy to do and a natural extension of our software abstractions. And modifying your system in natural ways is easy only if your abstractions easily map to those abstractions.

For Traditional Storage Arrays, fixed in their notions of disk layout, asking them to support a model where the on-disk layout is different from the client layout challenges the fundamental abstractions they have evolved over the last 15 years. And so they try and create approximations, like pudgy provisioning, but ultimately the extensions looks like an ill fitting suit, because they are.

Proven Architectures and Group Theory.

In my last post I explored notions of system architecture. Other than helping a new generation of software architects, a consumer of technology should legitimately ask:

What's in it for me?

There are a lot of consequences.

The one I want to focus on is this notion of proven. Vendors like to say lots of things about what is and is not proven, but if you believe what I said earlier, then here's what you need to know.

1. Buy a proven architecture.

Buying a proven architecture is not just a risk averse decision, it's a sane and technologically sound decision. Of course if a new architecture brings tremendous benefits, waiting to adopt the architecture until it is proven has other risks so there is a tradeoff.

2. Proven does not mean orthodox

What's important to note is that proven is more important than orthodox.Some vendors like to pooh-pooh their competition because the alternative architecture does not match their notions of orthodoxy.

Vendors who make appeals to orthodoxy are assuming that there exists one true of way of doing things. Or even better yet, are assuming customers are unwilling to accept the notion that there are multiple ways of solving the same problem.

3. Discount the "I am not insane, the world is insane" attack on proof.

There is a temptation among engineering companies to appeal to a customers engineering bias to pick something because the architecture is superior. Or even better to discount proof or evidence in favor of Fear, Uncertainty and Doubt about the architecture.

The reality is that every architecture of any robust reliable system represents the collected wisdom of many customer deployments.

Because if the software actually reflects how the world really works, then claiming the software doesn't work is like the man who says: I am not insane, the world is insane.