Extensible NetApp: Thinking outside of the box

The assault on RAID

Earlier this week I wrote about how the core architecture of Traditional Legacy Arrays limits the performance of RAID-6. Just to recap, RAID-6 requires an additional read and an additional write over and above RAID-5, limiting it’s performance and applicability.

What I didn’t do is explore the consequences of such limitations.

The first, and most obvious consequence is the proliferation of RAID-10. SATA disks are too big to be used with RAID-5. The only alternative is to use RAID-10 when you need both resiliency and performance!

This singular fact turns the economics of RAID on it’s head.

Let’s be clear, the reason RAID became so popular is because it solved two generic problems: performance and availability at a lower cost than the alternatives. The problem with disk drives is that they have a limited number of IOPS per spindle. The only way to get more IOPS is to combine more of them and spread the IOPS over the spindles. The downside is that the MTBF (mean-time-between-failures) of the disk drives makes the group of disk drives increasingly less reliable.

So if you’re a customer that has business critical application that requires availability and performance, you’re more or less screwed. To get the availability and the performance you need to 2-3x the storage. As an example, consider GFS, that uses mirroring and no RAID. GFS will have at will have at least 3 full copies of the data.

Enter RAID.

RAID is so common place now-a-days that we forget how miraculous it must have appeared to folks when it first showed up. RAID allowed the computer industry to provide resiliency and performance at a lower cost! So for example, using RAID-5, you only need 1/5 (4+1) additional storage to get terrific data resiliency compared to the 3x that GFS requires.

Enter Big Drives

The problem with RAID-5 and RAID-4 is that the probability of various kinds of failures increases as the disk drive capacity increases. No surprise.

NetApp pioneered the use of RAID-6 with our FAS systems, and because of WAFL we are able to preserve the RAID value proposition of increasing capacity and performance with modest increase in physical capacity.

But we’re not the market.

As much as I would love to be the top-dog in the storage industry, the Traditional Legacy Array vendors, and in particular EMC, don’t have a very good RAID-6 implementation. Or rather, as I said earlier, their core architecture precludes a RAID-6 implementation that they can recommend for all workloads.

And because of their share of the market, they define the technology expectations for everyone who intends to use storage.

The problem is that we don’t live in a static universe. Software companies, like Microsoft, are continuously trying to reduce the total cost of their solutions. And although it’s nice to be able to bill customers for RAID-10, the reality is that for every dollar that goes into the hardware, Microsoft makes less money.

So Microsoft and the rest of the software industry, faced with the reality that the dominant Traditional Legacy Array vendors, were pushing RAID-10, or basically 2x the physical capacity for each byte of data stored, looked elsewhere for a solution.

It must have been particularly galling to software vendors that disk drive capacities were increasing, disk drive cost per GB per decreasing, but the storage infrastructure was increasing in cost reducing the total available pie.

Enter host based replication etc…

The cool thing about technology trends and curves, is that everyone is staring at them.

If I was a software application architect here’s what I would observe. The biggest cost for an application from a hardware perspective is the storage, and biggest cost factor is the disk drives.

Today with RAID-5, the picture looks like this:

The red controllers are provided by Traditional Legacy Array vendors, and there is a small increase in the total capacity to get excellent resiliency (the pink area).

But with bigger SATA drives, and the absence of a high performance RAID-6, the world looks like this:

Ouch. The total cost of the solution just jumped up! You get cheaper disk drives, but you need more of them!

Now if I am an application architect, I think to myself, you know that RAID stuff can be done in software, and I can do the mirroring at the application level, and the server vendors sell really cheap disk drives, so maybe the picture can really be made to look like this:

Each server has a pool of disk drives, and mirroring is done in the host to obtain some degree of storage resiliency. Yes the costs are greater than RAID-5, but it’s not clear they are that much higher than RAID-10 … And you might even believe they were lower than RAID-10 because you’re saving on the dollars you had to give to the storage company.

So what does this have to do with NetApp?

I really believe that RAID-10 mirroring or DAS is an inefficient way to manage disk drives.

I also believe that the popularity of DAS/Host mirroring solutions is a response to the failure of the market leaders to build an array that can preserve the RAID value proposition.

Since RAID-10 and DAS/Host mirroring are a response but the wrong fundamental approach, I believe that architectures, like NetApp’s, that make RAID-6 work will ultimately win out.

And oftentimes in the debate about DAS vs shared storage, we confuse the specific problems of the Traditional Legacy Array vendors, with the right long term technology trend.

Better Than Real Fibre Channel: The Other Guys

Brian Rawlings, correctly, took me to task over my parochial focus on WAFL as the only Better Than Real Fibre Channel technology.

The reality is that the kind of things WAFL does are being done by other vendors. I'll claim not as well, or not as elegantly or not as completely, but there are other vendors who are breaking with Traditional Storage Array architectural orthodoxy that disk layout must correspond to what the client thinks the disk layout is.

So just for the record, the other guys include but are not limited to the folks at 3Par, Compellent, and Equallogic.

And for the record, the future belongs to us (just that more of the future belongs to NetApp :-))

The importance of architecture

This is a more technical/philosophical post than my usual commentary, forewarned is forearmed.

Earlier I wrote about the notion that software is like group theory. What I said was that there exist abstractions that form in a software system over time, and that the key to understanding and improving a software system is to understand those abstractions.

I also argued, that those abstractions were not accidental, but were actual manifestations in software of how the world really worked. By making the case that these were real abstractions and not artificial constructs, I argued that changing or fighting them was done at your peril.

What's important, of course, is that any mathematical, and therefore by extension, software abstraction is an approximation of the world, not the real thing. And that approximation may help you solve some problems better than others ...

When we argue about software and hardware systems in the corporate storage blog-o-sphere we are arguing about specific notions of how the world is actually constructed. I believe part of the acrimony and bitterness is that we're not just puffing our chests and cheering our team on, but we actually have different wold views.

Enough with the blah, blah, architecture, blah, group theory, blah

My recent posts about WAFL reminded me of a theory Prof. David Cheriton repeated in his Stanford course on distributed systems. What Prof. Cheriton observed was that many times innovation happens when, by exploiting a computer trend, you relax a previously hard constraint making a previously hard thing easy.

The example he used, if my increasingly faulty memory serves me right, was the use of HTML/XML and ASCII to serialize or marshall and then deserialize or unmarshall parameters.

A canonical problem in distributed system design is how to transport data over the wire. The problem is that data is represented as a stream of binary digits in main memory. The only thing that gives it context is the structure software imposes on the data.

The problem in a distributed system is that the software can be written in two different programming languages running on different hardware meaning that there has to be well understood language and hardware independent rules about what the stream of bits mean.

For example, suppose I have remote procedure call int foo(int b). I need to convert the input parameter b into some format that can be transported over the wire, and interpreted at the other side as an integer with the same value.  This transformation is what is known as marshalling and unmarshalling, I'll call it translation.

The traditional approach to this problem was to use a binary protocol that encoded the integer into a system independent format as you went over the wire and then translated the binary data into the system dependent format at the point where it needed to be interpreted. Software on either side of the wire using the translator could send and receive data.

Building such and infrastructure that is robust is a relatively hard but straightforward problem.

Building such an infrastructure that is widely used by everyone was impossible.

The problem was that there was no industry standard for the translators. For computer programs to talk to each other over the wire they needed proprietary and expensive binary translators. CORBA, for example, was a failed attempt at creating such a standard.

Enter HTML/XML

HTML/XML are horribly inefficient compared to a binary translator. HTML/XML are inefficient  with their use of bandwidth and their use of compute resources. For example a four byte integer quantity can turn into an 11 byte string, plus the meta-data required to tag the value.

But HTML/XML do not require a proprietary binary translator on either side of the wire.

So the idea was to relax your requirements for performance to get ease of use. And that was okay because CPUs  were becoming faster, memory was becoming cheaper and bandwidth was becoming more plentiful.

The net result was that HTTP+HTML/XML became the defacto mechanism for programs to communicate to each other over the internet and things like ONC RPC/CORBA/JAVA RMI/DCOM died or are dying a slow and increasingly painful death.

But if that's too abstract and confused...

Computer programming languages are another good example.

Programs written in C can be faster and more efficient than programs written in Java. But the time to develop, debug and qa those programs is vastly greater than an equivalent program written in Java. 

By trading off computational efficiency (memory and CPU resources), programmers are made more efficient, resulting in more software being developed.

I am storage architect not a computer science historian...

When Dave Hitz and James Lau architected WAFL they relaxed the fundamental constraint that file systems had preserved for a long time, namely that the client notion of disk layout actually corresponded to the on-disk-layout.

This was a profound decision. By relaxing that constraint, WAFL had taken on the job of providing the illusion to clients that the disk-layout was sequential even if it wasn't.

At the time that might have appeared to be a ridiculous thing to do. And given the reaction of the other guys (vendors of Traditional Legacy Arrays) still seams to be a ridiculous thing to do.

But...

If you looked at the computer trends, maybe it wasn't. By exploiting the increasing performance of CPU's, the increasing density of memory, it became possible to write smarter and smarter algorithms that were more and more capable of providing the illusion that the disk layout was sequential even if it wasn't (for more on this topic see an earlier post)

And by relaxing that constraint, it became possible to deliver innovation like thin provisioning, Real Snapshots and deduplication without compromising performance.

Why?

All of these features, primary dedup, Real Snapshots, high performance RAID-6, require an on-disk layout that is different from the client layout. For our, NetApp, internal abstractions, these are easy to do and a natural extension of our software abstractions. And modifying your system in natural ways is easy only if your abstractions easily map to those abstractions.

For Traditional Storage Arrays, fixed in their notions of disk layout, asking them to support a model where the on-disk layout is different from the client layout challenges the fundamental abstractions they have evolved over the last 15 years. And so they try and create approximations, like pudgy provisioning, but ultimately the extensions looks like an ill fitting suit, because they are.

Real Fibre Channel is really Traditional Legacy Array

Chuck Hollis likes to call the CLARiiON LUN a Real LUN.

I've taken to mocking Chuck and EMC by calling their storage Real Fibre Channel.

But after my recent series on RAID-6, and Thin Provisioning and the fact that Real Fibre Channel has architectural  flaws that make it less than useful for either of those critical capabilities, and the limitations around snapshots, I've been thinking that the term was just plain wrong.

Real Fibre Channel is like the PDP-11:  A legacy hardware architecture that has had it's moment in the sun.

Henceforth, I'll call the CLARiiON and the architectural dinosaurs that are like it, what they really are:

Traditional Legacy Array.

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Real Thin Provisioning vs Pudgy Provisioning

Last week I observed that thin provisioning creates fragmentation, but I never got into the details.

And I further observed that our competition wasn’t really doing thin provisioning. I didn’t have a catchy name for that kind of provisioning, and today I do: pudgy provisioning.

Today I want to go into the details.

In a Real Fibre Channel environment, the space for the LUNs are pre-allocated and assigned to individual LUN's.

The picture above tries to show the story when using Real Fibre Channel LUNs.

When a LUN gets allocated the free space in the storage array gets committed to the particular LUN. This space allocation is not just a logical accounting trick, but is an allocation of particular disk blocks. In effect, the space allocation is pre-committed to a LUN before the application using the LUN actually uses the space.

The rationale for this scheme is to ensure that the disk layout that the array uses is identical to the disk layout the application assumes to ensure good sequential throughput. For more check out this blog entry.

The problem is that to ensure good performance the storage array wastes space.

What about thin provisioning?

The design goal for thin provisioning is to reduce the amount of waste by deferring the space allocation in the array to when the space actually gets used by the application.

The picture above tries to show the theory of thin provisioning. Unlike Real Fibre Channel space only gets committed to the LUN only when it’s actually used, eliminating the waste.

The picture, however, is misleading, because it implies that the layout on disk doesn’t change.

Enter Fragmentation

The problem with thin provisioning is that applications do not necessarily do their write operations in a way that ensures that data layout on disk is preserved.

Suppose the write order for the application is

But then the problem is that the layout on disk looks like this:

NetApp, because this kind of layout issues just happens to be endemic to how WAFL works, has solved this problem.

If you can’t do thin provisioning, how about pudgy provisioning?

If you don’t have WAFL and 15 years of experience dealing with the problem, making this work as well as WAFL is hard.

But there are shortcuts.

WAFL will allocate a disk block on disk as a disk block gets written by the application.

The most obvious alternative is to instead of doing a one-to-one disk block allocation to do a one-to-many disk block allocation. Or what I like to call pudgy provisioning.

Let’s be precise:

Whenever a single block allocation is made, rather than allocate  a single block at the next free space in the array, the array allocates an entire region.

The value of this approach is that the on disk order matches the application’s notion of what the disk order is:

.

The problem of course, is that , if the application doesn’t allocate all of the storage blocks then you’ve got waste.

Huh?

Let me see if I can follow this logic:

1. Application groups ask for too much storage
2. Real Fibre Channel creates waste because space is allocated before it’s used
3. Thin provisioning allows you to allocate only space that is used
4. If you don’t have WAFL then fragmentation on disk becomes a performance problem
5. So instead of thin provisioning do pudgy allocation where you allocate more space than the application needs but not as much as before
6. Which leads to waste…

This makes perfect sense.

SANscreen Lives!

Chris Evans, The Storage Architect, declared victory over SANScreen with EMC’s latest offering.

And it’s the usual set of claims about how EMC, the company that brought us ECC, is better at delivering management tools and products.

Huh?

Management software is about delivering simplicity to the customer, and EMC has a lot of value in their infrastructure and their offerings, but the value of simplicity is not what comes to mind.

EMC infrastructure continues to suffer from a proliferation of incompatible architectures that provide point value in the data center.

Their management tools seems to be following that trend. They have multiple management consoles for management products. Even their unified storage architecture has no consistent way to manage replication or storage provisioning.

Their answer to the management complexity they create with their infrastructure is to create another management infrastructure.

But enough about His Excellency Mr. Complexity

What really surprised me was the outrageous claim that SANscreen has made very little progress in the years since the acquisition. He derisively cited the progress as meaningless reporting.

Earlier this year I wrote about the recent SANscreen 5.0 release.

Just to remind folks, SANscreen 5.0 added

1. iSCSI support
2. Storage tiering

And what I didn’t talk about then was the support we added for VMware, called VMInsight.

So if managing your SAN and iSAN and managing your space adds no value, well then I don’t know what does.

And yes, I know there is a collection of EMC management tools….

I have no doubt that through a combination of ControlCenter and SAN Configuration Advisor, and who knows what else, there is an overlapping value proposition that you could construct using EMC products.

But that’s the wrong way to go.

In management software, fewer products is better than more products. The fewer products, the fewer workflows, the less training, the simpler operations are.

And SANscreen not only delivers compelling  value but it also delivers the value for heterogeneous infrastructure and in one place.

The increasingly meaningless question of storage utilization

I've been recently engaged in a series of internal discussions about the question of what is storage efficiency.

As a company, NetApp has continued to deliver amazing amount of value by allowing our customers to store huge amounts of data on a relatively small storage foot print and yet gets routinely criticized for our inefficiency.

In point of fact, we create a marketing program to put our money where our mouth is, hell we even run a local marketing campaign, and still we get asked: well how efficient are you really?

Which makes me wonder if we're just barking up the wrong tree.

Storage Efficiency is so last century

SAN storage architectures built around Real Fibre Channel principles are what created the current insanity that frames the storage efficiency discussion.

They declared that the storage problem could be best be solved by creating a storage architecture that first required you to combine lots of storage into a single disk pool, and then statically parceled out the resources into smaller chunks that were handed out to servers. Storage utilization was a measure of how effective you were in using your storage pool. And then the question was of the storage you had allocated how efficient you were in using that.

Great.

So the entire storage industry is married to a storage management model that says that storage efficiency is a measure of how efficiently you used antediluvian architectures based around static partitioning.

Wonderful.

But the problem, and here's the rub Better Than Real Fibre Channel can do things that Real Fibre Channel can not. Because Better Than Real Fibre Channel can use block level reference counts, Better Than Real Fiber Channel can store 255 full copies of data at the cost of that actual unique data within those 255 copies.

So if I have a snapshot, a single NetApp snapshot, I have a full copy of the file system. But instead of being applauded for storing a huge amount of data on a small amount of storage, I have to explain the WAFL overhead.

I'll be blunt, the WAFL overhead allows you to store what to a client appears to be a full copy of the file system on the minimal amount of disk blocks necessary. You wanna do that using Real Fibre Channel, I really hope you like buying disk drives.

And if it was a couple of EMC bloggers I wouldn't mind. But this discussion has permeated the entire computer industry.

And I wouldn't care too much except the question of how we should measure storage system efficiency is not even being discussed because of the fusillade of FUD. And because at the end of the day no one really wants to talk about how inefficient their architectures are.

If they did, storage customers would learn exactly why it's better to use RAID10 to protect your storage instead of replicating your data at the application level.

It's almost 2010 ...

NetApp and Data Domain, and soon others, have demonstrated that a storage system that uses indirection between the disk block and the logical disk block can store more data on less storage than we ever imagined.

And instead of talking about how efficiently we do that, in terms of CPU/Memory, or effectively in terms of the impact to applications we continue to discuss what is the most pointless of conversations:

If I treat my very sophisticated storage array like a set of big dumb disk drives I bought at the local electronics discount store, what's the usable capacity?

Blech.