Extensible NetApp: Thinking outside of the box

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.

Understanding WAFL Performance: The F Word

An astute reader of my series would have observed that WAFL is able to achieve its surreal write performance because WAFL does not guarantee that two disk blocks are sequential even if the application writer assumed they were sequential. So for example, if the application thinks that it’s writing two disk blocks in sequential order, they may not appear sequentially on disk.

In many quarters this is perceived as violating an implicit contract between the storage array and the client of the storage array. And in many ways, this is at the heart of the Real vs Better Than Real FiberChannel debate.

The core question is: does this matter?

So let me, right off the bat, say: of course it does. If an application goes to the trouble to assume that two disk blocks are sequentially located on disk, then it’s assuming a certain kind of performance when it requests the data. If the storage array turns that sequential operation into several random operations, then the application performance will degrade. This is just a fact of life.

Case closed, then. WAFL sucks.

Not so fast there storage architect.

Let me first observe, that the traditional storage arrays are able to transform random write operations into sequential operations through the use of clever caching and architectures that rely on global shared memory.

In a similar vein, it is possible to do the same thing for read operations, you can apply clever algorithms to make what is a fragmented on disk layout, appear to be sequential by being clever.

Making this concrete

Suppose we have an application that writes out a file in the following order:

Where the data A was at offset 0, and B was sequentially located after A in terms of offset.

The application then has the expectation that if it operates on the file by sequentially accessing A, B and C, that it will get sequential performance.

On the other hand the application has the expectation that if it operations on the file by accessing A, D, F, that the performance will be that of random disk IO.

Last week I showed this picture that showed how WAFL did write operations:

Now let me show a different picture that shows how disk blocks could get laid out by WAFL.

The light green blocks represent free blocks. The blocks marked P are parity blocks. The blocks that are not green and have the letters A, B, C, D, E, F, G etc represent allocated blocks. A single allocated file on this set of disks is represented by a sequence of blocks share the same color. In this example we have four files.

In this example, a sequential read operation would start at a specific row, and then read the blocks in a column starting from the row. A sequential read for example would be K1, L1 and M1.

So now if we consider an application that wants to read sequentially, you can imagine how this turns into a sequence of random read operations.

For example suppose the applications wants to read blocks A, B, C and D of the blue file and then wants to read E, F and G. A simplistic implementation would do one random read to read blocks A and B, another random read for block C, and a third random read for D. A clever implementation however, would recognize that A, B, C and D although not sequentially located on disk, can be read sequentially if you willing to skip over uninteresting disk blocks. A clever implementation would also perform read-ahead to read blocks E, F and G, such that the request for the next set of blocks came from memory rather than from disk, minimizing the total IO to disk.

To be fair, constructing such algorithms that work well in general is a hard problem. As I said earlier, the relatively simple problem of a single processor scheduler that has been studied for almost 40 years still produces new insights, this is a relatively new problem and we’re still learning.

But here’s where things work in our favor. All of these algorithms benefit from increasing compute power because you can spend more time thinking, and increasing memory so you look further into the future.  Both of those trends have worked in our favor over the last 15 years.

CPU performance has increased so much faster than disk drive performance, that we are able to perform very sophisticated computations before we do any disk operations. Memory capacity has increased dramatically as well allowing us to store more things in memory before we need to actually perform an operation.

But it’s more than just hardware. It’s also having a brilliant engineering team that has been focused on solving this problem for almost fifteen years. And no amount of hardware can replicate that experience.

But don’t trust me trust benchmarks that have validated this approach. Alex McDonald in his blog shows how we were able to sustain our performance even if the face of daunting random IO.

But why does this matter, in general?

Fragmentation of the on-disk layout is not just a property of WAFL, it also is a property of things like deduplication, thin provisioning and Real Snapshots. And the reality is making those things perform well requires the same kind of algorithms and sophistication that you need to make Better Than Real FiberChannel work.

The good news is that we’ve been working on those problems for a long time, which is why our benchmarks always have used snapshots and enabled thin provisioning.

Oh that might be the reason our competitors don’t have those things enabled and recommended turning them off when you need good performance...

The one good (storage virtualization) and the one bad (fine grained HSM) idea of every recession

In a recession, the storage group like every corporate function is asked to do more with less. One of the particular challenges with storage is that once a byte is consumed, it's hard to unconsume it. You can't just delete disk blocks willy nilly.

So you take a deep breath

One approach is to look at the stuff you already have and see if you can make better use of it. For example, when times are good we don't ask our users to delete unnecessary data, when times are bad we send out "spring cleaning" emails, and crunch down quotas. And all of that works but to a point. This is the moral equivalent of holding your breath. It helps, but it is unsustainable.

And then you try to optimize

Once it's clear that you can no longer hold your breath you need to start looking at more system approaches to doing more with less. And that's when things like storage tiering get revisited. When you were flush, everybody got better storage than they needed. When you're short on cash, and you still need to grow your storage capacity it makes a lot of sense to re-examine every application and move things around to optimize for the right cost/performance/availability tier of each application.

And in that maelstorm of applications being moved around, the ugly step-child of storage, fine-grained hierarchical storage management (HSM), makes it's appearance. Fine-grained HSM is storage tiering taken to it's absurd conclusion. If moving large blobs of data to the right tier is good, then moving small files around must be better! And if moving large blobs of data transparently is a good idea, then moving small files transparently must be all that better!

And because re-configuration of your storage is hard, and because HSM is impossible, storage and NAS virtualization appliances make their appearance. The basic idea is that by inserting some kind of layer of indirection between the host and the storage, a virtualization device allows transparent migration of data between the storage tiers. And if you do enough migration and move things around just so the net effect will be an optimized storage infrastructure!

The other benefit the storage virtualization gang tout is that by using this new layer you are able to unify disparate storage islands in your infrastructure into bigger storage pools. Bigger storage pools should create better efficiences. Right?

Except that the market for storage virtualization has failed. The market failed because if you're inserting a bump-in-the-wire you better be adding enough value to over-come the market's aversion to bumps-in-the-wire (and if you want to understand what I said you might find this one-two-three-four part series interesting ). On top of which, the bump's in the wire were being inserted by companies that were unable to provide the right level of support to make their new infrastructure work.

And so along with the storage virtualization appliances, the term went down in flames.

Which is unfortunate, because storage virtualization is the only systemic way to actually reduce your storage costs. Storage virtualization, a term hijacked by products whose only function in life appears to be to simplify migration, is a technique that allows you to reduce the amount of storage you consume by exploiting data redundancy through indirection. Storage virtualization, furthermore, allows you to separate the logical storage container from the physical infrastructure the storage container is actually stored on simplify data migration.

Enter Better Than Real FiberChannel

So for example, WAFL, NetApp's storage virtualization layer, allows us to create space-efficient clones, eliminate data redundancy through dedup and take snapshots that are performance neutral, space efficient and plentiful. Furthermore, WAFL allows us to replicate data between different physical infrastructures (SATA to FC, low-to-high and back again). In fact, the unique capabilities that WAFL has are what lead to our last promotion.

If you choose to use Better Than Real FiberChannel you can systemically move to a lower cost basis for your storage.

The problem for most Real FC vendors and ultimatley their customers, is that because Real FC vendors consider storage virtualization to be a dirty word (or at least a dirty unsavory and performance killing layer of indirection, benchmarks be damned) they can not exploit the information in the array the way WAFL does. The net effect is more storage being poorly utilized.

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

Progress of a kind.

When I wrote my series on WAFL, I was hoping to cure the storage industry of a virulent form of Fear, Uncertainty, and Doubt  (FUD) centered around WAFL performance.

The story, typically, goes like this:

1. WAFL is a file system and hence not a real storage system
2. WAFL performs great but over time the performance degrades
3. When it degrades it gets much worse than everyone else's
4. And worse there are all these latency spikes
5. Because WAFL is a file system (cue spooky music!)

I figured, if I killed point 1, then the rest of this FUD would die...

So imagine my surprise when i read Karl'sblog on WAFL

What I didn't count on was the virulence of this FUD.

Instead of ridding the industry of the FUD, the FUD changed into:

1. WAFL is this unique thing that nobody understands
2. WAFL performs great but over time the performance degrades
3. When it degrades it gets much worse than everyone else's
4. And worse there are all these latency spikes
5. Because WAFL is this thing nobody understands

Well, that's progress of a kind. 

Previously the argument would have begun with an appeal to general misunderstandings of how SAN and WAFL worked together. Now the argument is based on an appeal to the basic lack of information about our system works, that allows the author to argue whatever point they want, and then leverage a synthetic benchmark that proves whatever the author wants.

But back to Karl and his blog...

Karl was making the following bold assertions based on his research:

FAS systems not suitable for Exchnage

But of course, Avanade's white paper would disagree.

FAS systems perform poorly over time

But our SPC-1 longevity results would differ

A large number of NetApp customers have been tricked by our sales force

After all, that is the only possible explanation for our market share given how inappropriate our system is for the kinds of workloads we've been selling quite well into.

The problem that Karl has is that without appealing to some basic architectural understanding that we can all share, he is forced to create synthetic workloads that prove his point. And the problem is that there are real workloads with real data that prove the opposite point.

Removing a document does not change the facts

Unfortunately EMC has chosen to remove the white paper I talked about

Deleting said white paper does not change the facts.

The Clariion has fake snapshots. They are fake because they are not:

- performance neutral

- space and time efficient

Using Clariion snapshots storage architects have to choose between getting there on time and driving sober. NetApp's real snapshots, on the other hand, allow storage architects to offer to their application customers

- Lower RPO because you can take more snapshots with low space utilization

- Lower RTO because less has to be moved

without having to choose between performance and data protection. 

No  amount of document management on EMC's web site can change those facts.

Since you can no longer get an EMC document to tell you the truth about performance crippling Clariion snapshots, I suggest you go to the SPC web site where you can see the data behind this chart:

Using Clariion's fake snapshots is performance crippling and not recommended for workloads where performance is important.

Deleting documents does not hide the truth.

A Reminder of the Value of REAL Snapshots

One of the more irritating aspects about being a technologist instead of a marketing professional is that I have to deal with the marketing folks of other companies. I don't have to deal with them personally, of course, but I do have to deal with their misappropriation of technology terms.

And the most irritating misappropriation is the use of the term snapshot to describe any technology that creates a point-in-time copy of a range of disk blocks.

Ideally, I wouldn't have to write this blog entry, because every storage customer out there would understand that a snapshot was, by definition:

- space efficient

- performance neutral

And that, therefore, using storage snapshots a customer would have:

- improved RPO because they can have more copies of data

- improved RTO because they can move less data on a restore

- no need to trade off RPO/RTO against performance of the system

But as I am reminded again and again, and today by no less of an authority than EMC's own white paper, if you are using a Clariion a snapshot is

- performance crippling.

As EMC says on page 8:

For Exchange, snapshot replication on Clariion is not recommended because it can impact performance.

As a storage architect you are forced to choose between your recovery time and application performance, which is kind of like choosing between arriving on time and driving sober.

So if you happen to be an EMC customer, or unfamiliar with real snapshots you may find the rest of this post interesting.

First a history lesson

When NetApp's founders invented WAFL they created a unique mechanism for creating a point-in-time copy of a collection of blocks that was constant-time with respect to the number of blocks, and extraordinarily space efficient. They chose to call that technology a snapshot.

What was true then, remains true today, Snapshot do not impose any meaningful overhead on the storage system in terms of performance, and are extremely space efficient.

Net-net...

Using a NetApp Snapshot you can have more recovery points without having to worry about the performance impact on your storage array and because of their space efficiency, not worry about the cost.

But that's not all.

Having snapshots on the primary system can also improve your recovery time.

Why?

Well it turns out, as I've said here, the amount of data that has to be moved is the key determinant in restore performance. And if you have a business critical application the amount of time you are down waiting for a restore to be completed can be measured in real dollars.

So if the data is on the primary storage system, and the storage vendor has a product like SnapManager for Exchange restore times can be extraordinarily fast.

Plowing through the SPC-1 Haze or How to Enjoy Your Dessert

I fully expect to see over the next month a vast and voluminous debate within the blogosphere about the validity, or not, of SPC-1, of NetApp's numbers or of EMC's numbers.

I am, anxiously, anticipating the point by point dissection of the data. There will be tits and there will be tats from both sides. And the press will get involved.

But that's not what I found intriguing about NetApp's results.

What I found intriguing was that, finally, you could eat your cake and have it too.

There is a presumption, and a reasonably valid one, that you have two choices:

Virtualization or Performance

Given any piece of hardware, if you have more instructions to execute you must, by definition be slower. And since NetApp has more virtualization it must perform worse.

But if you are a data center operator, what EMC is really saying is the following:

You should be willing to ignore the simplicity, ease-of-use, that NetApp storage brings to the table, because you know that their performance must be worse than ours because we have less virtualization.

After all, virtualization, must make things worse.

But what does this mean in practical terms?

Let's consider snapshots.

The beauty of NetApp snapshots is that their performance impact is minimal, and that means that you can keep them on your primary storage system.

But why is that important? Because if you have to do a recovery (you know restore from a backup) you don't have to copy a lot of data. And the amount of data that has to be moved is the single biggest factor in any recovery situation.

Now if you believe the EMC FUD about this benchmark result, you will be told that virtualization and, therefore, NetApp snapshots must affect performance. And let's assume that you do believe them.

What EMC is really trying to say is the following:

Hey, Mr. Overworked Storage Architect, you can trade off performance during normal operations for crappy recovery times if you buy our stuff vs crappy performance during normal operations with astounding recovery times if you buy their stuff.

And so now you have to sit in a room and figure out the following:

For which applications do I care about recovery times and for which do I care about operational performance?

And if you make a mistake, you might find yourself in a world of hurt. Either because the operational performance is worse than expected, or some recovery is taking so long that it is affecting the business...

What we're saying is that NetApp storage eliminates that debate. And that's why our systems are simpler. Because you can have the performance you need, and the recovery times you require.

And that is why this SPC-1 benchmark was so important.