Extensible NetApp: Thinking outside of the box

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Understanding WAFL Performance: The F Word

An astute reader of my series would have observed that WAFL is able to achieve its surreal write performance because WAFL does not guarantee that two disk blocks are sequential even if the application writer assumed they were sequential. So for example, if the application thinks that it’s writing two disk blocks in sequential order, they may not appear sequentially on disk.

In many quarters this is perceived as violating an implicit contract between the storage array and the client of the storage array. And in many ways, this is at the heart of the Real vs Better Than Real FiberChannel debate.

The core question is: does this matter?

So let me, right off the bat, say: of course it does. If an application goes to the trouble to assume that two disk blocks are sequentially located on disk, then it’s assuming a certain kind of performance when it requests the data. If the storage array turns that sequential operation into several random operations, then the application performance will degrade. This is just a fact of life.

Case closed, then. WAFL sucks.

Not so fast there storage architect.

Let me first observe, that the traditional storage arrays are able to transform random write operations into sequential operations through the use of clever caching and architectures that rely on global shared memory.

In a similar vein, it is possible to do the same thing for read operations, you can apply clever algorithms to make what is a fragmented on disk layout, appear to be sequential by being clever.

Making this concrete

Suppose we have an application that writes out a file in the following order:

Where the data A was at offset 0, and B was sequentially located after A in terms of offset.

The application then has the expectation that if it operates on the file by sequentially accessing A, B and C, that it will get sequential performance.

On the other hand the application has the expectation that if it operations on the file by accessing A, D, F, that the performance will be that of random disk IO.

Last week I showed this picture that showed how WAFL did write operations:

Now let me show a different picture that shows how disk blocks could get laid out by WAFL.

The light green blocks represent free blocks. The blocks marked P are parity blocks. The blocks that are not green and have the letters A, B, C, D, E, F, G etc represent allocated blocks. A single allocated file on this set of disks is represented by a sequence of blocks share the same color. In this example we have four files.

In this example, a sequential read operation would start at a specific row, and then read the blocks in a column starting from the row. A sequential read for example would be K1, L1 and M1.

So now if we consider an application that wants to read sequentially, you can imagine how this turns into a sequence of random read operations.

For example suppose the applications wants to read blocks A, B, C and D of the blue file and then wants to read E, F and G. A simplistic implementation would do one random read to read blocks A and B, another random read for block C, and a third random read for D. A clever implementation however, would recognize that A, B, C and D although not sequentially located on disk, can be read sequentially if you willing to skip over uninteresting disk blocks. A clever implementation would also perform read-ahead to read blocks E, F and G, such that the request for the next set of blocks came from memory rather than from disk, minimizing the total IO to disk.

To be fair, constructing such algorithms that work well in general is a hard problem. As I said earlier, the relatively simple problem of a single processor scheduler that has been studied for almost 40 years still produces new insights, this is a relatively new problem and we’re still learning.

But here’s where things work in our favor. All of these algorithms benefit from increasing compute power because you can spend more time thinking, and increasing memory so you look further into the future.  Both of those trends have worked in our favor over the last 15 years.

CPU performance has increased so much faster than disk drive performance, that we are able to perform very sophisticated computations before we do any disk operations. Memory capacity has increased dramatically as well allowing us to store more things in memory before we need to actually perform an operation.

But it’s more than just hardware. It’s also having a brilliant engineering team that has been focused on solving this problem for almost fifteen years. And no amount of hardware can replicate that experience.

But don’t trust me trust benchmarks that have validated this approach. Alex McDonald in his blog shows how we were able to sustain our performance even if the face of daunting random IO.

But why does this matter, in general?

Fragmentation of the on-disk layout is not just a property of WAFL, it also is a property of things like deduplication, thin provisioning and Real Snapshots. And the reality is making those things perform well requires the same kind of algorithms and sophistication that you need to make Better Than Real FiberChannel work.

The good news is that we’ve been working on those problems for a long time, which is why our benchmarks always have used snapshots and enabled thin provisioning.

Oh that might be the reason our competitors don’t have those things enabled and recommended turning them off when you need good performance...

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

Backup, the new storage tiers and real snapshots

When I first described my new terminology for storage tiers, Tim Burlowski, asked how backup and recovery fit into that model.

It turns out that that is an interesting question.

Backup, the only working ILM/HSM solution

Step 10 feet away, and you realize that backup is HSM.

The backup process creates cold data, the backup image. The backup software them moves the backup copy to cheaper tier of storage. When a restore is performed, the backup software moves the backup copy from the cheaper tier to a more expensive tier.

The question on the table, then, is where does this form of HSM still make sense?

Captive IOPS

One obvious place where HSM as part of backup process is for the backup of data on a Captive IOPS tier. The tier is built using very expensive storage. Storing cold data that will probably never get accessed seems odd and a waste of money.

But that's not necessarily true.

If the Captive tier is built using disk drives, and the application  has a high IOPS data density, a lot of the capacity of the disks is  unused. The cheapest place to store backup copies, in that case, would be on the same disks you're using for primary data.

If the Captive tier is built using SSD's like EMC storage, then it makes a lot of sense to use a backup solution to move backup images off of the flash onto cheaper storage. If the application has a low IOPS data density, then a better approach is to use flash cache in front of disk.

If the Captive tier is built using a flash cache in front of disk, the cheapest place to store the backup copies may be on the disk drives.

More generally, the backup methodology of a captive tier made up of flash cache and disk, will be the same as the backup methodology of the capacity tier.

So what about the capacity tier and backup?

In the general case, the capacity tier will be backed up much in the same way that it always has been.

A capacity tier that has real snapshots can eliminate the HSM part of the backup process.

In fact, in an earlier series on backup I proved that for a VMware infrastructure, the cheapest place to store backup copies was on the same disks that stored the active data.

Now I want to generalize that comment for all storage..

If you consider IOPS density, disk drives must contain large amounts of cold data. A subset of the cold data in a data center is the backup data. That data can either be stored on a separate set of disks or on the same disks that are being used to serve IOPS.

To store the data on the same disks the following things need to be true

1. Creating the backup image must consume minimal IOPS
2. The existence of the backup image must impose no performance penalty
3. Be able to store a large number of images.
4. The storage must be configured using some form of RAID-6 or RAID 1-0
5. The cost of the backup images must be cost effective

It turns out that for NetApp systems all five are true.

So as disk capacities expand, storage architects who have the option to use real snapshots will stop moving data off of the capacity tier to some cheaper tier.

Said in a slightly more poetic way:

The final resting place for data on disk will be the disk where the data first got created.

The net effect will be a much simpler and more cost effective storage infrastructure to support backup.

One minor addendunm added after I wrote this post.

As Martin G says in the comments below, having the backup copies on the local disks doesn't protect you from site failures. So, of course, some form of storage replication is required. Thankfully NetApp has a cool technology called SnapMirror that replicates all of the snapshots to a remote destination in a space efficient way.

In an earlier post on VMware and backup I show how all of this fits together.

I should have included this need to replicate the data in my original blog post. So I am rectifying that error now.

The principle of uniform resource latency

Just before I commented on Karl Dohm's quixotic quest to prove the NetApp sales team were snake oil salesmen, I was talking about IOPS data density.

In that post, I threw out the following statement:

I'll also state that applications expect consistent performance rather than variable performance and perform worse as variability increases.

This is a very important and fundamental concept and underlies a lot of how I think about Flash and storage tiers will shake out. So I will spend a little bit of time on this point.

So let's begin at the beginning.

Within a company, which is a conglomeration of humans working to a single set of goals, consistency is very prized. Consistency makes long term planning possible. Variability makes long term planning impossible.

The software systems that exist to support human beings must, therefore, fit into a world where people expect consistency and predictability.

As a result, software solutions have been designed and architected around providing consistent behavior.

Over time, software architectures have evolved to the point where some layer of software, typically the application, expects all the resources it uses to have the same performance.

What do I mean?

Well an application that uses a database is designed around the assumption that regardless of what field in the database is accessed, there is some maximum time the database will take to respond. The application is then designed to work given that maximum. When the application gets better behavior from the database, the user of the application is pleasantly surprised, but at no time does the user of the application experience worse performance than the expected maximum time.

In practice what this means is that application authors expects the underlying software (database, operating system, file system, storage, server, network) to provide uniform resource access to the resources they manage. Vast amounts of innovation are invested in software to provide a sand-box that provides the illusion of performance uniformity.

From a narrow and simplistic storage system point-of-view, this implies that along with IOPS there is an expectation of uniform latency, which is why benchmarks like SPC and SFS measure latency. And just in case I forget to say this, providing uniformly good latency is not just a SAN thing, but is also a NAS thing too…

And why all this effort?

Because of the simplicity of programming model. To provide that simplified model, a significant amount of complexity and cost are introduced throughout the infrastructure, but as an industry we have chosen to make that tradeoff.

So how to provide uniform resource access?

At one extreme, Seymour Cray remarked You can't fake what you don't have, and designed a system to provide uniformly excellent performance. At the other extreme you have the web which promises nothing.

What's important to note is that between spending a lot of money, and doing nothing there are lot of well established in-between places, also known as caching, that make it possible to provide on average excellent latency at a fraction of the worst case cost.

In my next in this series, I'll consider, and this time I promise, Flash and the new storage tiers.

IOPS data density and latency

As I started to write about flash and storage tiers, I realized there were some basic underlying concepts that were not clear in my head. So I will first try to clarify those before I get to flash.

Earlier I wrote about the IOPS density of spinning media. The point of the post was this graph:

What this graph shows, other than my inability to use SnagIt to correctly cut-n-paste an image, is that at some point in time the number of hot bytes of data on a disk drive is negligible compared to the number of cold bytes of data.

After I started writing my series on storage tiers, I realized that there is a corresponding notion of IOPS density, but this time for the data not for the media.

So I am going to define a new term:

IOPS data density: The ratio of hot vs cold data for any dataset at any given point in time.

For some datasets, for example, home directories, the IOPS data density is quite low. If you read the paper Meaurement and Analysis of Large Scale File System Workloads by Andrew Leung et al, you'll notice in section 4.2 that the amount of data that is transferred between the client and the server is a fraction of the overall amount of data that is available.

The problem with the term IOPS data density is that as defined, it does not capture the notion of latency. Users of data don't care about IOPS, they care about the latency (or time) it takes to perform an operation.

I'll state that the performance of an application, where an application may be a database or an NFS client,  is related to the average performance of the average IO. I'll also state that applications expect consistent performance rather than variable performance and perform worse as variability increases.

So to expand my definition a little bit:

IOPS data density: The ratio of hot vs cold data for any dataset at any point in time. The application using the dataset expects the hot data to have a sufficiently low latency that is reasonably uniform across all of the hot data.

Next we explore the different kinds of IOPS data density and how they map to the different storage tiers.

IOPS density

Some time ago, I started to mumble about big O notation. And then I got distracted by Chuck and then WAFL, but I still wanted to get back to that topic.

So one of the more interesting ratios in the storage industry is the ratio between IOPS, and storage capacity, also called IOPS Density.

First let us define some terms:

• IOPS = Disk Read/Write Per Second
• Capacity = Capacity
• IOPS Density = IOPS/Capacity

But what does IOPS density measure? My opinion is that it measures the amount of hot data a storage system can sustain. The larger the ratio, the more hot-data, the more applications that can be supported by the storage system.

An element of tribal disk drive knowledge is that spindle capacity is doubling and IOPS per spindle are growing linearly or n/2^n.

So one of the implications of that tribal knowledge is that we are staring at a stunning collapse of IOPS density on a per-spindle basis:

And of course this curve is divorced of actual measurements and actual data, it's just an interesting observation.

If we look at that graph some more

the area under the graph is the total amount of hot data.

Now if we look at time step 10:

At time step 10, disks are adding capacity without effectively adding any IOPS. Which means that the capacity had better be storing cold data.

Which is a pretty way of saying that for every time step after 10, the ratio really of IOPS to Capacity is 1/2^n, which means IOPS density is 0.

Interesting, huh?