Extensible NetApp: Thinking outside of the box

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.

An update to system management

Steve Klinkner, in his usual understated way, has begun an excellent series on the new system manager
user interface that was just released for our FAS system.

I am, personally, very pleased with the product and am using it to manage my home filer.

I strongly urge folks who are interested to read Steve's posts on this topic.

High Noon in Australia

NetApp’s Australia team has offered to give an EMC customer up to 50 TB of raw storage if they’re willing to migrate their existing vmware environment to NetApp storage. The customer has to go public with the results and regardless of the results gets to keep the storage.

This feels a little bit like a scene in a bad cowboy movie, where one of the gun-slingers frustrated with the trash-talking of some other gunslinger in the saloon says: Let’s settle this outside at high-noon.

What I found interesting about this marketing activity is that it goes to the heart of what I love about NetApp’s marketing. We’re not about spreading FUD and lies or name calling. We’re about building our success on top of our customer’s success. Which may be why we choose to participate in standard benchmarks rather than proclaim how useless they are.

At the end of the day the activity will prove that we were not stacking the deck unfairly when we compared RAID-DP to RAID-10, that the performance is good enough, that it’s not too hard to make work, and guess what you do save storage.

Understanding WAFL Performance: The F Word

An astute reader of my series would have observed that WAFL is able to achieve its surreal write performance because WAFL does not guarantee that two disk blocks are sequential even if the application writer assumed they were sequential. So for example, if the application thinks that it’s writing two disk blocks in sequential order, they may not appear sequentially on disk.

In many quarters this is perceived as violating an implicit contract between the storage array and the client of the storage array. And in many ways, this is at the heart of the Real vs Better Than Real FiberChannel debate.

The core question is: does this matter?

So let me, right off the bat, say: of course it does. If an application goes to the trouble to assume that two disk blocks are sequentially located on disk, then it’s assuming a certain kind of performance when it requests the data. If the storage array turns that sequential operation into several random operations, then the application performance will degrade. This is just a fact of life.

Case closed, then. WAFL sucks.

Not so fast there storage architect.

Let me first observe, that the traditional storage arrays are able to transform random write operations into sequential operations through the use of clever caching and architectures that rely on global shared memory.

In a similar vein, it is possible to do the same thing for read operations, you can apply clever algorithms to make what is a fragmented on disk layout, appear to be sequential by being clever.

Making this concrete

Suppose we have an application that writes out a file in the following order:

Where the data A was at offset 0, and B was sequentially located after A in terms of offset.

The application then has the expectation that if it operates on the file by sequentially accessing A, B and C, that it will get sequential performance.

On the other hand the application has the expectation that if it operations on the file by accessing A, D, F, that the performance will be that of random disk IO.

Last week I showed this picture that showed how WAFL did write operations:

Now let me show a different picture that shows how disk blocks could get laid out by WAFL.

The light green blocks represent free blocks. The blocks marked P are parity blocks. The blocks that are not green and have the letters A, B, C, D, E, F, G etc represent allocated blocks. A single allocated file on this set of disks is represented by a sequence of blocks share the same color. In this example we have four files.

In this example, a sequential read operation would start at a specific row, and then read the blocks in a column starting from the row. A sequential read for example would be K1, L1 and M1.

So now if we consider an application that wants to read sequentially, you can imagine how this turns into a sequence of random read operations.

For example suppose the applications wants to read blocks A, B, C and D of the blue file and then wants to read E, F and G. A simplistic implementation would do one random read to read blocks A and B, another random read for block C, and a third random read for D. A clever implementation however, would recognize that A, B, C and D although not sequentially located on disk, can be read sequentially if you willing to skip over uninteresting disk blocks. A clever implementation would also perform read-ahead to read blocks E, F and G, such that the request for the next set of blocks came from memory rather than from disk, minimizing the total IO to disk.

To be fair, constructing such algorithms that work well in general is a hard problem. As I said earlier, the relatively simple problem of a single processor scheduler that has been studied for almost 40 years still produces new insights, this is a relatively new problem and we’re still learning.

But here’s where things work in our favor. All of these algorithms benefit from increasing compute power because you can spend more time thinking, and increasing memory so you look further into the future.  Both of those trends have worked in our favor over the last 15 years.

CPU performance has increased so much faster than disk drive performance, that we are able to perform very sophisticated computations before we do any disk operations. Memory capacity has increased dramatically as well allowing us to store more things in memory before we need to actually perform an operation.

But it’s more than just hardware. It’s also having a brilliant engineering team that has been focused on solving this problem for almost fifteen years. And no amount of hardware can replicate that experience.

But don’t trust me trust benchmarks that have validated this approach. Alex McDonald in his blog shows how we were able to sustain our performance even if the face of daunting random IO.

But why does this matter, in general?

Fragmentation of the on-disk layout is not just a property of WAFL, it also is a property of things like deduplication, thin provisioning and Real Snapshots. And the reality is making those things perform well requires the same kind of algorithms and sophistication that you need to make Better Than Real FiberChannel work.

The good news is that we’ve been working on those problems for a long time, which is why our benchmarks always have used snapshots and enabled thin provisioning.

Oh that might be the reason our competitors don’t have those things enabled and recommended turning them off when you need good performance...