Extensible NetApp: Thinking outside of the box

Almost famous

NetApp recently released a corporate video.

And at the very end of the video, you can see me wearing my red t-shirt and hat. I am part of a larger group greeting a customer who thanks to NetApp has navigated through the realm of data complexity.

Image vs Logical Backup

What I love about mismanaging my own IT shop (my house), is that I learn first hand what not to do.

So for example, a logical backup scheme (backup incremental files according to some schedule) is great if you want to do a logical restore (restore a single) but sucks wind if you want to do a physical restore.

And yes, I knew that.

So for my laptop I rely on a image based backup utility because I don't really care about the data on the laptop as I keep all of my important data on a Filer.

This does make single file restore painful, but do-able.

Unfortunately, the other day, my wife really needed an image to do a full image restore of her laptop. And unfortunately, the clever logical scheme she was relying on got FUBAR'ed making it impossible to do a restore.

So now we have two backup strategies for her laptop: a logical scheme so she can recover individual files, and a physical schemes so she can recover her entire hard drive quickly. The theory being that any file she needs can be recovered, and she can be back on her feet working without having to wait for some logical scheme to try and piece together all of the files that need to be restored.

The problem with this approach is that it doubles the cost of our storage, never mind the complexity of having two distinct approaches to data protection.

Which reminds me again, why I love snapshots. A snapshot is an image based backup, but it allows you to browse the backup as if it were a logical image, providing logical restore level granularity. But thanks to the magic of SnapRestore, provides a constant time technology that allows you to restore all of the data all at once.

What happens when it rains in the clouds?

The recent data loss event at Carbonite, discussed here by the Storage Architect got me thinking about the cloud and cost.

What I learned, and continue to learn is that in the data protection space, it's not about cost, it's about insurance. And insurance like everything suffers from the dictum you get what you paid for. And the problem I struggle with is if the data loss is worth several millions of dollars to your business, when it happens, is taking any chances worth while?

Of course it is, because not every piece of data is worth millions of dollars.

Not everyone can afford the best insurance policies, not every thing needs the best insurance policies, and having an insurance policy is better than none. In the market Carbonite services, the reality is that most people don't have an off site backup strategy for most of their data. And if they do it relies on too much human intervention.

So is Carbonite flawless? No, but then again no storage or data protection infrastructure is flawless. And so you kind of do the actuary analysis and figure out how much are you willing to risk. And if the cost of loss is within the limits of what you can tolerate go for it.

To me the question of cloud computing vs your own data center is really a question of core competence. And that really boils down to who will have the best average staff. Or more pedantically, for any particular application whose staff can do a better job managing the application, the in-house or cloud team.

Average Staff

So in my house, the staff that maintains my IT infrastructure is me. And although I have a high opinion of my technical skills, I have a very poor opinion of my ability to set up and manage my infrastructure.

So in fact, I would be willing to outsource some functions to the cloud, because I figure that they will be able to do a better job of managing the infrastructure than I will. What I am betting is that although the other guy will make mistakes, he will make fewer mistakes than I will. I would pay serious money to someone to manage my network because I hate doing it.

But I am not willing to outsource all functions. For example, I do not want someone other than me to maintain my desktop. I like the setup, I like the tools and I like the configuration. In fact, I believe I am better staff than any third party for managing the desktop.

What the hell does this have to do about the rain?

Cloud providers will have failures, and because they are publicly traded companies, they will have to proclaim those failures to the world. But I wonder if those failures are like the rain? They give us something to talk about, but at the end of the day, it's just not that important. And it's not that important, because for the most part, we're all rationale human beings who can do cost/benefit analysis, and on average make the right choices. Sure some people will get screwed, especially those that can not make the right trade-offs, but they would have screwed up some other way anyways.

Something to think about as we talk about the rain.

Fixed a broken link to Chris Evan's blog. Pointed out by one of my readers.

Understanding WAFL Performance: How RAID changes the performance game.

In my last post I explained why PAM improves performance, but I didn’t answer the other question of why WAFL kicked ass at low utilization, and more interestingly why is the PAM a read-only cache?

A number of our competitors have correctly pointed out that their cache’s are read-write and have asked, fairly, is our cache the wrong kind of cache?

To answer the question I need to talk about RAID, something I dutifully ignored in my last set of posts.

Understanding how RAID affects storage performance

This is not a primer on RAID but rather an attempt to explain how designing a high performance storage system is impacted by RAID.

Yesterday I pointed out that a Real FiberChannel RAID array can defer writes in memory such that the writes are sequential. I then said that WAFL is different, but what I described and showed didn’t seem that different. So what is different?

When you’re writing data to disk using RAID, it’s not sufficient to just write a disk block to a specific location, but you must also update the RAID parity information to allow reconstruct to take place.

So let’s assume you have this picture:

Each column represents a disk drive, and each row represents a RAID stripe.

The application sending writes to the array will force the Real FiberChannel array to update locations A1, B3, D2, D4, F3 and H1.

However if you modify location A1, you also have to modify location AP, ditto for BP, DP, FP, and HP. But because AP etc are parity computations for the whole stripe to modify them you have to first read them.

So write updates A1, B3, D2, D4, F3 and H1 really turn into 6 reads and 12 writes.

Alright, but if we look at this a little bit more carefully, because of the write cache, I don’t really need 6 reads but 5 reads. Consider stripe D, since I know D2 and D3 are updating the same stripe I can only do one read for the entire stripe.

Now that’s why write cache is good!

The more writes the storage array defers, the more likely the storage array get writes to the same stripe, the fewer additional reads the storage array will end up doing.

But WAFL is better, after all it’s Better Than Real FiberChannel

WAFL relaxes the requirement that Real FiberChannel imposes that an update to the same logical disk block must be to the same physical disk block.

So for WAFL here’s what the would happen.

Suppose the original layout looked liked this

 

Then a sequence of updates to A1, B3, D2, D4, E3, F3, G3 and H1 happen. What WAFL will do is instead of over-writing the same blocks it will write the data as a full stripe in I1 and J2 and then free up the original blocks. So for WAFL this sequence of 8 writes would appear as one write where you were sequentially writing more data.

The net layout will look like this:

So that’s why WAFL doesn’t need as much write cache

Unlike Real FiberChannel, WAFL only has to wait for enough data to write out a single stripe to disk. If there is enough free space, WAFL’s write performance is constrained by Data ONTAP’s sequential write throughput.

So that’s why WAFL performance kick’s ass!

A Real FiberChannel array has to work harder than WAFL (more disk IO’s) when there is a lot of free space. That’s why a Real FiberChannel array will benefit from additional write cache. However, there is a limit to how much you can cache, and ultimately you will end up doing more work.

A Better Than Real FiberChannel array like WAFL, will do very little work when the array has plenty of free space and therefore perform much better. A side effect, though, is that as the array free space shrinks, WAFL’s will have to do more work to find free space, and that more work will translate into lower performance. This is why as the storage utilization increases WAFL performance reaches the same level as other storage arrays.

WAFL Performance: Making writes go faster with fewer IOPS

Like every storage array, Data ONTAP attempts to make the latency of the average read and write operation be lower than the actual latency of the underlying disk drive.

Unlike Real FiberChannel systems, how Data ONTAP does achieves this is quite unique.

The Latency Challenge

If you’re a storage array life looks like a set of random read and write operations that you have to immediately respond to. The temptation is to take a request and immediately go to do disk to get it as fast as you possibly can.

But that’s kind-a-stupid.

When you look at read operations you realize that they tend to be clustered around the same piece of data. So although an application may be asking for one 4k block of data, the reality is that the next piece of data it will ask for is probably sequentially next to the first 4k block of data, or is in fact the same 4k block of data!  Given that, it makes sense to do two things:

1. Keep the 4k around in memory
2. Get more than 4k when you go to disk so that the next request is serviced from memory and not disk.

In effect by using memory you can reduce what would have been three IOPS into one actual disk IO.

Write operations are a little bit more complicated. Obviously when you acknowledge a write you’re making a promise to the application that the data is committed to stable storage. Superficially this would imply that every write operation has to go to disk since that’s the only available non-volatile storage.

In this picture each box represents a distinct write operations. The letters designate the order in which they were received. If the storage array were to write to the data to disk as they arrived, then the storage array would perform 8 IOs. Furthermore, the latency of each IO would be the latency of the disk drive itself.

Enter battery backed memory…

Except it turns out that we can turn DRAM into non-volatile memory if we’re willing to keep it powered!

What that means is that once the data is in memory, and the storage array is sure that data will eventually make to disk, the write can be acknowledged. Now that the write operations are in memory it’s possible to schedule the write operations so that they are done sequentially.

In this picture the order in which the write operations are committed to disk is a->b->h->i->e->f->c->d->g even though the order that the application thinks they were committed to disk is a->b->c->d->e->f->g->h->i. Because the array re-orders the write operations, they can be written in one single IO, effectively a single sequential write operation.

The challenge is that you need to buffer enough write operations to make this efficient.

But WAFL, of course, is different

What I just described is how Real FiberChannel works. Effectively the location of the blocks (a,b,c, etc) are fixed on disk. All the storage array can do is re-order when you write them to their fixed locations.

What WAFL does is determine where the data has to go when the data is ready to be written. So using the same picture:

Rather than trying to put each block (a, b, c, d etc) in some pre-arranged location, WAFL finds the first bit of available space and writes the data sequentially. Like Real FiberChannel, Better than Real Fiber Channel will transform the random write operations into sequential operations reducing the total number of IOPS required.

Now what’s interesting is that WAFL doesn’t require that much memory to buffer the writes. And what’s even more interesting is because the fixed overhead to flush data to disk is negligible, there is no real value in holding onto the data in memory for very long.

Read and Write and IOPS

If you’ve been following this story, you’ll have figured out something very interesting: a write operation can be deferred but a read operation must go to disk. In other words, the array can choose when it goes to disk to commit a write, but if the data is not in memory, the array must go right then and there to get the data from disk!

So what makes a disk behave in a random fashion? Why the read operations, because the writes are always done sequentially!

So why do you need a lot of IOPS? Not to service the writes, because those are being written sequentially, but to service the read operations. The more read operations the more IOPS. In fact if you are doing truly sequential write operations, then you don’t need that many IOPS ….

But it’s read operations to DISK not read operations in general!

Aha! Now I get it…

The reason the PAM card is only a read cache is that adding more memory for writes doesn’t improve WAFL write performance… We’re writing as fast as we can write data sequentially to disk.

But adding the PAM card absorbs more read operations which reduces the number of IOPS that the storage system requires. And the best way to express reduced number of IOPS is requiring both fewer and slower disks!

If you can’t comment because of  an inability to get past the CAPTCHA, try a different browser. I’ve had success with IE. We’re having some problems right now and hopefully they will get fixed soon.

WAFL Performance: Disk Drives, Latency, Throughput and IOPS

Cisco decides to explode the server market, and I want to talk about disk drives. What's wrong with me?

Last week I spent time trying to disentangle notions of randomness and sequential access. And after a long meadering presentation, what I think I landed on was a useful and meaningful definition namely:

A device is sequential if the software accessing the device can detect differences in access time depending on how data is accessed. The sequential access is the fast access, the random access is the slow access

An important consequence of this definition is that because time moves forward for every man, the number of random and sequential accesses that can be performed over any time period are effectively time/T for random access or time/D for sequential access. But disk drives have an interesting property that changes this a little bit. Because of the way disk drives work, to move between two random location you need to, in general, perform a sequential operation, so the sequential operation is in, fact, free once you've done the random operation.

Now it turns out that because of the peculiar relationship between random and sequential operations, the only interesting measure for disk is the number of random operations, and that's what we call IOPS.

Although IOPS are important, another user-visible aspect of disk drives is their latency and throughput. Now the latency of a disk drive is pretty horrible compared to it's throughput. So there is a lot of value if you can hide the latency of a disk drive. 

The core trick to improving latency is to avoid going to disk as much as possible. Imagine if you could read all of the data into memory. Then the latency of the data access is the latency of accessing memory! The problem with storing everything in memory is cost, and the annoying property that memory is volatile, so you have to go to disk.

Now it turns out, that you only have to go to disk when you write data and when data that you need is not in memory.

So now let's consider the picture I am painting for you. A disk drive has a certain number of IOPS. Those IOPS need to be used to write data to disk and to read data from disk that is not in memory. If you can cache more data in memory then you can improve your write throughput because you can have a larger share of the IOPS be for write operations. The interesting side effect, is that as you reduce the number of read operations, you reduce the number of total IOPS you require for a given workload!

So the question is then is if the disk drives are all the same, why does WAFL write performance kick ass? After all given the same amount of memory, and the same workload, why does WAFL perform so well? And why does a relatively small amount of memory allow WAFL to cut the total number of spindles it requires in half and sustain the same throughput and latency?

Next post, promise.

Understanding WAFL Performance: Disk Drives

Disclaimer:

If you've been working in the storage industry, bought storage or in general have a good understanding of storage system design, the rest of this blog entry is really uninteresting. So skip ahead to the next post. If you want to read about disk drive performance and storage system design at a fairly basic level, then read on. This post will serve as a bridge to my follow on post.

The world is really sequential

My first computer was an IBM PC circa 1981. That computer had among many other things a 5.25" floppy drive. And I remember reading that the floppy drive unlike the tape drive was a random access media.

 

But if you looked at the floppy you noticed something really bizarre, that it had this spinning platter that spun like LP on a round table. When I compared it to the tape drives my friends had on their

Commodore 64's I couldn't for the life of me figure out why a tape drive was sequential but a floppy was random access. In both cases you had to move this physical media backwards and forwards.

Confused, perplexed and bewildered, I asked a buddy of my dad's why the floppy was random and the tape was sequential, and his response was simple: speed of access. Of course, I had no idea what he was talking about, I nodded sagely filed away the story in my brain, and went on my merry life.

Some 20+ years later it became fairly obvious what he meant. Something Dave explained in an earlier post, but I'll repeat in my own words.

The pompous definition

Every technology that can be used to store data cost-effectively (memory, disk, tape, flash) has the property that getting some data is faster than getting other data.

More precisely, suppose it takes some time T to seek to a specific location to get some data, if the next piece of data is sequentially located, it will take time D to seek to that location where D is very much less to T.

From the picture on the left, it takes time T to get to the place where the 1 is stored, but it takes time D to read the 0 next to the 1.

A medium appears to be random access if the time it takes for an application to make each request of data is approximately on the order of T. If the time to make a request is T, then the application doesn't care about how the data is layed out and can make any request whenever it wants.

On other hand if the time it takes an application to request data is on the order of D or even less than D, then the medium the data is stored appears to be sequential. The reason it appears to be sequential, is because the application that is accessing the data will change it's requests to access the data to be sequential so as to get better performance. In other words, the way the data will get requested will be a relatively small number of random seeks followed by a much larger number of sequential seeks.

Huh?

Every media has a time it takes to get to where the data is located (seek time), and a time to get the data.  But if you drill down a little bit further, it turns that seeking to a completely different location takes more time than just getting the next piece of data. Using a DVD as an example, playing the next frame frame of the movie takes a lot less time than skipping to the next scene. In both cases you have to seek to get the next frame, just in one case you have to seek for a less time.

So fine.

So given that in both cases there is seek, what makes something sequential and something random? The device asking for the data. Consider a modern processor that can request billions of bits a data second. If the thing feeding the data can not get feed the processor at that rate, then the speed of the processor is defined by the feeder. If the seeks can't get any faster, then you need to minimize the number of seeks. If you minimize the number of seeks, then you're basically operating the device as if it were a sequential medium, because you seek to a specific location and then try and get as much information as fast as possible by reading the next piece of information that is located at the next sequential location.

So what does this have to do with disk drives?

Well it turns out that disk drive performance, today, is so slow compared to processor speeds, that the entire design of high performance storage systems is how to convert random access operations into a set of sequential operations.

In other words, whereas the application wants to seek all over the disk drive, the storage system using a variety of tricks minimizes the amount of disk movement that is necessary.

There are two basic approaches depending on whether you are doing a read or a write.

Read

Most applications exhibit a certain locality of reference. What that means is that the next piece of data they require happens to be close to the previous piece of data. Using that principle, storage systems use large amounts of memory to store data to avoid going to disk, and when they go to disk read more than was requested.

Write

It turns out that how to make write operations sequential is a bit of black magic. And that will be the topic of my next post.

Understanding WAFL Performance: Why does WAFL performance kick ass at low utilization?

Many moons ago during an HP - NetApp blog death match, Alex posted a graph

that showed ONTAP performance degrading as storage utilization increased. The graphic was part of a longer technical report that debunked another EMC "technical" report.

What I found fascinating about the graph wasn’t that WAFL performance degraded over time, but that for very low storage utilization WAFL outperformed the CLARiiON by a factor of two.

The recent brouhaha surrounding the announcement of the PAM card made me want to answer the question: Why does WAFL performance kick ass at low utilization?

After I dug into the answer some it became clear, that the answer to that question was also the answer to other very interesting question: Why does a small amount of DRAM save you so much disk?

What I found really interesting was that although every storage vendor has a lot of cache, because of the way WAFL works, and how disk drives function, the combination of a small amount of DRAM for read caching could have a very dramatic impact on WAFL performance.

A side question that emerged is whether other technologies would derive the same benefit as WAFL.

So in my next series of posts, I intend to try and answer some of those questions.