Extensible NetApp: Thinking outside of the box

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

FAS Deduplication is here for the long haul

The current tussle between NetApp and EMC has, predictably, created a dust-up amongst the corporate blog-o-sphere. After all the storage industry believes in death-match-competitive-blogging, none of this polite, civilized discourse, here we reach for the knives and hack!

And it would be tempting to point out that Chuck and Storagezilla are like Aesop’s fox, but that would be too easy.

However, there is a point of confusion in the storage and analyst community about the relationship between FAS dedup and inline dedup technologies like the one provided by Data Domain. There is some opinions that say that this proposed acquisition is driven by flaws in our technology. There are some customers who may be wondering whether this reflects some flaws in our technology.

So let me state again, what I said earlier:

The rest of the storage industry has adopted inline because they lack NetApp's storage virtualization layer that enables primary deduplication.

But because that confusion is real, I wanted to remind folks about why FAS dedup is unique. And more importantly, why NetApp FAS dedup is different and must be different to meet the market requirements our storage plays in than other inline dedup solutions.

I went over this in a post several months ago in a post titled A little digression on Inline Dedup.

The basic points I made are the following:

1. Deduplication is a compute intensive operation
2. Single core performance is stagnating
3. Latency isn’t something it’s the only thing
4. Backup workloads are latency insensitive but throughput sensitive
5. Primary storage workloads are latency sensitive

And I observed that because of 1, 2 and 5 any inline process is problematic if not deeply flawed.

And then in another post I observed that the important question for dedup isn’t how efficient the dedup process is, but how well do you perform after the dedup process is complete.

Our deduplication process performs well because of it is integrated into our core WAFL data structures and because, unlike everyone else in the storage industry, we’ve solved the random IO performance problem introduced by on-disk layout fragmentation.

FAS dedup is the most widely deployed dedup solution, the only workable primary storage dedup solution, realizes tremendous cost savings, and continues to provide compelling value to our customers.

Additional Information and Where to Find It

As mentioned above, in connection with the proposed acquisition of Data Domain, on June 4, 2009, NetApp filed with the SEC a Registration Statement on Form S-4 (Commission File Number 333-159722) containing a Proxy Statement/Prospectus for Data Domain’s stockholders, and NetApp and Data Domain each plan to file with the SEC other documents regarding the proposed transaction. The definitive Proxy Statement/Prospectus will be mailed to the stockholders of Data Domain. BEFORE MAKING ANY INVESTMENT OR VOTING DECISION, INVESTORS AND SECURITY HOLDERS OF DATA DOMAIN ARE URGED TO READ THE PROXY STATEMENT/PROSPECTUS AND OTHER DOCUMENTS FILED WITH THE SEC CAREFULLY IN THEIR ENTIRETY BECAUSE SUCH DOCUMENTS CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION.

Investors and security holders will be able to obtain free copies of the Registration Statement, Proxy Statement/Prospectus and other documents filed with the SEC by NetApp and Data Domain through the web site maintained by the SEC at www.sec.gov and by contacting NetApp Investor Relations at (408) 822-7098 or Data Domain Investor Relations at (408) 980-4909. In addition, investors and security holders will be able to obtain free copies of the documents filed with the SEC on NetApp’s website at www.netapp.com and on Data Domain’s website at www.datadomain.com.

Participants in the Acquisition of Data Domain

NetApp, Data Domain and their respective directors, executive officers and certain other members of management and employees may be deemed to be participants in the solicitation of proxies in respect of the proposed transaction. Information regarding these persons is set forth in the Proxy Statement/Prospectus described above. Additional information regarding NetApp’s executive officers and directors is included in NetApp’s definitive proxy statement on Schedule 14A, as filed with the SEC on July 14, 2008, and additional information regarding Data Domain’s executive officers and directors is included in Data Domain’s Annual Report on Form 10-K and Form 10-K/A, as filed with the SEC on March 13, 2009 and April 30, 2009, respectively. You can obtain free copies of these documents from NetApp or Data Domain using the contact information above.

How Flash killed the TLA

I come not to praise the traditional legacy storage array (TLA), but to bury it. The architectural model that served it so well for so long, has reached the end of it’s useful lifespan. Killed ironically by the technology that it thought would prolong it, Flash.

Sadly for many customers, the end of the TLA will be more like  a long slow death spiral, rather than an abrupt demise. Sad because those customers will spend money and time trying to make the wrong architecture work.

Why?

The traditional storage array is optimized for read operations. The array minimizes the amount of effort it has to do on a read by putting the disk block exactly where the client thinks it is. This means on a read, the array has to do a trivial lookup to get the disk block.

However, the downside is that the array makes the write operations more expensive.

When you factor in RAID and the impact of RAID on write performance, the TLA offers a poor value proposition.

But the poor write performance was okay as long as the read operations dominated.

Enter bigger memories

The problem with the TLA architecture was that it was designed in a pre-64 bit era, and in a pre-flash era. In that era, the servers connected to the storage array were memory starved. Because they were memory starved, the buffer cache on the server was quite small, resulting in more IO operations for reads.

In the new 64 bit era, and especially with the availability of Flash, the servers are no longer memory starved. You can now have more volatile memory than you ever had before. What that means is that the average server is rarely doing a read operation, but instead is mostly doing write operations.

The impact on the TLA is that the workload shifts from being dominated by read operations to a bigger mix of write operations.

The problem, then, is that the poorly optimized path is now a bigger piece of the overall workload mix.

Or more to the point, that poor write behavior  around RAID-6, suddenly becomes a very big issue.

Which is why, after all, the TLA vendor is recommending RAID-10.

Their poor write performance is forcing them to throw more hardware at the problem.

And their poor write performance is forcing application administrators to look at alternative storage architectures.

Enter write optimized storage arrays

Write optimized storage arrays, like ONTAP, are designed for this new world order. The downside, of course, is that the read operations are more expensive, but if the mix shifts between read and write, then that’s a reasonable tradeoff.

And as I’ve said before you can solve the read performance problem …

And why do I pick this date?

Because it has been almost a year and a half since the time the Better Than Real Fibre Channel guys demonstrated that their architecture outperformed the TLA.

An architecture optimized for read was beaten by an  architecture optimized for writes, and the response continues to be silence.

So goodbye TLA, you are so last decade…

15K RPM Drives, Not dead yet?

EMC, to their credit, were the first storage vendor to put an SSD into their array.

One point of view, was that the emergence of SSD was the end of the high performance disk drive. After all SSD’s offer vastly improved performance over 15K RPM drives, and on IOPS/$ basis are cheaper along every meaningful dimension.

Except, I have this nagging doubt.

SSDs are a great technology. They are fast, they are cheap and they look like disk drives. The problem with an SSD is that data at rest is vastly more expensive if it’s on an SSD and not on spinning rust.

So?

As I have said earlier, the reason disk drives have been so successful in the market place is that application IOPS data density maps very well to IOPS density of disk drives.

Most of the data an application has is cold data. So paying for high performance for cold data is a very expensive proposition. If the array can provide high performance to hot data even if the data is stored on slow storage, then the value of Flash as a medium for storing the data is lessened.

In other words, yes 15K RPM drives don’t provide uniformly excellent performance, but they may provide good enough performance for enough of the data to make SSD’s a niche.

Now, I’m not arguing that SSD’s are not the future, nor am I arguing that 15K RPM drives are the past, I am just observing that the future is murky.

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.

Understanding WAFL Performance: The F Word

An astute reader of my series would have observed that WAFL is able to achieve its surreal write performance because WAFL does not guarantee that two disk blocks are sequential even if the application writer assumed they were sequential. So for example, if the application thinks that it’s writing two disk blocks in sequential order, they may not appear sequentially on disk.

In many quarters this is perceived as violating an implicit contract between the storage array and the client of the storage array. And in many ways, this is at the heart of the Real vs Better Than Real FiberChannel debate.

The core question is: does this matter?

So let me, right off the bat, say: of course it does. If an application goes to the trouble to assume that two disk blocks are sequentially located on disk, then it’s assuming a certain kind of performance when it requests the data. If the storage array turns that sequential operation into several random operations, then the application performance will degrade. This is just a fact of life.

Case closed, then. WAFL sucks.

Not so fast there storage architect.

Let me first observe, that the traditional storage arrays are able to transform random write operations into sequential operations through the use of clever caching and architectures that rely on global shared memory.

In a similar vein, it is possible to do the same thing for read operations, you can apply clever algorithms to make what is a fragmented on disk layout, appear to be sequential by being clever.

Making this concrete

Suppose we have an application that writes out a file in the following order:

Where the data A was at offset 0, and B was sequentially located after A in terms of offset.

The application then has the expectation that if it operates on the file by sequentially accessing A, B and C, that it will get sequential performance.

On the other hand the application has the expectation that if it operations on the file by accessing A, D, F, that the performance will be that of random disk IO.

Last week I showed this picture that showed how WAFL did write operations:

Now let me show a different picture that shows how disk blocks could get laid out by WAFL.

The light green blocks represent free blocks. The blocks marked P are parity blocks. The blocks that are not green and have the letters A, B, C, D, E, F, G etc represent allocated blocks. A single allocated file on this set of disks is represented by a sequence of blocks share the same color. In this example we have four files.

In this example, a sequential read operation would start at a specific row, and then read the blocks in a column starting from the row. A sequential read for example would be K1, L1 and M1.

So now if we consider an application that wants to read sequentially, you can imagine how this turns into a sequence of random read operations.

For example suppose the applications wants to read blocks A, B, C and D of the blue file and then wants to read E, F and G. A simplistic implementation would do one random read to read blocks A and B, another random read for block C, and a third random read for D. A clever implementation however, would recognize that A, B, C and D although not sequentially located on disk, can be read sequentially if you willing to skip over uninteresting disk blocks. A clever implementation would also perform read-ahead to read blocks E, F and G, such that the request for the next set of blocks came from memory rather than from disk, minimizing the total IO to disk.

To be fair, constructing such algorithms that work well in general is a hard problem. As I said earlier, the relatively simple problem of a single processor scheduler that has been studied for almost 40 years still produces new insights, this is a relatively new problem and we’re still learning.

But here’s where things work in our favor. All of these algorithms benefit from increasing compute power because you can spend more time thinking, and increasing memory so you look further into the future.  Both of those trends have worked in our favor over the last 15 years.

CPU performance has increased so much faster than disk drive performance, that we are able to perform very sophisticated computations before we do any disk operations. Memory capacity has increased dramatically as well allowing us to store more things in memory before we need to actually perform an operation.

But it’s more than just hardware. It’s also having a brilliant engineering team that has been focused on solving this problem for almost fifteen years. And no amount of hardware can replicate that experience.

But don’t trust me trust benchmarks that have validated this approach. Alex McDonald in his blog shows how we were able to sustain our performance even if the face of daunting random IO.

But why does this matter, in general?

Fragmentation of the on-disk layout is not just a property of WAFL, it also is a property of things like deduplication, thin provisioning and Real Snapshots. And the reality is making those things perform well requires the same kind of algorithms and sophistication that you need to make Better Than Real FiberChannel work.

The good news is that we’ve been working on those problems for a long time, which is why our benchmarks always have used snapshots and enabled thin provisioning.

Oh that might be the reason our competitors don’t have those things enabled and recommended turning them off when you need good performance...

Understanding WAFL Performance: How RAID changes the performance game.

In my last post I explained why PAM improves performance, but I didn’t answer the other question of why WAFL kicked ass at low utilization, and more interestingly why is the PAM a read-only cache?

A number of our competitors have correctly pointed out that their cache’s are read-write and have asked, fairly, is our cache the wrong kind of cache?

To answer the question I need to talk about RAID, something I dutifully ignored in my last set of posts.

Understanding how RAID affects storage performance

This is not a primer on RAID but rather an attempt to explain how designing a high performance storage system is impacted by RAID.

Yesterday I pointed out that a Real FiberChannel RAID array can defer writes in memory such that the writes are sequential. I then said that WAFL is different, but what I described and showed didn’t seem that different. So what is different?

When you’re writing data to disk using RAID, it’s not sufficient to just write a disk block to a specific location, but you must also update the RAID parity information to allow reconstruct to take place.

So let’s assume you have this picture:

Each column represents a disk drive, and each row represents a RAID stripe.

The application sending writes to the array will force the Real FiberChannel array to update locations A1, B3, D2, D4, F3 and H1.

However if you modify location A1, you also have to modify location AP, ditto for BP, DP, FP, and HP. But because AP etc are parity computations for the whole stripe to modify them you have to first read them.

So write updates A1, B3, D2, D4, F3 and H1 really turn into 6 reads and 12 writes.

Alright, but if we look at this a little bit more carefully, because of the write cache, I don’t really need 6 reads but 5 reads. Consider stripe D, since I know D2 and D3 are updating the same stripe I can only do one read for the entire stripe.

Now that’s why write cache is good!

The more writes the storage array defers, the more likely the storage array get writes to the same stripe, the fewer additional reads the storage array will end up doing.

But WAFL is better, after all it’s Better Than Real FiberChannel

WAFL relaxes the requirement that Real FiberChannel imposes that an update to the same logical disk block must be to the same physical disk block.

So for WAFL here’s what the would happen.

Suppose the original layout looked liked this

 

Then a sequence of updates to A1, B3, D2, D4, E3, F3, G3 and H1 happen. What WAFL will do is instead of over-writing the same blocks it will write the data as a full stripe in I1 and J2 and then free up the original blocks. So for WAFL this sequence of 8 writes would appear as one write where you were sequentially writing more data.

The net layout will look like this:

So that’s why WAFL doesn’t need as much write cache

Unlike Real FiberChannel, WAFL only has to wait for enough data to write out a single stripe to disk. If there is enough free space, WAFL’s write performance is constrained by Data ONTAP’s sequential write throughput.

So that’s why WAFL performance kick’s ass!

A Real FiberChannel array has to work harder than WAFL (more disk IO’s) when there is a lot of free space. That’s why a Real FiberChannel array will benefit from additional write cache. However, there is a limit to how much you can cache, and ultimately you will end up doing more work.

A Better Than Real FiberChannel array like WAFL, will do very little work when the array has plenty of free space and therefore perform much better. A side effect, though, is that as the array free space shrinks, WAFL’s will have to do more work to find free space, and that more work will translate into lower performance. This is why as the storage utilization increases WAFL performance reaches the same level as other storage arrays.

WAFL Performance: Making writes go faster with fewer IOPS

Like every storage array, Data ONTAP attempts to make the latency of the average read and write operation be lower than the actual latency of the underlying disk drive.

Unlike Real FiberChannel systems, how Data ONTAP does achieves this is quite unique.

The Latency Challenge

If you’re a storage array life looks like a set of random read and write operations that you have to immediately respond to. The temptation is to take a request and immediately go to do disk to get it as fast as you possibly can.

But that’s kind-a-stupid.

When you look at read operations you realize that they tend to be clustered around the same piece of data. So although an application may be asking for one 4k block of data, the reality is that the next piece of data it will ask for is probably sequentially next to the first 4k block of data, or is in fact the same 4k block of data!  Given that, it makes sense to do two things:

1. Keep the 4k around in memory
2. Get more than 4k when you go to disk so that the next request is serviced from memory and not disk.

In effect by using memory you can reduce what would have been three IOPS into one actual disk IO.

Write operations are a little bit more complicated. Obviously when you acknowledge a write you’re making a promise to the application that the data is committed to stable storage. Superficially this would imply that every write operation has to go to disk since that’s the only available non-volatile storage.

In this picture each box represents a distinct write operations. The letters designate the order in which they were received. If the storage array were to write to the data to disk as they arrived, then the storage array would perform 8 IOs. Furthermore, the latency of each IO would be the latency of the disk drive itself.

Enter battery backed memory…

Except it turns out that we can turn DRAM into non-volatile memory if we’re willing to keep it powered!

What that means is that once the data is in memory, and the storage array is sure that data will eventually make to disk, the write can be acknowledged. Now that the write operations are in memory it’s possible to schedule the write operations so that they are done sequentially.

In this picture the order in which the write operations are committed to disk is a->b->h->i->e->f->c->d->g even though the order that the application thinks they were committed to disk is a->b->c->d->e->f->g->h->i. Because the array re-orders the write operations, they can be written in one single IO, effectively a single sequential write operation.

The challenge is that you need to buffer enough write operations to make this efficient.

But WAFL, of course, is different

What I just described is how Real FiberChannel works. Effectively the location of the blocks (a,b,c, etc) are fixed on disk. All the storage array can do is re-order when you write them to their fixed locations.

What WAFL does is determine where the data has to go when the data is ready to be written. So using the same picture:

Rather than trying to put each block (a, b, c, d etc) in some pre-arranged location, WAFL finds the first bit of available space and writes the data sequentially. Like Real FiberChannel, Better than Real Fiber Channel will transform the random write operations into sequential operations reducing the total number of IOPS required.

Now what’s interesting is that WAFL doesn’t require that much memory to buffer the writes. And what’s even more interesting is because the fixed overhead to flush data to disk is negligible, there is no real value in holding onto the data in memory for very long.

Read and Write and IOPS

If you’ve been following this story, you’ll have figured out something very interesting: a write operation can be deferred but a read operation must go to disk. In other words, the array can choose when it goes to disk to commit a write, but if the data is not in memory, the array must go right then and there to get the data from disk!

So what makes a disk behave in a random fashion? Why the read operations, because the writes are always done sequentially!

So why do you need a lot of IOPS? Not to service the writes, because those are being written sequentially, but to service the read operations. The more read operations the more IOPS. In fact if you are doing truly sequential write operations, then you don’t need that many IOPS ….

But it’s read operations to DISK not read operations in general!

Aha! Now I get it…

The reason the PAM card is only a read cache is that adding more memory for writes doesn’t improve WAFL write performance… We’re writing as fast as we can write data sequentially to disk.

But adding the PAM card absorbs more read operations which reduces the number of IOPS that the storage system requires. And the best way to express reduced number of IOPS is requiring both fewer and slower disks!

If you can’t comment because of  an inability to get past the CAPTCHA, try a different browser. I’ve had success with IE. We’re having some problems right now and hopefully they will get fixed soon.