Extensible NetApp: Thinking outside of the box

DAS Disruption part 8: How Big Disks killed RAID-5 and RAID-10

When I first identified this DAS disruption I was wondering if I was all alone. Interestingly, one of our top guys in the Microsoft space and an MVP Exchange architect, John Fulbright, identified the same trend starting in a different place. Whereas I started from first principles, he started from looking at competitive pricing.

What we both figured out was that the DAS Disruption was triggered by the failure of the dominant storage vendors to build an effective and efficient RAID-6 implementation.

In this series on the DAS Disruption, and earlier, I’ve remarked how RAID-5 and RAID-10 are dead because of Big Disks.

The Microsoft team has today written an excellent post that puts numbers to the difference in the risk profile between RAID-5, RAID-10 and RAID-6 or RAID-DP.

They produced this nice graphical visual:

 

that shows exactly what I’ve been saying about RAID-6 for some time.

You should check out their whole blog and the analysis.

FAS Deduplication is here for the long haul

The current tussle between NetApp and EMC has, predictably, created a dust-up amongst the corporate blog-o-sphere. After all the storage industry believes in death-match-competitive-blogging, none of this polite, civilized discourse, here we reach for the knives and hack!

And it would be tempting to point out that Chuck and Storagezilla are like Aesop’s fox, but that would be too easy.

However, there is a point of confusion in the storage and analyst community about the relationship between FAS dedup and inline dedup technologies like the one provided by Data Domain. There is some opinions that say that this proposed acquisition is driven by flaws in our technology. There are some customers who may be wondering whether this reflects some flaws in our technology.

So let me state again, what I said earlier:

The rest of the storage industry has adopted inline because they lack NetApp's storage virtualization layer that enables primary deduplication.

But because that confusion is real, I wanted to remind folks about why FAS dedup is unique. And more importantly, why NetApp FAS dedup is different and must be different to meet the market requirements our storage plays in than other inline dedup solutions.

I went over this in a post several months ago in a post titled A little digression on Inline Dedup.

The basic points I made are the following:

1. Deduplication is a compute intensive operation
2. Single core performance is stagnating
3. Latency isn’t something it’s the only thing
4. Backup workloads are latency insensitive but throughput sensitive
5. Primary storage workloads are latency sensitive

And I observed that because of 1, 2 and 5 any inline process is problematic if not deeply flawed.

And then in another post I observed that the important question for dedup isn’t how efficient the dedup process is, but how well do you perform after the dedup process is complete.

Our deduplication process performs well because of it is integrated into our core WAFL data structures and because, unlike everyone else in the storage industry, we’ve solved the random IO performance problem introduced by on-disk layout fragmentation.

FAS dedup is the most widely deployed dedup solution, the only workable primary storage dedup solution, realizes tremendous cost savings, and continues to provide compelling value to our customers.

Additional Information and Where to Find It

As mentioned above, in connection with the proposed acquisition of Data Domain, on June 4, 2009, NetApp filed with the SEC a Registration Statement on Form S-4 (Commission File Number 333-159722) containing a Proxy Statement/Prospectus for Data Domain’s stockholders, and NetApp and Data Domain each plan to file with the SEC other documents regarding the proposed transaction. The definitive Proxy Statement/Prospectus will be mailed to the stockholders of Data Domain. BEFORE MAKING ANY INVESTMENT OR VOTING DECISION, INVESTORS AND SECURITY HOLDERS OF DATA DOMAIN ARE URGED TO READ THE PROXY STATEMENT/PROSPECTUS AND OTHER DOCUMENTS FILED WITH THE SEC CAREFULLY IN THEIR ENTIRETY BECAUSE SUCH DOCUMENTS CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION.

Investors and security holders will be able to obtain free copies of the Registration Statement, Proxy Statement/Prospectus and other documents filed with the SEC by NetApp and Data Domain through the web site maintained by the SEC at www.sec.gov and by contacting NetApp Investor Relations at (408) 822-7098 or Data Domain Investor Relations at (408) 980-4909. In addition, investors and security holders will be able to obtain free copies of the documents filed with the SEC on NetApp’s website at www.netapp.com and on Data Domain’s website at www.datadomain.com.

Participants in the Acquisition of Data Domain

NetApp, Data Domain and their respective directors, executive officers and certain other members of management and employees may be deemed to be participants in the solicitation of proxies in respect of the proposed transaction. Information regarding these persons is set forth in the Proxy Statement/Prospectus described above. Additional information regarding NetApp’s executive officers and directors is included in NetApp’s definitive proxy statement on Schedule 14A, as filed with the SEC on July 14, 2008, and additional information regarding Data Domain’s executive officers and directors is included in Data Domain’s Annual Report on Form 10-K and Form 10-K/A, as filed with the SEC on March 13, 2009 and April 30, 2009, respectively. You can obtain free copies of these documents from NetApp or Data Domain using the contact information above.

The assault on RAID

Earlier this week I wrote about how the core architecture of Traditional Legacy Arrays limits the performance of RAID-6. Just to recap, RAID-6 requires an additional read and an additional write over and above RAID-5, limiting it’s performance and applicability.

What I didn’t do is explore the consequences of such limitations.

The first, and most obvious consequence is the proliferation of RAID-10. SATA disks are too big to be used with RAID-5. The only alternative is to use RAID-10 when you need both resiliency and performance!

This singular fact turns the economics of RAID on it’s head.

Let’s be clear, the reason RAID became so popular is because it solved two generic problems: performance and availability at a lower cost than the alternatives. The problem with disk drives is that they have a limited number of IOPS per spindle. The only way to get more IOPS is to combine more of them and spread the IOPS over the spindles. The downside is that the MTBF (mean-time-between-failures) of the disk drives makes the group of disk drives increasingly less reliable.

So if you’re a customer that has business critical application that requires availability and performance, you’re more or less screwed. To get the availability and the performance you need to 2-3x the storage. As an example, consider GFS, that uses mirroring and no RAID. GFS will have at will have at least 3 full copies of the data.

Enter RAID.

RAID is so common place now-a-days that we forget how miraculous it must have appeared to folks when it first showed up. RAID allowed the computer industry to provide resiliency and performance at a lower cost! So for example, using RAID-5, you only need 1/5 (4+1) additional storage to get terrific data resiliency compared to the 3x that GFS requires.

Enter Big Drives

The problem with RAID-5 and RAID-4 is that the probability of various kinds of failures increases as the disk drive capacity increases. No surprise.

NetApp pioneered the use of RAID-6 with our FAS systems, and because of WAFL we are able to preserve the RAID value proposition of increasing capacity and performance with modest increase in physical capacity.

But we’re not the market.

As much as I would love to be the top-dog in the storage industry, the Traditional Legacy Array vendors, and in particular EMC, don’t have a very good RAID-6 implementation. Or rather, as I said earlier, their core architecture precludes a RAID-6 implementation that they can recommend for all workloads.

And because of their share of the market, they define the technology expectations for everyone who intends to use storage.

The problem is that we don’t live in a static universe. Software companies, like Microsoft, are continuously trying to reduce the total cost of their solutions. And although it’s nice to be able to bill customers for RAID-10, the reality is that for every dollar that goes into the hardware, Microsoft makes less money.

So Microsoft and the rest of the software industry, faced with the reality that the dominant Traditional Legacy Array vendors, were pushing RAID-10, or basically 2x the physical capacity for each byte of data stored, looked elsewhere for a solution.

It must have been particularly galling to software vendors that disk drive capacities were increasing, disk drive cost per GB per decreasing, but the storage infrastructure was increasing in cost reducing the total available pie.

Enter host based replication etc…

The cool thing about technology trends and curves, is that everyone is staring at them.

If I was a software application architect here’s what I would observe. The biggest cost for an application from a hardware perspective is the storage, and biggest cost factor is the disk drives.

Today with RAID-5, the picture looks like this:

The red controllers are provided by Traditional Legacy Array vendors, and there is a small increase in the total capacity to get excellent resiliency (the pink area).

But with bigger SATA drives, and the absence of a high performance RAID-6, the world looks like this:

Ouch. The total cost of the solution just jumped up! You get cheaper disk drives, but you need more of them!

Now if I am an application architect, I think to myself, you know that RAID stuff can be done in software, and I can do the mirroring at the application level, and the server vendors sell really cheap disk drives, so maybe the picture can really be made to look like this:

Each server has a pool of disk drives, and mirroring is done in the host to obtain some degree of storage resiliency. Yes the costs are greater than RAID-5, but it’s not clear they are that much higher than RAID-10 … And you might even believe they were lower than RAID-10 because you’re saving on the dollars you had to give to the storage company.

So what does this have to do with NetApp?

I really believe that RAID-10 mirroring or DAS is an inefficient way to manage disk drives.

I also believe that the popularity of DAS/Host mirroring solutions is a response to the failure of the market leaders to build an array that can preserve the RAID value proposition.

Since RAID-10 and DAS/Host mirroring are a response but the wrong fundamental approach, I believe that architectures, like NetApp’s, that make RAID-6 work will ultimately win out.

And oftentimes in the debate about DAS vs shared storage, we confuse the specific problems of the Traditional Legacy Array vendors, with the right long term technology trend.

Better Than Real Fibre Channel: The Other Guys

Brian Rawlings, correctly, took me to task over my parochial focus on WAFL as the only Better Than Real Fibre Channel technology.

The reality is that the kind of things WAFL does are being done by other vendors. I'll claim not as well, or not as elegantly or not as completely, but there are other vendors who are breaking with Traditional Storage Array architectural orthodoxy that disk layout must correspond to what the client thinks the disk layout is.

So just for the record, the other guys include but are not limited to the folks at 3Par, Compellent, and Equallogic.

And for the record, the future belongs to us (just that more of the future belongs to NetApp :-))

The importance of architecture

This is a more technical/philosophical post than my usual commentary, forewarned is forearmed.

Earlier I wrote about the notion that software is like group theory. What I said was that there exist abstractions that form in a software system over time, and that the key to understanding and improving a software system is to understand those abstractions.

I also argued, that those abstractions were not accidental, but were actual manifestations in software of how the world really worked. By making the case that these were real abstractions and not artificial constructs, I argued that changing or fighting them was done at your peril.

What's important, of course, is that any mathematical, and therefore by extension, software abstraction is an approximation of the world, not the real thing. And that approximation may help you solve some problems better than others ...

When we argue about software and hardware systems in the corporate storage blog-o-sphere we are arguing about specific notions of how the world is actually constructed. I believe part of the acrimony and bitterness is that we're not just puffing our chests and cheering our team on, but we actually have different wold views.

Enough with the blah, blah, architecture, blah, group theory, blah

My recent posts about WAFL reminded me of a theory Prof. David Cheriton repeated in his Stanford course on distributed systems. What Prof. Cheriton observed was that many times innovation happens when, by exploiting a computer trend, you relax a previously hard constraint making a previously hard thing easy.

The example he used, if my increasingly faulty memory serves me right, was the use of HTML/XML and ASCII to serialize or marshall and then deserialize or unmarshall parameters.

A canonical problem in distributed system design is how to transport data over the wire. The problem is that data is represented as a stream of binary digits in main memory. The only thing that gives it context is the structure software imposes on the data.

The problem in a distributed system is that the software can be written in two different programming languages running on different hardware meaning that there has to be well understood language and hardware independent rules about what the stream of bits mean.

For example, suppose I have remote procedure call int foo(int b). I need to convert the input parameter b into some format that can be transported over the wire, and interpreted at the other side as an integer with the same value.  This transformation is what is known as marshalling and unmarshalling, I'll call it translation.

The traditional approach to this problem was to use a binary protocol that encoded the integer into a system independent format as you went over the wire and then translated the binary data into the system dependent format at the point where it needed to be interpreted. Software on either side of the wire using the translator could send and receive data.

Building such and infrastructure that is robust is a relatively hard but straightforward problem.

Building such an infrastructure that is widely used by everyone was impossible.

The problem was that there was no industry standard for the translators. For computer programs to talk to each other over the wire they needed proprietary and expensive binary translators. CORBA, for example, was a failed attempt at creating such a standard.

Enter HTML/XML

HTML/XML are horribly inefficient compared to a binary translator. HTML/XML are inefficient  with their use of bandwidth and their use of compute resources. For example a four byte integer quantity can turn into an 11 byte string, plus the meta-data required to tag the value.

But HTML/XML do not require a proprietary binary translator on either side of the wire.

So the idea was to relax your requirements for performance to get ease of use. And that was okay because CPUs  were becoming faster, memory was becoming cheaper and bandwidth was becoming more plentiful.

The net result was that HTTP+HTML/XML became the defacto mechanism for programs to communicate to each other over the internet and things like ONC RPC/CORBA/JAVA RMI/DCOM died or are dying a slow and increasingly painful death.

But if that's too abstract and confused...

Computer programming languages are another good example.

Programs written in C can be faster and more efficient than programs written in Java. But the time to develop, debug and qa those programs is vastly greater than an equivalent program written in Java. 

By trading off computational efficiency (memory and CPU resources), programmers are made more efficient, resulting in more software being developed.

I am storage architect not a computer science historian...

When Dave Hitz and James Lau architected WAFL they relaxed the fundamental constraint that file systems had preserved for a long time, namely that the client notion of disk layout actually corresponded to the on-disk-layout.

This was a profound decision. By relaxing that constraint, WAFL had taken on the job of providing the illusion to clients that the disk-layout was sequential even if it wasn't.

At the time that might have appeared to be a ridiculous thing to do. And given the reaction of the other guys (vendors of Traditional Legacy Arrays) still seams to be a ridiculous thing to do.

But...

If you looked at the computer trends, maybe it wasn't. By exploiting the increasing performance of CPU's, the increasing density of memory, it became possible to write smarter and smarter algorithms that were more and more capable of providing the illusion that the disk layout was sequential even if it wasn't (for more on this topic see an earlier post)

And by relaxing that constraint, it became possible to deliver innovation like thin provisioning, Real Snapshots and deduplication without compromising performance.

Why?

All of these features, primary dedup, Real Snapshots, high performance RAID-6, require an on-disk layout that is different from the client layout. For our, NetApp, internal abstractions, these are easy to do and a natural extension of our software abstractions. And modifying your system in natural ways is easy only if your abstractions easily map to those abstractions.

For Traditional Storage Arrays, fixed in their notions of disk layout, asking them to support a model where the on-disk layout is different from the client layout challenges the fundamental abstractions they have evolved over the last 15 years. And so they try and create approximations, like pudgy provisioning, but ultimately the extensions looks like an ill fitting suit, because they are.

Real Fibre Channel is really Traditional Legacy Array

Chuck Hollis likes to call the CLARiiON LUN a Real LUN.

I've taken to mocking Chuck and EMC by calling their storage Real Fibre Channel.

But after my recent series on RAID-6, and Thin Provisioning and the fact that Real Fibre Channel has architectural  flaws that make it less than useful for either of those critical capabilities, and the limitations around snapshots, I've been thinking that the term was just plain wrong.

Real Fibre Channel is like the PDP-11:  A legacy hardware architecture that has had it's moment in the sun.

Henceforth, I'll call the CLARiiON and the architectural dinosaurs that are like it, what they really are:

Traditional Legacy Array.

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Real Thin Provisioning vs Pudgy Provisioning

Last week I observed that thin provisioning creates fragmentation, but I never got into the details.

And I further observed that our competition wasn’t really doing thin provisioning. I didn’t have a catchy name for that kind of provisioning, and today I do: pudgy provisioning.

Today I want to go into the details.

In a Real Fibre Channel environment, the space for the LUNs are pre-allocated and assigned to individual LUN's.

The picture above tries to show the story when using Real Fibre Channel LUNs.

When a LUN gets allocated the free space in the storage array gets committed to the particular LUN. This space allocation is not just a logical accounting trick, but is an allocation of particular disk blocks. In effect, the space allocation is pre-committed to a LUN before the application using the LUN actually uses the space.

The rationale for this scheme is to ensure that the disk layout that the array uses is identical to the disk layout the application assumes to ensure good sequential throughput. For more check out this blog entry.

The problem is that to ensure good performance the storage array wastes space.

What about thin provisioning?

The design goal for thin provisioning is to reduce the amount of waste by deferring the space allocation in the array to when the space actually gets used by the application.

The picture above tries to show the theory of thin provisioning. Unlike Real Fibre Channel space only gets committed to the LUN only when it’s actually used, eliminating the waste.

The picture, however, is misleading, because it implies that the layout on disk doesn’t change.

Enter Fragmentation

The problem with thin provisioning is that applications do not necessarily do their write operations in a way that ensures that data layout on disk is preserved.

Suppose the write order for the application is

But then the problem is that the layout on disk looks like this:

NetApp, because this kind of layout issues just happens to be endemic to how WAFL works, has solved this problem.

If you can’t do thin provisioning, how about pudgy provisioning?

If you don’t have WAFL and 15 years of experience dealing with the problem, making this work as well as WAFL is hard.

But there are shortcuts.

WAFL will allocate a disk block on disk as a disk block gets written by the application.

The most obvious alternative is to instead of doing a one-to-one disk block allocation to do a one-to-many disk block allocation. Or what I like to call pudgy provisioning.

Let’s be precise:

Whenever a single block allocation is made, rather than allocate  a single block at the next free space in the array, the array allocates an entire region.

The value of this approach is that the on disk order matches the application’s notion of what the disk order is:

.

The problem of course, is that , if the application doesn’t allocate all of the storage blocks then you’ve got waste.

Huh?

Let me see if I can follow this logic:

1. Application groups ask for too much storage
2. Real Fibre Channel creates waste because space is allocated before it’s used
3. Thin provisioning allows you to allocate only space that is used
4. If you don’t have WAFL then fragmentation on disk becomes a performance problem
5. So instead of thin provisioning do pudgy allocation where you allocate more space than the application needs but not as much as before
6. Which leads to waste…

This makes perfect sense.

The increasingly meaningless question of storage utilization

I've been recently engaged in a series of internal discussions about the question of what is storage efficiency.

As a company, NetApp has continued to deliver amazing amount of value by allowing our customers to store huge amounts of data on a relatively small storage foot print and yet gets routinely criticized for our inefficiency.

In point of fact, we create a marketing program to put our money where our mouth is, hell we even run a local marketing campaign, and still we get asked: well how efficient are you really?

Which makes me wonder if we're just barking up the wrong tree.

Storage Efficiency is so last century

SAN storage architectures built around Real Fibre Channel principles are what created the current insanity that frames the storage efficiency discussion.

They declared that the storage problem could be best be solved by creating a storage architecture that first required you to combine lots of storage into a single disk pool, and then statically parceled out the resources into smaller chunks that were handed out to servers. Storage utilization was a measure of how effective you were in using your storage pool. And then the question was of the storage you had allocated how efficient you were in using that.

Great.

So the entire storage industry is married to a storage management model that says that storage efficiency is a measure of how efficiently you used antediluvian architectures based around static partitioning.

Wonderful.

But the problem, and here's the rub Better Than Real Fibre Channel can do things that Real Fibre Channel can not. Because Better Than Real Fibre Channel can use block level reference counts, Better Than Real Fiber Channel can store 255 full copies of data at the cost of that actual unique data within those 255 copies.

So if I have a snapshot, a single NetApp snapshot, I have a full copy of the file system. But instead of being applauded for storing a huge amount of data on a small amount of storage, I have to explain the WAFL overhead.

I'll be blunt, the WAFL overhead allows you to store what to a client appears to be a full copy of the file system on the minimal amount of disk blocks necessary. You wanna do that using Real Fibre Channel, I really hope you like buying disk drives.

And if it was a couple of EMC bloggers I wouldn't mind. But this discussion has permeated the entire computer industry.

And I wouldn't care too much except the question of how we should measure storage system efficiency is not even being discussed because of the fusillade of FUD. And because at the end of the day no one really wants to talk about how inefficient their architectures are.

If they did, storage customers would learn exactly why it's better to use RAID10 to protect your storage instead of replicating your data at the application level.

It's almost 2010 ...

NetApp and Data Domain, and soon others, have demonstrated that a storage system that uses indirection between the disk block and the logical disk block can store more data on less storage than we ever imagined.

And instead of talking about how efficiently we do that, in terms of CPU/Memory, or effectively in terms of the impact to applications we continue to discuss what is the most pointless of conversations:

If I treat my very sophisticated storage array like a set of big dumb disk drives I bought at the local electronics discount store, what's the usable capacity?

Blech.

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.