Extensible NetApp: Thinking outside of the box

Snake Oil, Object Stores and Dinosaurs

On Monday I called File Virtualization Snake Oil, and Martin called me on that.

And before that I told Chuck that his object store pontifications were wrong, and he called me a dinosaur.

So let me defend myself...

Let me explain Snake Oil

My objection to File Virtualization is to the set of products in the market that don't work. Products like F5 Acopia and Rainfinity. They simply do not work. They are bumps in the wire that don't scale. They break data protection, they make backup harder, and they create all sorts of new failure modes. 

Furthermore, their main value prop is the ability to move data between tiers, and frankly there are simpler, more robust and more elegant ways to do that, even before NetApp introduced Data Motion

Object Stores

The proponents of object stores are the new wave of file system bigots. A traditional file system consists of an object store (a series of inodes with a sequence of blocks) and a namespace (a directory structure that points to a series of inodes). To find an object you do a look up into the namespace and then retrieve the data from the object store. That's how they all work, and have worked since Kernighan and Ritchie wrote their original file system.

Now I am told, that embedded name look up through directories is *flawed*. That it's in the past.

Let's be clear, extracting the namespace from the object store, and declaring the death of file systems is surprising.

The user-view of a file system is that they do a name lookup and then retrieve the object from the object store. The only question we are asking is whether the name lookup must be hierarchical and must be enforced in the storage system.

And guess what, it doesn't to both. But just because I removed the namespace look up from being embedded with the object store, doesn't mean I suddenly don't have a file system. I just have a file system that consists of multiple layered elements.

And I think having those layered elements is goodness. Because embedding the name space lookup in the storage device is too constricting. And guess what we've been doing that since 1994 when we invented this http protocol. Hmm... coincidentally NAS was invented then as well...

Real Thin Provisioning vs Pudgy Provisioning

Last week I observed that thin provisioning creates fragmentation, but I never got into the details.

And I further observed that our competition wasn’t really doing thin provisioning. I didn’t have a catchy name for that kind of provisioning, and today I do: pudgy provisioning.

Today I want to go into the details.

In a Real Fibre Channel environment, the space for the LUNs are pre-allocated and assigned to individual LUN's.

The picture above tries to show the story when using Real Fibre Channel LUNs.

When a LUN gets allocated the free space in the storage array gets committed to the particular LUN. This space allocation is not just a logical accounting trick, but is an allocation of particular disk blocks. In effect, the space allocation is pre-committed to a LUN before the application using the LUN actually uses the space.

The rationale for this scheme is to ensure that the disk layout that the array uses is identical to the disk layout the application assumes to ensure good sequential throughput. For more check out this blog entry.

The problem is that to ensure good performance the storage array wastes space.

What about thin provisioning?

The design goal for thin provisioning is to reduce the amount of waste by deferring the space allocation in the array to when the space actually gets used by the application.

The picture above tries to show the theory of thin provisioning. Unlike Real Fibre Channel space only gets committed to the LUN only when it’s actually used, eliminating the waste.

The picture, however, is misleading, because it implies that the layout on disk doesn’t change.

Enter Fragmentation

The problem with thin provisioning is that applications do not necessarily do their write operations in a way that ensures that data layout on disk is preserved.

Suppose the write order for the application is

But then the problem is that the layout on disk looks like this:

NetApp, because this kind of layout issues just happens to be endemic to how WAFL works, has solved this problem.

If you can’t do thin provisioning, how about pudgy provisioning?

If you don’t have WAFL and 15 years of experience dealing with the problem, making this work as well as WAFL is hard.

But there are shortcuts.

WAFL will allocate a disk block on disk as a disk block gets written by the application.

The most obvious alternative is to instead of doing a one-to-one disk block allocation to do a one-to-many disk block allocation. Or what I like to call pudgy provisioning.

Let’s be precise:

Whenever a single block allocation is made, rather than allocate  a single block at the next free space in the array, the array allocates an entire region.

The value of this approach is that the on disk order matches the application’s notion of what the disk order is:

.

The problem of course, is that , if the application doesn’t allocate all of the storage blocks then you’ve got waste.

Huh?

Let me see if I can follow this logic:

1. Application groups ask for too much storage
2. Real Fibre Channel creates waste because space is allocated before it’s used
3. Thin provisioning allows you to allocate only space that is used
4. If you don’t have WAFL then fragmentation on disk becomes a performance problem
5. So instead of thin provisioning do pudgy allocation where you allocate more space than the application needs but not as much as before
6. Which leads to waste…

This makes perfect sense.

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.

A new program for Real FC customers who want Better Than Real FC utilization

Last night I wrote about how in every recession there is one good idea (storage virtualization) and one bad idea (fine grained HSM). And in that discussion I observed that whereas HSM is a bad idea whose time should never come, storage virtualization is actually a very good idea that has been poorly executed.

Todays NetApp announcement about extending our space efficiency program should really be the time and place where everyone re-examines the value of storage virtualization.

So what was the underlying problem storage virtualization tried to solve:

• How to use storage that was purchased during the good times more efficiently.

So let's recap, the problems with Storage Virtualization Classic were:

1. The virtualization layer really didn't add any features above and beyond migration
2. The storage efficiencies obtained through pooling of storage were not that significant
3. The vendors supporting it were not able to offer the same customer experience as the underlying storage providers.

The real problem was more insidious than that. Storage virtualization isn't something that you deploy during a recession, rather it should be part of your infrastructure to begin with. But to be part of the infrastructure it had better have compelling features, provide real demonstrable efficiences beyond pooling and the vendor supporting it better have a first class global support organization. Storage virtualization to be useful has to be useful in good and bad times.

Thankfully NetApp has a such a storage virtualization layer (WAFL). Customers who have deployed this technology benefit from a layer that:

1. Adds a whole slew of features like REAL snapshots, FlexClone
2. Really saves space through things like RAID-DP and dedup
3. And is backed by a company that has the heft to provide global customer support.

For those of you who have Real FC, the good news is that you can get the benefits of Better Than Real FC as well through a NetApp v-series. And what's really interesting is this announcement by NetApp that extends our program about space efficiency to Real FC when you deploy v-series.

And why do EMC and HP etc hate this program?

Well as advocates of Real FC, and sadly the biggest provider of underutilized disk out there, this program is really targetted at them. Look the recession has left with customers with disks that are under-utilized or not utilized. How about taking advantage of this program to get some real efficiences from the sunk cost in your Real FC infrastructure?

The one good (storage virtualization) and the one bad (fine grained HSM) idea of every recession

In a recession, the storage group like every corporate function is asked to do more with less. One of the particular challenges with storage is that once a byte is consumed, it's hard to unconsume it. You can't just delete disk blocks willy nilly.

So you take a deep breath

One approach is to look at the stuff you already have and see if you can make better use of it. For example, when times are good we don't ask our users to delete unnecessary data, when times are bad we send out "spring cleaning" emails, and crunch down quotas. And all of that works but to a point. This is the moral equivalent of holding your breath. It helps, but it is unsustainable.

And then you try to optimize

Once it's clear that you can no longer hold your breath you need to start looking at more system approaches to doing more with less. And that's when things like storage tiering get revisited. When you were flush, everybody got better storage than they needed. When you're short on cash, and you still need to grow your storage capacity it makes a lot of sense to re-examine every application and move things around to optimize for the right cost/performance/availability tier of each application.

And in that maelstorm of applications being moved around, the ugly step-child of storage, fine-grained hierarchical storage management (HSM), makes it's appearance. Fine-grained HSM is storage tiering taken to it's absurd conclusion. If moving large blobs of data to the right tier is good, then moving small files around must be better! And if moving large blobs of data transparently is a good idea, then moving small files transparently must be all that better!

And because re-configuration of your storage is hard, and because HSM is impossible, storage and NAS virtualization appliances make their appearance. The basic idea is that by inserting some kind of layer of indirection between the host and the storage, a virtualization device allows transparent migration of data between the storage tiers. And if you do enough migration and move things around just so the net effect will be an optimized storage infrastructure!

The other benefit the storage virtualization gang tout is that by using this new layer you are able to unify disparate storage islands in your infrastructure into bigger storage pools. Bigger storage pools should create better efficiences. Right?

Except that the market for storage virtualization has failed. The market failed because if you're inserting a bump-in-the-wire you better be adding enough value to over-come the market's aversion to bumps-in-the-wire (and if you want to understand what I said you might find this one-two-three-four part series interesting ). On top of which, the bump's in the wire were being inserted by companies that were unable to provide the right level of support to make their new infrastructure work.

And so along with the storage virtualization appliances, the term went down in flames.

Which is unfortunate, because storage virtualization is the only systemic way to actually reduce your storage costs. Storage virtualization, a term hijacked by products whose only function in life appears to be to simplify migration, is a technique that allows you to reduce the amount of storage you consume by exploiting data redundancy through indirection. Storage virtualization, furthermore, allows you to separate the logical storage container from the physical infrastructure the storage container is actually stored on simplify data migration.

Enter Better Than Real FiberChannel

So for example, WAFL, NetApp's storage virtualization layer, allows us to create space-efficient clones, eliminate data redundancy through dedup and take snapshots that are performance neutral, space efficient and plentiful. Furthermore, WAFL allows us to replicate data between different physical infrastructures (SATA to FC, low-to-high and back again). In fact, the unique capabilities that WAFL has are what lead to our last promotion.

If you choose to use Better Than Real FiberChannel you can systemically move to a lower cost basis for your storage.

The problem for most Real FC vendors and ultimatley their customers, is that because Real FC vendors consider storage virtualization to be a dirty word (or at least a dirty unsavory and performance killing layer of indirection, benchmarks be damned) they can not exploit the information in the array the way WAFL does. The net effect is more storage being poorly utilized.

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

MAD Blog: Flash and the 'storage randomizers'

updated to make sure that people realize this is a MAD blog

Chuck Hollis remarked the other day that EMC's approach to flash was, at an architectural level, superior to NetApp's. He dismissively described everyone else's approach as storage randomization and that storage randomization was inherently wrong-headed.

I was intrigued by that comment because the rationale he gave did not mesh with my understanding of the physical properties of flash.

First let me attempt to explain the difference Chuck draws between EMC and NetApp.

NetApp's approach to storage can be described with this diagram:

Essentially we take a collection of similar disks, FC or SATA, combine them into a bigger container, an aggregate and then provision volumes from which applications consume storage.

As data gets written we allocate physical blocks on demand.

EMC's approach can be described with this diagram

EMC's logical storage unit has blocks that are allocated up front to the physical disk block. In effect, there is no layer of indirection between the disk block on disk and the logical block that is presented to the application.

The argument Chuck made was that if you consider Flash storage, the performance of Flash would make any latency introduced by the virtualization layer of the array unacceptable.

The problem with that argument is that what you can do with disk, namely permanently associate a logical block of storage to a physical block of storage is not how flash must be used.

Now it turns out that an annoying property of Flash is that a particular disk block can only be written to a fixed number of times.

In an EMC context, if you directly mapped block 500 on the flash disk, to a storage unit at some point in time the writes to the disk block would fail because of the inherent physical properties of flash.

Flash disk drive vendors know about this so here's what they do:

What they do is they add their own layer of virtualization above the raw flash so that the logical block they present to the array can be remapped under the covers of the array. Given how traditional array's work, they have to do this because the direct mapping technology would destroy flash drives at too fast a rate.

In some corners this technology is called wear-leveling.

So if we go back to the EMC architecture Chuck was talking about we have the following picture:

The logical storage container that the storage admin has assigned to a physical flash disk block is actually a logical disk block sitting on top of the disk drive vendors virtualization layer.

1. If there is no layer of virtualization between the raw  flash and the logical container then you will wear out the flash too fast.
2. If you don't have your own, you have to rely on the FC SSD (or enterprise flash drive) vendors to provide it.
3. If the Flash Vendors are providing the virtualization layer what is your value? Are you just a bump in the wire between the disk and the client?
4. There is no direct mapping like there is in disk, so it's unclear what the value of providing that direct mapping is.

If we look at NetApp's approach our goal is to integrate the flash into our virtualization layer directly by exploiting the fact that WAFL can do much of the same function that flash virtualization does. You can check out Val's post for more detail on that.

So I don't know whose architecture is necessarily superior, but I am not sure I buy into the argument that NetApp's architectural approach is by definition inferior and by definition poorly suited to Flash like Chuck claimed.

Backup, the new storage tiers and real snapshots

When I first described my new terminology for storage tiers, Tim Burlowski, asked how backup and recovery fit into that model.

It turns out that that is an interesting question.

Backup, the only working ILM/HSM solution

Step 10 feet away, and you realize that backup is HSM.

The backup process creates cold data, the backup image. The backup software them moves the backup copy to cheaper tier of storage. When a restore is performed, the backup software moves the backup copy from the cheaper tier to a more expensive tier.

The question on the table, then, is where does this form of HSM still make sense?

Captive IOPS

One obvious place where HSM as part of backup process is for the backup of data on a Captive IOPS tier. The tier is built using very expensive storage. Storing cold data that will probably never get accessed seems odd and a waste of money.

But that's not necessarily true.

If the Captive tier is built using disk drives, and the application  has a high IOPS data density, a lot of the capacity of the disks is  unused. The cheapest place to store backup copies, in that case, would be on the same disks you're using for primary data.

If the Captive tier is built using SSD's like EMC storage, then it makes a lot of sense to use a backup solution to move backup images off of the flash onto cheaper storage. If the application has a low IOPS data density, then a better approach is to use flash cache in front of disk.

If the Captive tier is built using a flash cache in front of disk, the cheapest place to store the backup copies may be on the disk drives.

More generally, the backup methodology of a captive tier made up of flash cache and disk, will be the same as the backup methodology of the capacity tier.

So what about the capacity tier and backup?

In the general case, the capacity tier will be backed up much in the same way that it always has been.

A capacity tier that has real snapshots can eliminate the HSM part of the backup process.

In fact, in an earlier series on backup I proved that for a VMware infrastructure, the cheapest place to store backup copies was on the same disks that stored the active data.

Now I want to generalize that comment for all storage..

If you consider IOPS density, disk drives must contain large amounts of cold data. A subset of the cold data in a data center is the backup data. That data can either be stored on a separate set of disks or on the same disks that are being used to serve IOPS.

To store the data on the same disks the following things need to be true

1. Creating the backup image must consume minimal IOPS
2. The existence of the backup image must impose no performance penalty
3. Be able to store a large number of images.
4. The storage must be configured using some form of RAID-6 or RAID 1-0
5. The cost of the backup images must be cost effective

It turns out that for NetApp systems all five are true.

So as disk capacities expand, storage architects who have the option to use real snapshots will stop moving data off of the capacity tier to some cheaper tier.

Said in a slightly more poetic way:

The final resting place for data on disk will be the disk where the data first got created.

The net effect will be a much simpler and more cost effective storage infrastructure to support backup.

One minor addendunm added after I wrote this post.

As Martin G says in the comments below, having the backup copies on the local disks doesn't protect you from site failures. So, of course, some form of storage replication is required. Thankfully NetApp has a cool technology called SnapMirror that replicates all of the snapshots to a remote destination in a space efficient way.

In an earlier post on VMware and backup I show how all of this fits together.

I should have included this need to replicate the data in my original blog post. So I am rectifying that error now.

A response to Chuck on Deduplication for EMC

Chuck's response to my post Introducing Deduplication for EMC was interesting. Val's response to Chuck's post was hilarious and informative.

And although I was sorely tempted to walk away from this one, I couldn't resist the temptation to respond to the first question.

Question #1 -- What's Involved In Installing One Of These?

Chuck's answer to this question was interesting, not for the answer, but the kind of answer.

Here's what he did:

Assert Something Outrageous: The only deployment model was rip-n-replace

Observe something reasonable based on outrageous assertion: Rip-n-replace is expensive

Conclude Outrageous Proposal is Outrage: The deployment of a V-Series is impractical.

Although a good rhetorical device, this line of reasoning is not a very useful way to have a thoughtful conversation.

Because I am technologist and not in product marketing, let me offer a different, and I hope more thoughtful, response.

Needing more storage

Any growing enterprise will have to store more data, and storing more data will require more storage. Now it turns out that in practice there are only two ways to get more storage:

1. Buy more
2. Use what you have more efficiently

The real story about V-Series and Deduplication is that it really addresses the second use case. Using our V-Series technology you can use the disk you've already got more efficiently.

Who needs storage

There are three basic consumers of storage

1. Existing applications whose needs expand and the storage is providing good enough service at the right cost
2. Existing applications that are going through a technology refresh
3. New applications

Existing applications that just need more storage, especially if the storage is good enough and at the right price, are rarely a good target for any new technology. It's always easier to just use more of the same.

Existing applications that are going through a technology refresh are a good target for introducing new technology. If you're going through a technology refresh, you are probably looking at the whole stack from application to OS to server to storage trying to figure out how to get more for less money. Depending on how thorough the refresh is, you've already budgeted some amount of disruption and you're willing to live with that disruption.

New applications are an obvious target for introducing new technologies in an infrastructure. Processes and procedures have not yet been deployed, so there is a lot of opportunity to look at the best and most cost-effective way to meet your business objectives.

So the answer to Chuck's question is?

If you are looking at increasing the utilization of your existing infrastructure as you deploy new applications or go through technology refreshes for existing applications, then the answer to what's involved in deploying one of these is not rip-n-replace but

The complexity of deployment is the complexity of setting up a V-Series to use an EMC device.

So that suggests that the ROI for the device is a lot lower than the catastrophic 20 hours of down-time per-server that Chuck suggested.

Chuck and the Fox

Recently Chuck has taken to attacking features in smart storage arrays.

First we had the diatribe against the stupid storage admin who dared to want thin provisioning, then the rant about data placement to which I responded, and today we had the pooh-pooh'ing of a technology that over 2500 NetApp customers have deployed to save real dollars in their environments. 

Were Chuck's comments the sage advice of a patrician warning his younger brethren that these were bright and shiny toys that were not really valuable?

Or  was Chuck dismissing the value of technologies that NetApp customers had used to improve utilization and save money, because EMC's storage portfolio lacked the ubiquity and breadth of the capabilities that NetApp had?

The whole sequence of posts reminded me of a story my grandma used to tell me: a fable written by a Greek about a fox and grapes.

Could it be that Chuck was like the fox in Aesop's fable?

So without further ado,  I present a re-working of that masterpiece ...

One hot summer's day a Fox was strolling through a data center till he came to a bunch of storage arrays that were using thin provision and deduplication to improve storage utilization. "Just the thing to reduce my costs and improve my efficiencies," quoth he. Drawing back a few paces, he took a run and a jump, and tried to use those technologies on his EMC gear. Turning round again with a One, Two, Three, he tried but with no greater success. Again and again he tried to get thin provisioning to work, to figure out which of the four ways of doing deduplication he could make work, but at last had to give it up, and walked away with his nose in the air, saying: "I am sure they are worthless technologies."