Extensible NetApp: Thinking outside of the box

A new program for Real FC customers who want Better Than Real FC utilization

Last night I wrote about how in every recession there is one good idea (storage virtualization) and one bad idea (fine grained HSM). And in that discussion I observed that whereas HSM is a bad idea whose time should never come, storage virtualization is actually a very good idea that has been poorly executed.

Todays NetApp announcement about extending our space efficiency program should really be the time and place where everyone re-examines the value of storage virtualization.

So what was the underlying problem storage virtualization tried to solve:

• How to use storage that was purchased during the good times more efficiently.

So let's recap, the problems with Storage Virtualization Classic were:

1. The virtualization layer really didn't add any features above and beyond migration
2. The storage efficiencies obtained through pooling of storage were not that significant
3. The vendors supporting it were not able to offer the same customer experience as the underlying storage providers.

The real problem was more insidious than that. Storage virtualization isn't something that you deploy during a recession, rather it should be part of your infrastructure to begin with. But to be part of the infrastructure it had better have compelling features, provide real demonstrable efficiences beyond pooling and the vendor supporting it better have a first class global support organization. Storage virtualization to be useful has to be useful in good and bad times.

Thankfully NetApp has a such a storage virtualization layer (WAFL). Customers who have deployed this technology benefit from a layer that:

1. Adds a whole slew of features like REAL snapshots, FlexClone
2. Really saves space through things like RAID-DP and dedup
3. And is backed by a company that has the heft to provide global customer support.

For those of you who have Real FC, the good news is that you can get the benefits of Better Than Real FC as well through a NetApp v-series. And what's really interesting is this announcement by NetApp that extends our program about space efficiency to Real FC when you deploy v-series.

And why do EMC and HP etc hate this program?

Well as advocates of Real FC, and sadly the biggest provider of underutilized disk out there, this program is really targetted at them. Look the recession has left with customers with disks that are under-utilized or not utilized. How about taking advantage of this program to get some real efficiences from the sunk cost in your Real FC infrastructure?

The one good (storage virtualization) and the one bad (fine grained HSM) idea of every recession

In a recession, the storage group like every corporate function is asked to do more with less. One of the particular challenges with storage is that once a byte is consumed, it's hard to unconsume it. You can't just delete disk blocks willy nilly.

So you take a deep breath

One approach is to look at the stuff you already have and see if you can make better use of it. For example, when times are good we don't ask our users to delete unnecessary data, when times are bad we send out "spring cleaning" emails, and crunch down quotas. And all of that works but to a point. This is the moral equivalent of holding your breath. It helps, but it is unsustainable.

And then you try to optimize

Once it's clear that you can no longer hold your breath you need to start looking at more system approaches to doing more with less. And that's when things like storage tiering get revisited. When you were flush, everybody got better storage than they needed. When you're short on cash, and you still need to grow your storage capacity it makes a lot of sense to re-examine every application and move things around to optimize for the right cost/performance/availability tier of each application.

And in that maelstorm of applications being moved around, the ugly step-child of storage, fine-grained hierarchical storage management (HSM), makes it's appearance. Fine-grained HSM is storage tiering taken to it's absurd conclusion. If moving large blobs of data to the right tier is good, then moving small files around must be better! And if moving large blobs of data transparently is a good idea, then moving small files transparently must be all that better!

And because re-configuration of your storage is hard, and because HSM is impossible, storage and NAS virtualization appliances make their appearance. The basic idea is that by inserting some kind of layer of indirection between the host and the storage, a virtualization device allows transparent migration of data between the storage tiers. And if you do enough migration and move things around just so the net effect will be an optimized storage infrastructure!

The other benefit the storage virtualization gang tout is that by using this new layer you are able to unify disparate storage islands in your infrastructure into bigger storage pools. Bigger storage pools should create better efficiences. Right?

Except that the market for storage virtualization has failed. The market failed because if you're inserting a bump-in-the-wire you better be adding enough value to over-come the market's aversion to bumps-in-the-wire (and if you want to understand what I said you might find this one-two-three-four part series interesting ). On top of which, the bump's in the wire were being inserted by companies that were unable to provide the right level of support to make their new infrastructure work.

And so along with the storage virtualization appliances, the term went down in flames.

Which is unfortunate, because storage virtualization is the only systemic way to actually reduce your storage costs. Storage virtualization, a term hijacked by products whose only function in life appears to be to simplify migration, is a technique that allows you to reduce the amount of storage you consume by exploiting data redundancy through indirection. Storage virtualization, furthermore, allows you to separate the logical storage container from the physical infrastructure the storage container is actually stored on simplify data migration.

Enter Better Than Real FiberChannel

So for example, WAFL, NetApp's storage virtualization layer, allows us to create space-efficient clones, eliminate data redundancy through dedup and take snapshots that are performance neutral, space efficient and plentiful. Furthermore, WAFL allows us to replicate data between different physical infrastructures (SATA to FC, low-to-high and back again). In fact, the unique capabilities that WAFL has are what lead to our last promotion.

If you choose to use Better Than Real FiberChannel you can systemically move to a lower cost basis for your storage.

The problem for most Real FC vendors and ultimatley their customers, is that because Real FC vendors consider storage virtualization to be a dirty word (or at least a dirty unsavory and performance killing layer of indirection, benchmarks be damned) they can not exploit the information in the array the way WAFL does. The net effect is more storage being poorly utilized.

Flash Point: Missing the point on TLAs

Martin G, as he does so well, cut through the recent Flash brouhaha by pointing out that stuffing an SSD with a FiberChannel connector into a storage array is more a statement about the sad state of storage innovation than any miracle of modern engineering.

So what about NetApp?

NetApp's plans were announced in a whitepaper that described our strategy for integrating flash into our storage systems.

The PAM announcement, even though it's not Flash, is a proof point, another execution point on our roadmap to our destination. .

NetApp's recent announcement on TMS RAMSAN, doesn't really count as the great leap forward either. In my mind it's just more proof/evidence of that the roadmap.

But in spite of what  I just said, I think the TMS RAMSAN announcement is interesting in and of itself.

Why?

Because of the value of our software.

Our software lets you use less storage, make faster copies and keep more backup images than any other storage software out there. And flash just makes it all go faster. And by making it go faster, you're allowing customers who never thought they could benefit from our storage software benefit from it. And all of a sudden they are keeping more backups, using less storage and making faster copies than they were before. And all of a sudden that creates a new set of opportunities for them.

Because flash isn't a destination. Flash is an enabling technology.

The question is, enables what?

The short answer is it enables much faster reads at a fraction of the cost of DRAM.

But that isn't that interesting either.

What's really interesting is how does software adapt itself to Flash.

And no one knows the answers to those questions.

But here's what I do know, everytime you make something faster and cheaper and more plentiful, innovative people figure out a way to exploit that plenty to do new things.

So let me give you an example of what more faster reads enables.

Add more memory to our systems, and all of a sudden you don't need a FiberChannel disk tier in some infrastructures. And that suddenly frees up more money because you're buying cheaper disk, and it suddenly frees up time because you're no longer doing tiering. And that suddenly is very very interesting.

You make reads more plentiful, and guess what things change.

So focusing on the speed of Flash or how it's packaged is, in my mind, about as boring a discussion as you can imagine.

But hey, leave it to a storage customer to point out when storage vendors are too busy examining their plumage instead of focusing on the real value to the customer.

Adventures in consumer flash

I recently purchased a netbook. And to protect the innocent, I'll keep vendor names to myself.

As a technologist, the idea of an 8.9" laptop was intriguing. When I first bought my first smartphone 3 years ago, what I really wanted was a tiny laptop with an awesome battery life. I looked into UPMC laptops, but the problem was the cost (2000+ dollars) and the battery life (2 hours). If I bought one it had to replace my real computer because of the cost, and there was no way I was going to have as my only computer something that small.

Asus, recently, invented the netbook category. Essentially what they did was to look at the OLTPC (one laptop per child) guys and figure out how to make that product at a higher price point but with all of the functionality you really want.  And in doing that, Asus made a lot of money and it may mean that I won't be buying a new smartphone.

On a personal note, this is a lot like in the 80's when the Amstrad PC was released. Up until the Amstrad, the average Greek could not afford a PC. So computers based on the Z80 etc were very popular. When the Amstrad came out at a slighly higher price point than the Z80's but still ran all of the PC software, the Z80's etc  disappeared over night. 

So I got my netbook and discovered that the flash drive they had installed was slow. Not slow in a kinda-sorta-slow, but slow in a why-did-i-ever-buy-this-product slow. I wasnt' sure if I was ready to give up on the category entirely, so I bought a faster (more expensive) flash drive (really a mini pci-e module) to see if it was really the flash or something else.

Well it turns out that my new netbook is fast, and fun and furious with it's new storage.

If the performance delta between drives is so fantastic, (I've never seen such performance delta's between equivalent hard drives) this tells me that  software is going to determine the Flash experience.

More interestingly, it validated my faith that just putting in a flash drive into your system is about leveraging someone's else's IP. The real magic will happen when flash get's integrated into the virtualization layer directly. 

Tweets From Last Week

Tue 03 Feb 07:44:59	A response about spindle randomizers and flash http://tinyurl.com/b9zba9
Thu 05 Feb 07:53:23	Dark black kettle, chuck, calling NetApp pot black.
Thu 05 Feb 07:53:43	First it was real fiber channel, now it's frankenstorage!
Thu 05 Feb 07:54:21	This from the company that hopelessly distorted the term unified storage.
Thu 05 Feb 07:55:04	*running screaming for the hills*
Thu 05 Feb 08:10:32	@storagebod read http://tinyurl.com/ahuvn5
Thu 05 Feb 08:11:21	@storagebod the NX4 is not unified storage. And for EMC to call it is, well, jaw dropping.
Thu 05 Feb 08:11:58	@storagezilla It's not just about fixing the device, 'zilla, it's about how you manage it. And it's not about a GUI.
Thu 05 Feb 08:14:29	I couldn't resist jumping in.. http://tinyurl.com/ckden2
Thu 05 Feb 08:15:51	And to be fair to Chuck, his rant is not about v-series and RAMSAN, this is about v-series bcs that technology is far more disturbing ...
Thu 05 Feb 08:32:33	@storagezilla the GUI is an essential component of unified management but not sufficient.
Thu 05 Feb 08:40:20	@Storagezilla Too long to twit, but if you read my blog you'll see what I mean. Same way to do replication, provision and manage storage.
Thu 05 Feb 08:41:18	@davegraham imagine if those customers actually knew what they were missing.
Fri 06 Feb 12:54:39	NetApp has Better Than Real FiberChannel (tm), I couldn't resist.... http://bit.ly/TVW59
Fri 06 Feb 12:55:04	But regardless it is interesting to see our very different view points on storage architecture.
Fri 06 Feb 16:55:09	@Storagezilla It's not about cache. It's about how to use flash and disk to make storage more valuable.

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

MAD blog: Chuck demands customers stop buying frankenstorage

I never thought I would see the day where Chuck Hollis would admit that Frankenstorage was a term (a term I believe coined by NetApp to describe the NX4 and EMC VTL).

I never though that I would see Chuck tell customers to not buy Frankenstorage. Good for you Chuck, I thought. Finally calling a shovel a shovel instead of a Lexus.

I thought can it be? Can EMC's VP of Marketing (or Spin?) finally be telling customers something truthful about unified storage? Like how using four different technologies for provisioning and data protection does not unified storage make?

Nope.

It's the damnable v-series. The damnable v-series. That's the piece of technology he hates. A piece of technology he went out of his way to distort facts about. And it's not because of our support of the TMS device, but because you stick a v-series in front of a Symm or a CLARiON and you suddenly need less disk. And that saves money. Not funny money but real dollars and anything he can say or due to blunt that point he will say and do. And if all you're using the Symm for is disk drives, you need less of their software. And that has got to hurt.

Of course the v-series and RAMSAN combo is not a unified storage device. But EMC has yet to ship any device that meets a reasonable definition of such a device. And for Chuck to lecture about any term, after all he coined Real FiberChannel(tm), is darn funny. So unlike the the original Frankenstorage, other than hardware management, all of your day-to-day operations are done consistently in a uniform way. You know using the ONTAP tools. But you've read that rant of mine before.

So I suddenly realized what the issue was: Professional Jealousy. Dr. Frankenstein is pissed off that someone else knows how to put together a device made up of multiple hardware devices and add value to customers.

And not only can we do that, but we can make our systems simpler to manage and adminsister than what the good doctor produced.

Hmm...

MAD Blog: Flash and the 'storage randomizers'

updated to make sure that people realize this is a MAD blog

Chuck Hollis remarked the other day that EMC's approach to flash was, at an architectural level, superior to NetApp's. He dismissively described everyone else's approach as storage randomization and that storage randomization was inherently wrong-headed.

I was intrigued by that comment because the rationale he gave did not mesh with my understanding of the physical properties of flash.

First let me attempt to explain the difference Chuck draws between EMC and NetApp.

NetApp's approach to storage can be described with this diagram:

Essentially we take a collection of similar disks, FC or SATA, combine them into a bigger container, an aggregate and then provision volumes from which applications consume storage.

As data gets written we allocate physical blocks on demand.

EMC's approach can be described with this diagram

EMC's logical storage unit has blocks that are allocated up front to the physical disk block. In effect, there is no layer of indirection between the disk block on disk and the logical block that is presented to the application.

The argument Chuck made was that if you consider Flash storage, the performance of Flash would make any latency introduced by the virtualization layer of the array unacceptable.

The problem with that argument is that what you can do with disk, namely permanently associate a logical block of storage to a physical block of storage is not how flash must be used.

Now it turns out that an annoying property of Flash is that a particular disk block can only be written to a fixed number of times.

In an EMC context, if you directly mapped block 500 on the flash disk, to a storage unit at some point in time the writes to the disk block would fail because of the inherent physical properties of flash.

Flash disk drive vendors know about this so here's what they do:

What they do is they add their own layer of virtualization above the raw flash so that the logical block they present to the array can be remapped under the covers of the array. Given how traditional array's work, they have to do this because the direct mapping technology would destroy flash drives at too fast a rate.

In some corners this technology is called wear-leveling.

So if we go back to the EMC architecture Chuck was talking about we have the following picture:

The logical storage container that the storage admin has assigned to a physical flash disk block is actually a logical disk block sitting on top of the disk drive vendors virtualization layer.

1. If there is no layer of virtualization between the raw  flash and the logical container then you will wear out the flash too fast.
2. If you don't have your own, you have to rely on the FC SSD (or enterprise flash drive) vendors to provide it.
3. If the Flash Vendors are providing the virtualization layer what is your value? Are you just a bump in the wire between the disk and the client?
4. There is no direct mapping like there is in disk, so it's unclear what the value of providing that direct mapping is.

If we look at NetApp's approach our goal is to integrate the flash into our virtualization layer directly by exploiting the fact that WAFL can do much of the same function that flash virtualization does. You can check out Val's post for more detail on that.

So I don't know whose architecture is necessarily superior, but I am not sure I buy into the argument that NetApp's architectural approach is by definition inferior and by definition poorly suited to Flash like Chuck claimed.

Have and eat your cake.

One of my co-workers, Vikram Shukla, sent me this picture of the cake from our party to celebrate being voted #1.

One of those rare cases where you can have and eat your cake.

Deeper into the scheduler

When I wrote about the failure of hybrid multi-processor CPU schedulers, I made the surprising leap that I saw the same failure occurring in the storage space. Namely that multiple conflicting virtualization layers in the storage stack are creating significant inneficiences.

But before I get to the storage discussion, I think it's worthwhile to explain my point about the hybrid scheduling a little bit more. But before I do that, let me try and explain what a CPU scheduler is and does.

So what is a CPU scheduler?

Wikipedia has it's definition, which I'll copy because I think it's a good one:

The scheduler is concerned mainly with:

• CPU utilization - to keep the CPU as busy as possible.
• Throughput - number of process that complete their execution per time unit.
• Turnaround - amount of time to execute a particular process.
• Waiting time - amount of time a process has been waiting in the ready queue.
• Response time - amount of time it takes from when a request was submitted until the first response is produced.

More specifically the scheduler is the piece of software in an operating system that performs those tasks.

But in English?

As a user of a computer, you have a bunch of things you want to accomplish: you want to surf the web, you want to write comments in blogs, and you want to send email to your buddies. To perform each of those tasks you use a variety of programs. For example, you may use FireFox to surf the web and Outlook to send email to your buddies.

To perform any task, the program you want to use has to run. To run the program has to use CPU resources and memory resources.

In any modern operating system (OS), the CPU and Memory resources are not available to your programs without going through the OS. The OS decides who and how much of the physical resources your programs can use.

Why?

After all you want to use Firefox, and you're god of your computer, why not just give all of the CPU and all of the memory to Firefox?

Because it turns out that there are lots of other things that you want to run at the same time, for example, you want to get notifications from Outlook about your meetings, and you wnat notifications from your IM that you're buddy is online, and you want to respond to that email while you're downloading that program from the internet.

So ideally you would like to have all of those things going on at the same time.

But there is a problem. You want ten different things to go on at the same time, but you only have one physical processor! Remember, for anything to happen the programs have to use some CPU time. And if there is only CPU and only one program can be using the CPU at a time, you can't do more than one thing at a time!

Enter human fraility.

Human beings are very slow. Consider a web page. After the page is downloaded we take several minutes to read the page's contents. And during that time, the computer is effectively just sitting around doing nothing.

Well, wait-a-sec, that's when I can run all of those other programs (like Outlook and IM). During the time, when I am just wasting the computer's time trying to respond to the information it just produced, the computer can use the CPU to do other things, like see if I have a meeting invite or check to see if my buddy is on IM!

And that's what the OS scheduler does, it checks to see if there is any other program that wants to run and gives the CPU to that other program (which is what we mean by the term CPU Utilization).

Well okay, so far so good, but I am the computer's god, and I want the computer to always give CPU time to the program I want to run, not to some other johnny come lately.

And that's the other thing the scheduler does. Not every program is created equal. The program the user wants to use are the highest priority programs. Other things are background tasks that have to give way for the user's program. So, in our example, when Firefox needs to run, the CPU scheduler gives the CPU to Firefox, because it a higher priority program. And this is what we mean by Response Time.

But there is one more thing the CPU scheduler does, suppose I have two programs of equal priority. Let's say the mp3 ripping software and my web browser. And let's say that I want both of them to use as much CPU as possible. In this scenario the CPU is never idle, it's running at 100% utilization.  In that case, what the CPU scheduler does is make sure that both programs, my mp3 software and my browser, get a fair share of the CPU so that their respective waiting time is in proportion to their priority.

In the spirit of the day, CPU schedulers virtualize the real CPUs! By virtualizing the CPU, the throughput of the system as a whole improves as idle CPU's are made busy.