Extensible NetApp: Thinking outside of the box

MAD blog: Chuck demands customers stop buying frankenstorage

I never thought I would see the day where Chuck Hollis would admit that Frankenstorage was a term (a term I believe coined by NetApp to describe the NX4 and EMC VTL).

I never though that I would see Chuck tell customers to not buy Frankenstorage. Good for you Chuck, I thought. Finally calling a shovel a shovel instead of a Lexus.

I thought can it be? Can EMC's VP of Marketing (or Spin?) finally be telling customers something truthful about unified storage? Like how using four different technologies for provisioning and data protection does not unified storage make?

Nope.

It's the damnable v-series. The damnable v-series. That's the piece of technology he hates. A piece of technology he went out of his way to distort facts about. And it's not because of our support of the TMS device, but because you stick a v-series in front of a Symm or a CLARiON and you suddenly need less disk. And that saves money. Not funny money but real dollars and anything he can say or due to blunt that point he will say and do. And if all you're using the Symm for is disk drives, you need less of their software. And that has got to hurt.

Of course the v-series and RAMSAN combo is not a unified storage device. But EMC has yet to ship any device that meets a reasonable definition of such a device. And for Chuck to lecture about any term, after all he coined Real FiberChannel(tm), is darn funny. So unlike the the original Frankenstorage, other than hardware management, all of your day-to-day operations are done consistently in a uniform way. You know using the ONTAP tools. But you've read that rant of mine before.

So I suddenly realized what the issue was: Professional Jealousy. Dr. Frankenstein is pissed off that someone else knows how to put together a device made up of multiple hardware devices and add value to customers.

And not only can we do that, but we can make our systems simpler to manage and adminsister than what the good doctor produced.

Hmm...

MAD Blog: Flash and the 'storage randomizers'

updated to make sure that people realize this is a MAD blog

Chuck Hollis remarked the other day that EMC's approach to flash was, at an architectural level, superior to NetApp's. He dismissively described everyone else's approach as storage randomization and that storage randomization was inherently wrong-headed.

I was intrigued by that comment because the rationale he gave did not mesh with my understanding of the physical properties of flash.

First let me attempt to explain the difference Chuck draws between EMC and NetApp.

NetApp's approach to storage can be described with this diagram:

Essentially we take a collection of similar disks, FC or SATA, combine them into a bigger container, an aggregate and then provision volumes from which applications consume storage.

As data gets written we allocate physical blocks on demand.

EMC's approach can be described with this diagram

EMC's logical storage unit has blocks that are allocated up front to the physical disk block. In effect, there is no layer of indirection between the disk block on disk and the logical block that is presented to the application.

The argument Chuck made was that if you consider Flash storage, the performance of Flash would make any latency introduced by the virtualization layer of the array unacceptable.

The problem with that argument is that what you can do with disk, namely permanently associate a logical block of storage to a physical block of storage is not how flash must be used.

Now it turns out that an annoying property of Flash is that a particular disk block can only be written to a fixed number of times.

In an EMC context, if you directly mapped block 500 on the flash disk, to a storage unit at some point in time the writes to the disk block would fail because of the inherent physical properties of flash.

Flash disk drive vendors know about this so here's what they do:

What they do is they add their own layer of virtualization above the raw flash so that the logical block they present to the array can be remapped under the covers of the array. Given how traditional array's work, they have to do this because the direct mapping technology would destroy flash drives at too fast a rate.

In some corners this technology is called wear-leveling.

So if we go back to the EMC architecture Chuck was talking about we have the following picture:

The logical storage container that the storage admin has assigned to a physical flash disk block is actually a logical disk block sitting on top of the disk drive vendors virtualization layer.

1. If there is no layer of virtualization between the raw  flash and the logical container then you will wear out the flash too fast.
2. If you don't have your own, you have to rely on the FC SSD (or enterprise flash drive) vendors to provide it.
3. If the Flash Vendors are providing the virtualization layer what is your value? Are you just a bump in the wire between the disk and the client?
4. There is no direct mapping like there is in disk, so it's unclear what the value of providing that direct mapping is.

If we look at NetApp's approach our goal is to integrate the flash into our virtualization layer directly by exploiting the fact that WAFL can do much of the same function that flash virtualization does. You can check out Val's post for more detail on that.

So I don't know whose architecture is necessarily superior, but I am not sure I buy into the argument that NetApp's architectural approach is by definition inferior and by definition poorly suited to Flash like Chuck claimed.

MAD Blog: Onaro is not as much rubbish!

I couldn't help but laugh when I read Martin's blog about ECC.

When I first started working in the storage management space, i tried to understand who the competition was and understand why they were successful.

It was only after I experienced ECC in all of its pointless glory, did I truly appreciate how much opportunity for innovation there was in this space. And it was precisely because of that opportunity that I worked in storage management for about 5 years. One of the things we tried to do at NetApp was to try and build products our customers wanted to use. Products that solved real problems. And I wish I could say we had the perfect product, but that would be foolish. I can say that there were a lot of customers who really liked what they got. I hope some of them didn't think it was rubbish...

But I was delighted to read in a later post that Onaro may be useful. That was indeed high praise from someone so obviously and justifiably frustrated with the state of the art in vendor SRM tools.

When I first met the Onaro guys i was genuinely impressed with what they had. This was a very smart team, with a very cool product underpinned by some very interesting technology that solved a problem that, frankly, EMC and ECC should have solved a years ago.

To this day, Chuck's comments about NetApp's Onaro acquisition remind me of this parable.

MAD Blog: HP not letting it go ...

There is a specter haunting HP's Karl Dohm, the specter of NetApp's sale force snookering customers about the real value of NetApp storage.

Patrick mentions the true test, i.e. that there are many happy NetApp customers who are running Exchange.  There is truth to this of course, but it isn't a good basis of comparison because every major array vendor has happy Exchange customers.  However, its reasonable to say that these installations can't know what they don't know. 

Karl can not accept the notion that the value NetApp brings to the market is real. After first trying to prove that NetApp can not run on Exchange:

The problems we are talking about here are the core of WAFL, and are clearly not easy to fix - or they would be already fixed.  NetApp is not unique is having problems of course, all array vendors have their strong and weak points.  But to assert that WAFL has no weaknesses around fragmentation, performance, and capacity utilization defies common sense.  The old wounds are there for a reason.

Karl had to back peddle and admit:

I'm not saying you can't run Exchange successfully with NetApp.  In fact I'm sure you can.  The question looking for an answer is whether the user gets good value in choosing NetApp to run Exchange.

So reading Pat Cimprich's latest response was particularly gratifying.

Look, the FUD that HP is currently throwing out there is age old FUD that EMC has been throwing out for a while.

After 10 years of watching an artiste like EMC spread FUD, watching this ham-fisted attempt at spreading the same FUD is, well, embarrassing. And having to respond to it, is well irritating.

A Response

Couldn't resist linking to this blog from Pat Cimprich of Avanade which is a response to Karl's comments on WAFL.

Progress of a kind.

When I wrote my series on WAFL, I was hoping to cure the storage industry of a virulent form of Fear, Uncertainty, and Doubt  (FUD) centered around WAFL performance.

The story, typically, goes like this:

1. WAFL is a file system and hence not a real storage system
2. WAFL performs great but over time the performance degrades
3. When it degrades it gets much worse than everyone else's
4. And worse there are all these latency spikes
5. Because WAFL is a file system (cue spooky music!)

I figured, if I killed point 1, then the rest of this FUD would die...

So imagine my surprise when i read Karl'sblog on WAFL

What I didn't count on was the virulence of this FUD.

Instead of ridding the industry of the FUD, the FUD changed into:

1. WAFL is this unique thing that nobody understands
2. WAFL performs great but over time the performance degrades
3. When it degrades it gets much worse than everyone else's
4. And worse there are all these latency spikes
5. Because WAFL is this thing nobody understands

Well, that's progress of a kind. 

Previously the argument would have begun with an appeal to general misunderstandings of how SAN and WAFL worked together. Now the argument is based on an appeal to the basic lack of information about our system works, that allows the author to argue whatever point they want, and then leverage a synthetic benchmark that proves whatever the author wants.

But back to Karl and his blog...

Karl was making the following bold assertions based on his research:

FAS systems not suitable for Exchnage

But of course, Avanade's white paper would disagree.

FAS systems perform poorly over time

But our SPC-1 longevity results would differ

A large number of NetApp customers have been tricked by our sales force

After all, that is the only possible explanation for our market share given how inappropriate our system is for the kinds of workloads we've been selling quite well into.

The problem that Karl has is that without appealing to some basic architectural understanding that we can all share, he is forced to create synthetic workloads that prove his point. And the problem is that there are real workloads with real data that prove the opposite point.