EMC's (botched) Storage Capacity Utilization Magic Trick

One of my favorite movies is The Prestige.  Among the many virtues a movie needs to make it to my "favorites list" is the ability to stay enjoyable after repeated viewings, and perhaps even reveal a bit more each time you see it.  This movie delivers on both counts.

The plot cleverly revolves around the 3 parts or acts of "every great magic trick":

1. The Pledge
2. The Turn
3. The Prestige

Ironically much like seeing a movie over and over again, EMC's Machiavellian competitive tactics against NetApp have become ubiquitous enough that the entire storage industry has been subjected to repeated examples of the same FUD.  And all of NetApp's competitors have enjoyed EMC's show :)

Yet far more interesting to me is what each new slant on EMC's FUD claims exposes about their overall motivation, as each attack increases the transparency of their overall strategy.  But that's the subject of a future post :)

Onto our magic show

Michael Caine opens and closes this great movie by narrating this haunting yet captivating passage (first two quotes on top).  It perfectly describes Chuck's little exercise last week.  Let's break it down for the purposes of our discussion.

The Pledge

Every great magic trick consists of three parts or acts. The first part is called "The Pledge". The magician shows you something ordinary: a deck of cards, a bird or a man. He shows you this object. Perhaps he asks you to inspect it to see if it is indeed real, unaltered, normal. But of course... it probably isn't.

That would be Chuck's basic setup to his comparison:

• a relatively standardized use case that most vendors document with specific recommendations (e.g. Microsoft Exchange)
• a decent number of usable disks, say, 120
• an idealized 146GB disk capacity
• configurations done in accordance with vendor-supplied recommendations for Microsoft Exchange environments

 

Chuck's storage "Object" in this sleight of hand includes a claimed "recommendation" of RAID5 using 8 data drives for every parity drive, known as a 8+1 configuration.  I'll spare the usual rant against this ridiculous approach, but suffice it to say this RAID setup completely contradicts EMC's own published best-practices for Exchange 2007 on CLARiiON (pg. 15, document #H4060, Oct. 2007) which state:

It is generally accepted that RAID 1/0 is a better choice for random-write environments like Exchange

Ditto for EMC's Exchange on DMX recommendation (pg. 7-49) which states:

As a matter of Best Practice, EMC generally recommends RAID1 to be the primary choice in RAID configuration for reasons of reliability and availability and performance for high-end deployments of Microsoft Exchange Server.

In fact, all of EMC's DMX, CLARiiON & Celerra submissions to Microsoft's Exchange Solution Review Program state the exact same thing!  For brevity and clarity, I'll skip over the fact Chuck also completely ignored EMC's own drive formatting / right-sizing of about 10% as well as the well-known CLARiiON 5 drive vault reservation policy.  Apparently Chuck has his perfect Magic Trick Pledge all laid out now awaiting ...

The Turn

The second act of the magic trick is called "The Turn". The magician takes the ordinary object and makes it do something extraordinary. Now you're looking for the secret... but you won't find it, because of course you're not really looking. You don't really want to know. You want to be fooled.

In Chuck's case, EMC's competitive archeologists had uncovered a couple of IBM N Series OEM and NetApp StoreVault plus Exchange whitepapers which contain outdated information.  Due to editorial cycles those particular documents have not yet incorporated the well established longstanding capacity-oriented best-practices for NetApp FAS arrays. The "EMC Turn" in this Magic Trick would have you assume (quite incorrectly that) NetApp FAS arrays still require 100% Space and Fractional Reservations for every LUN!

Of course had EMC's competitive team bothered to review more applicable FAS best-practices (released over 2 years ago!) or the SnapManager for Exchange 5.0 Admin Guide, they would have found plenty of scenarios where it is perfectly acceptable to safely tune NetApp Space or Fractional Reservations down from 100% all the way to 0%, depending on your requirements.

I encourage everyone to study those whitepapers, but for those in a hurry allow me to summarize:

• To maximize application availability via instant restores while maximizing storage efficiency, NetApp enables optional Thin Provisioning of snapshots alongside LUN's (also themselves Thinly or Thickly provisioned) on the same FlexVol
• In order to "first, do no harm", NetApp FAS arrays default to the most conservative setting of 100% reservation for any LUN, before space for snapshots is accounted for on the same FlexVol
• When data is overwritten on the LUN *and* snapshots are enabled, FAS storage admins can configure two related policies to automatically and safely deal with potential contention between LUN's and snapshots for physical space:
  • FlexVol AutoGrow - which automatically provisions additional capacity for the FlexVol containing the affected LUN(s) upto the limit of the underlying RAID group aggregate.  This policy is often defined by the storage admin.
  • Snapshot AutoDelete - which automatically deletes snapshots by various pre-defined criteria such as oldest, newest, custom name or application-scheduled.  This ensures LUN's always have physical space to write to, and does not allow snapshots to grab a LUN's physical space.  For maximum application fidelity, this policy is often defined by the application admin.
  • Database Dismount - As a failsafe, if the storage and/or application admin(s) accidentally misconfigure their automated policies, this intelligent application-aware feature proactively dismounts any databases on the affected LUN(s) in order to ensure application consistency and prevent any possible unintentional data corruption.
• Therefore the recommended setting for fractional space reservations with NetApp LUN's depends entirely on the business requirement for how many snapshots are to be retained in order to satisfy data recovery or cloning criteria.
• As a result, acceptable values for safe NetApp FAS fractional space reservations are as low as 0% (which would mean very frequent snapshot deletions at the capacity limit) or higher as needed.

True to form, Kostadis does a great job of summarizing the history of why unlike on other arrays, NetApp customers often consider NetApp snapshots valuable enough to consume primary storage right alongside actual primary data.  Kostadis also reviews how our Snapshot Thin Provisioning has evolved away from the initially conservative 100% space reservation model, to the granular modern and safe offering today which enables fractional reservations ranging down to zero, meaning no unnecessary capacity overhead where it's not needed.

To paraphrase the key passage of the movie again "of course EMC's competitive teams weren't really looking.  Chuck didn't really want to know.  The EMC fanboys wanted to be fooled!"

Tomorrow we'll conclude with some movie spoilers (you've been warned!) and the much anticipated final part of all great Magic Tricks - The Prestige - where we'll also review & validate NetApp's proposed configuration for Chuck's Pledge based on our findings above.

Comments

[Edwin]

I'd rather not wait for tomorrow, so could you net this out for me? How much more than 34% efficient would NetApp be using Fractional Reservations set to 0% and 10%?

[Val Bercovici]

Hi Edwin,

At FSR=0, our storage efficiency wins Chuck's magic circus sideshow with 34+37=71%. OTOH - with FSR=10, we're down in the 68% territory, less than the magic 70%.

That's if you trust Chuck's numbers actually deliver 70% for a CLARiiON :-)

[Alex McDonald]

Beyond Exchange, the space problems for a CLARiiON further multiply.

VMware? No deduplication of VMDKs on a CLARiiON. EMC doesn't do it. That's according to Chuck Hollis, so you don't take my word for it.
http://chucksblog.typepad.com/chucks_blog/2008/07/have-i-seen-thi.html

Oh dear. That's the downside of them thar "EMC real LUNs (tm, pats pending)".

Real capacity issues.

Nice debunk, Val. As they say where I come from, it's a belter!

[FastFreddy]

StoreVault? StoreVault!

You've got to be effin sh*tting me!

Chuckie lays out a >120 disk requirement - and then compares to a single shelf SMB system capable of spinning only 12 drives?

Just how desparate is EMC to make this trick work?

[neto from Brazil]

Hi Val,

This is neto from Brazil

How are you?

Pledge...

Fallacy.

http://technet.microsoft.com/en-us/library/bb738146(EXCHG.80).aspx

or ...

http://blogs.netapp.com/databases/2008/08/snapshot-snapsh.html

see the last picture... :-)

All the best my best friend

neto
NetApp - I love this company!

[Insider442]

This is more than just a random volley from EMC. Our sales droid from EMC just dropped off a "confidential" sales doc which seems to be a verbatim clone of Chuckie's blog on this subject.

What to do? Why would Chuck publicly post a confidential EMC document? Why would their sales guys share it so liberally outside the company?

If there's enough demand here and people think the contents are already effectively in the public domain, I'll find a way to share it online.

[Val Bercovici]

Thanks for all the comments folks!

FastFreddy - welcome! Yeah the link to the old StoreVault paper also caught my eye, but I didn't want to waste too much bandwidth on it. I agree it is yet another indication of the level of desperation @ EMC to try and make their weak efficiency argument stick. And as Alex points out above, we haven't even begun to discuss mature thin provisioning technology for multi-tenancy requirements or today's killer app - PRIMARY DEDUPE! :)

Insider442 - welcome as well! Now IANAL, so please make sure you're not violating any NDA as you handle that confidential doc from EMC. OTOH - If you get a professional legal opinion that those contents are now in the public domain via Chuck's blog posts over the past few days, then I'm sure we'd all like to see it. But remember, when in doubt just leave well enough alone!

There seems to be plenty of interest in this topic throughout the storage blogosphere, so I'm hoping my upcoming follow-up post tonight will be equally well received...

[Drew Meyer]

Hey Val-

That StoreVault (now S family) paper represents my attempt to state as clearly and openly as possible the approach we take to data protection. I'm quite flattered that it is being used to compare to an EMC box!

Fundamentally Data ONTAP works the same way in all locations. If you understand it once, you 'get it' for all of your systems. The StoreVault paper is a great starting place but, as you well know, we make a number of default assumptions for that product and it is by no means an accurate portrayal of the extent of customization possible.

Just thought your readers might want to know.

Drew Meyer

[Val Bercovici]

Hi Drew,

Great point! Let me be clear that I am a big fan of that paper (from way back in the day when Jon Toigo also highlighted it). The candor and advice in that paper was refreshing and certainly appreciated by the typically less than enterprise storage-savvy SMB audience for which it was intended.

My main point above was how sad it is that EMC felt the need to abuse the candor and intended audience of that paper completely out of context for their "Pledge" which consisted of 200 odd spindle configurations (with about a dozen shelves!) often found at the top of most storage vendor's modular mid-range arsenals.

But you've probably got the better perspective on this. EMC has just validated that the NetApp S Family is more than a match for the AX4 and is indeed comparable to their CLARiiON CX4 family! :)

[Val Bercovici]

Even though the HP EVA perspective is not part of my response, let me provide a link to their position for completeness, as some of you are now coming over here from other sites / blogs:

http://www.communities.hp.com/online/blogs/datastorage/archive/2008/08/29/emc-distortion-about-capacity-efficiency.aspx