NetApp - Exposed

Live by the Blog, die by the Blog

As expected, Chuck wasted no time in issuing the predictable response to our SPC announcement from yesterday.  We knew he would regurgitate the tired old defense of SPC not being a relevant benchmark, etc...

So what surprised us?

Well it seems as if Chuck forgot his own open SPC invitation to the storage community.  So let me quote the most interesting section and save you a trip over there:

"... We've never done an SPC test, and probably will never do one.  Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it."

Perhaps he never expected anyone to make the investment in setting up a CLARiiON to pass the rigorous SPC audit process?  Yes, that same audit process that:

1. forces you to strictly follow all performance best-practices, and
2. has now been passed by nearly every storage vendor that matters ... except EMC themselves.

As per my prior post on this topic yesterday, NetApp has over a decade of fully transparent and sensible benchmark configurations published for anyone to review.  More recently, the VeriTest and SPC reports consistently prove how our allegedly "architecturally inferior" FAS arrays clearly outperform their CLARiiON counterparts.  That mounting volume of credible evidence certainly presents challenges for those in the industry who still attempt to dismiss NetApp's performance in real-world production customer deployments.

What else surprised us?

It seems as if in his haste for a pithy response, Chuck also forgot to read the actual press release itself.  He claims he'd like to us to focus on more customer value.  So I guess:

• Delivering 23% better performance,
• At a lower price,
• With 15 less drives,
• Offering 68% higher capacity utilization,
• Protecting against all possible double-disk failure scenarios,
• Capturing consistent online recovery points without sacrificing performance,
• All while still enabling rich functionality such as efficient & space-saving Thin Provisioning

... doesn't deliver value to customers?  Hmmm.  With that type of logic at EMC, maybe VMware wasn't the reason EMC stock was down over 5% yesterday.

A brief history of time (in the world of NetApp benchmarks, now including SPC-1)

Stephen Hawking’s seminal book builds on a long tradition established by popular scientists such as Carl Sagan and Albert Einstein.  All of them broke into mainstream pop culture by helping demystify incredibly complex topics into layman’s terms.  If any topic in IT cries out for demystification it would be the game, the art, and the alchemy of benchmarking.

Each of these NetApp results below has a unique “rest of the story”, yet there’s also some startling similarities between them all.  I’ll elaborate on one untold story behind each of these results across subsequent blog posts – and expose some disturbing patterns contrasting NetApp’s approach to benchmarking with our competition’s.

My list of NetApp performance reports

TPC-C:  The first time a networked storage vendor had ever proven leading price / performance worthy of publishing a result under that workload.   AFAIK – after all these years, and an additional joint result with IBM & MS SQL2005, that distinction is still unique to NetApp.  Competing storage vendors will complain that this is a server-focused benchmark.  That’s what I would do as well if my array couldn’t satisfy the strict price / performance criteria required to augment rather than hinder TPC-C results for that demanding enterprise-class workload.

SPECsfs:  NetApp engineering cut their teeth on this one, and we still take pride in leading all comers in the key areas of lowest response times as well as highest number of operations per second.

VeriTest:  This organization certifies results published by various IT suppliers.  It’s important to me that VeriTest (now owned by Lionbridge) maintains most of these same IT suppliers as customers – ensuring their objectivity in order to maintain their very survival.  From a technical perspective, NetApp introduced snapshot performance comparisons for the first time in our VeriTest comparison reports against EMC.  The resulting exposure was very revealing to the industry and has helped many customers and partners size their systems factoring in this important functionality.

ESRP:  Microsoft strongly discourages using these results for benchmark-style comparisons.  And for good reason, as many storage vendors can’t resist the urge to do just that.  Yet given the wide latitude of Exchange configurations supported and lack of formal auditing procedures, Exchange architects truly need to read beyond the headlines and fully absorb the details of what was published before drawing conclusions comparing any two reports.  Nevertheless I’m a big fan of this program since it provides a meaningful indicator of various storage solutions for sizing purposes – if you do your homework.

TR-3521:  I’ve nicknamed this “the accidental benchmark” since we had no plans of publishing it ourselves.  Yet EMC’s clever work of Imagineering about 16 months ago prompted our performance engineering team to publish true results under this workload.  Ironically this also became the genesis of our formal SPC-1 efforts by inspiring our desire to add the highest possible level of transparency and credibility to this effort.

Avanade:  The strong momentum of our alliance with Avanade prompted them to formally test NetApp’s FAS arrays in order to help their own consultants size the storage component of their Exchange, SharePoint, SQL Server and other Microsoft Server Infrastructure solutions.  What they produced is one of the most impressive storage performance and functionality reports I have ever seen.

SPC-1:  Beepy covered it well today.  Industry firsts include usage of RAID-6 (and resulting dramatic advantage in usable capacity) in any independently audited storage performance benchmark, as well as the inclusion of multiple snapshot copies measured with the same objective scrutiny.  Perhaps the most controversial “industry first” is accepting Chuck Hollis’ invitation to benchmark the EMC CLARiiON and formally publishing the SPC-1 results after an independent audit.

Shock & Awe

I am in complete awe of EMC’s marketing prowess when it comes to pre-announcements like this week’s Flash SSD support for their stagnant Symmetrix DMX-4 product line. There is no doubt the clever strategic leaking of the press release to the Wall Street Journal (which does not have the technical prowess to  properly see thru the marketing hype) followed by analyst and media briefings helped fuel the fire of positive exposure. Extensive blogging wisely helped further fan the flames. Kudos to EMC for orchestrating such an event.

But why am I still shocked?

Because the so-called industry pundits were actually acting more like industry parrots. The way I see it:

• EMC got away with spinning high performance despite their failure to disclose even a single benchmark result or reference customer confirming the benefit in their environment.
• EMC got away without conceding the true costs of this solution when deployed in a recommended best-practice config. For example has anyone considered that microsecond latencies for persistent storage means parity calculation overhead matters? Does that mean EMC will recommend mirroring these SSD’s for maximum performance? What would that cost?
• Given the internal bottlenecks in the DMX architecture and subsequent limits in numbers of SSD’s which can be configured, has EMC delivered metrics showing their expensive flash storage option delivers better cost/performance than merely adding more RAM to the host?
• From a storage utilization perspective, doesn't the (unnamed large U.S. financial institution) customer quote in the WSJ article confirm that EMC forces customers to buy more capacity than they need so that they can short-stroke the necessary level of performance out of it all?

Mojo, Mario and Gary got it right. Respectively, they pointed out this is merely a marketing exercise which needs to be validated via proper Bench-Sizing and demonstrated to have better resulting manageability than superior approaches such as more scalable caching.

EMC won this battle, but who will win the war?

By pre-announcing a forthcoming flash storage option ahead of their rivals, EMC has created much-needed halo effect for the aging DMX platform. EMC has also exposed the potential of flash technology to the enterprise storage market.

However “Shock & Awe” doesn’t win wars. I predict storage vendors who can successfully deliver the benefits of flash technology to the broadest possible set of customers with a consistent management framework will be the ultimate winners.

True students of WAFL and Flash should be positively giddy at the potential for the two technologies to come together in an almost perfectly complementary way. I’m very excited about our roadmap in this regard, but I’ll have to wait for another day before I can share anything formal with you on this blog.