NetApp - Exposed

The Recession of 2008 - Summarized in 8 Short Minutes

Like a slow motion train wreck, I can't stop watching CNBC lately in order to witness what I hope is (only this once) in-a-lifetime worldwide stock market crash.  The implications for the storage economy are vast, but before I pontificate on the topic in more detail later this week, I encourage a quick read of Dave's and Marc's respective takes.

Candor is a NetApp hallmark and I am proud of how our executives are resisting the shallow and superficial tendency of some competitors to feign indifference to the burning of Rome around them.

NetApp's Prestige

On Monday of this week we reviewed EMC's botched attempt at a multi-vendor storage efficiency comparison and yesterday we debunked some lingering misconceptions around provisioning capacity for LUN's on a NetApp SAN.

It was probably not Chuck's intention, but he's actually helped bring incremental attention to NetApp's industry-leading storage efficiency via some interesting disclosures by EMC and Industry Experts along the way.

Time to get back to the movie analogies and wrap-up what we've learned as a result of it all

Tales from the crypt - Sizing and Provisioning a NetApp SAN

Yesterday I promised to conclude my analysis of EMC's (botched) Storage Capacity Utilization Magic Trick with the NetApp Prestige.  Allow me to maintain the suspense a little longer while I indulge a bit more in the Turn, courtesy of some illustrative screenshots.

Some would have you believe configuring 99.99% LUN reservations on a NetApp SAN would result in storage admin's being immediately covered by the umbrella of a mushroom cloud explosion ...

EMC's (botched) Storage Capacity Utilization Magic Trick

One of my favorite movies is The Prestige.  Among the many virtues a movie needs to make it to my "favorites list" is the ability to stay enjoyable after repeated viewings, and perhaps even reveal a bit more each time you see it.  This movie delivers on both counts.

The plot cleverly revolves around the 3 parts or acts of "every great magic trick":

1. The Pledge
2. The Turn
3. The Prestige

Performance as a measure of usable capacity

EMC’s StorageAnarchist took the bait and asked the logical question I was hoping he would when I blogged on the relative complexity of calculating usable capacity on feature rich and resilient enterprise storage arrays. He was at a disadvantage though, because he is operating under the spell of EMC’s competitive FUD which is (sadly for him) far removed from customer reality.

A simple question deserves a simple answer (even if the questioner won't like it)

I do owe the Anarchist thanks though, for picking such nice round numbers to work with, since that will help. To summarize, basically he wanted to know what 100TB raw (200 x 500GB disks) would look like as usable under a true enterprise (OLTP) workload. As a matter of fact, his question has actually been asked and definitively answered for both of us earlier this year.

The answer is at least 62TB usable (conservatively, see better number below) for NetApp and 38TB usable for EMC (as well as most of the rest of the mainstream primary storage vendors).

In January 2008, NetApp published a benchmarked result for a proven OLTP workload agreed upon by most of the enterprise storage industry. This public benchmark result was rigorously and independently audited. As with all highly respected and credible benchmarks, it’s open to peer review by experts from competing vendors in our industry.

So, what's the catch?

Calculating Usable Capacity - Rocket or Nuclear Science?

Marketing claims recognized by independent experts

About a month ago, B&S editor Mary Jander published a blog covering the recent Green Enterprise IT awards, which featured NetApp and 7 other winning vendors.  NetApp was cited for improving primary (gross) usable capacity from 40% (already above the Unix/Windows industry average of 30%) to 60% for our internal production systems which feature all advanced data protection and replication features enabled.  Note that the judges reviewed NetApp's infrastructure for this award before we released our unique FAS deduplication functionality for primary as well as secondary data sets.  Our internal analysis shows deploying FAS deduplication increases NetApp usable capacity numbers even higher.

NetApp was also recognized by the Uptime Institute in that same study for reducing our total number of storage systems from 50 to 10, cutting our number of racks from 25.83 to 5.48, and decreasing direct power consumption by 41,184 kWh per month.

It's not easy being green

However over in the comments section of that B&S blog today, one user seemingly wasn't impressed by the resulting 60% usable capacity number, so I offered up a short explanation (comment #6).  As Paul Harvey likes to say, here's the rest of the story...

A brief history of time (in the world of NetApp benchmarks, now including SPC-1)

Stephen Hawking’s seminal book builds on a long tradition established by popular scientists such as Carl Sagan and Albert Einstein.  All of them broke into mainstream pop culture by helping demystify incredibly complex topics into layman’s terms.  If any topic in IT cries out for demystification it would be the game, the art, and the alchemy of benchmarking.

Each of these NetApp results below has a unique “rest of the story”, yet there’s also some startling similarities between them all.  I’ll elaborate on one untold story behind each of these results across subsequent blog posts – and expose some disturbing patterns contrasting NetApp’s approach to benchmarking with our competition’s.

My list of NetApp performance reports

TPC-C:  The first time a networked storage vendor had ever proven leading price / performance worthy of publishing a result under that workload.   AFAIK – after all these years, and an additional joint result with IBM & MS SQL2005, that distinction is still unique to NetApp.  Competing storage vendors will complain that this is a server-focused benchmark.  That’s what I would do as well if my array couldn’t satisfy the strict price / performance criteria required to augment rather than hinder TPC-C results for that demanding enterprise-class workload.

SPECsfs:  NetApp engineering cut their teeth on this one, and we still take pride in leading all comers in the key areas of lowest response times as well as highest number of operations per second.

VeriTest:  This organization certifies results published by various IT suppliers.  It’s important to me that VeriTest (now owned by Lionbridge) maintains most of these same IT suppliers as customers – ensuring their objectivity in order to maintain their very survival.  From a technical perspective, NetApp introduced snapshot performance comparisons for the first time in our VeriTest comparison reports against EMC.  The resulting exposure was very revealing to the industry and has helped many customers and partners size their systems factoring in this important functionality.

ESRP:  Microsoft strongly discourages using these results for benchmark-style comparisons.  And for good reason, as many storage vendors can’t resist the urge to do just that.  Yet given the wide latitude of Exchange configurations supported and lack of formal auditing procedures, Exchange architects truly need to read beyond the headlines and fully absorb the details of what was published before drawing conclusions comparing any two reports.  Nevertheless I’m a big fan of this program since it provides a meaningful indicator of various storage solutions for sizing purposes – if you do your homework.

TR-3521:  I’ve nicknamed this “the accidental benchmark” since we had no plans of publishing it ourselves.  Yet EMC’s clever work of Imagineering about 16 months ago prompted our performance engineering team to publish true results under this workload.  Ironically this also became the genesis of our formal SPC-1 efforts by inspiring our desire to add the highest possible level of transparency and credibility to this effort.

Avanade:  The strong momentum of our alliance with Avanade prompted them to formally test NetApp’s FAS arrays in order to help their own consultants size the storage component of their Exchange, SharePoint, SQL Server and other Microsoft Server Infrastructure solutions.  What they produced is one of the most impressive storage performance and functionality reports I have ever seen.

SPC-1:  Beepy covered it well today.  Industry firsts include usage of RAID-6 (and resulting dramatic advantage in usable capacity) in any independently audited storage performance benchmark, as well as the inclusion of multiple snapshot copies measured with the same objective scrutiny.  Perhaps the most controversial “industry first” is accepting Chuck Hollis’ invitation to benchmark the EMC CLARiiON and formally publishing the SPC-1 results after an independent audit.

Benchmarking – The art

So far I’ve been able to blog about the good and bad forms of Bench-Marketing. Now let’s spend some time on the right way to benchmark systems for maximum storage consultant, analyst, reseller and customer audience benefit.

Bench-Sizing

Storage Bench-Sizing is still a bit of an art, but also very close to being a science, since most vendors publish via many well-accepted and public processes. SPECsfs, SPC, ESRP and others are all good examples.  Bench-Sizing helps vendors highlight the practical strengths of their products in a credible manner. That implies adherence to at least the following criteria:

• Picking a high-volume product line to benchmark vs some exotic high-end of your product portfolio which only a precious few can afford
• Ensuring the system being tested is configured following published best-practices for production deployments, thereby …
• … avoiding “benchmark special” tuning of the system
• Picking a mainstream workload, representative of a majority of customer deployments
• Ensuring a neutral 3^rd party has reviewed or ideally audited the results for accuracy

Some competing storage vendors fulfill a few of the Bench-Sizing criteria above, but none do it as thoroughly or completely as NetApp. Practically all of the public benchmarks NetApp has published satisfy all of the criteria above. We do not bother with “benchmark special” configurations to win bragging rights in drag races. We don’t disable strong data protection options or avoid rich value-add features in the systems we report on. We simply run our storage solutions as customers would expect.

Most importantly, we always ensure someone other than our own engineers actually reviews the results for trustworthiness.  That may be a formal auditor, or perhaps a 3rd party verification authority.  This helps avoid nasty instances of Imagineered benchmarks that have no place in rational storage evaluation discussions.

This concludes my introductory series exposing Benchmarking the Game, the Art and the Alchemy. Always try to understand the nature of any benchmark you review with these definitions in mind and you’ll be sure to avoid wasting your time when it comes to actually using the system you’re reviewing.