A brief history of time (in the world of NetApp benchmarks, now including SPC-1)

Stephen Hawking’s seminal book builds on a long tradition established by popular scientists such as Carl Sagan and Albert Einstein.  All of them broke into mainstream pop culture by helping demystify incredibly complex topics into layman’s terms.  If any topic in IT cries out for demystification it would be the game, the art, and the alchemy of benchmarking.

Each of these NetApp results below has a unique “rest of the story”, yet there’s also some startling similarities between them all.  I’ll elaborate on one untold story behind each of these results across subsequent blog posts – and expose some disturbing patterns contrasting NetApp’s approach to benchmarking with our competition’s.

My list of NetApp performance reports

TPC-C:  The first time a networked storage vendor had ever proven leading price / performance worthy of publishing a result under that workload.   AFAIK – after all these years, and an additional joint result with IBM & MS SQL2005, that distinction is still unique to NetApp.  Competing storage vendors will complain that this is a server-focused benchmark.  That’s what I would do as well if my array couldn’t satisfy the strict price / performance criteria required to augment rather than hinder TPC-C results for that demanding enterprise-class workload.

SPECsfs:  NetApp engineering cut their teeth on this one, and we still take pride in leading all comers in the key areas of lowest response times as well as highest number of operations per second.

VeriTest:  This organization certifies results published by various IT suppliers.  It’s important to me that VeriTest (now owned by Lionbridge) maintains most of these same IT suppliers as customers – ensuring their objectivity in order to maintain their very survival.  From a technical perspective, NetApp introduced snapshot performance comparisons for the first time in our VeriTest comparison reports against EMC.  The resulting exposure was very revealing to the industry and has helped many customers and partners size their systems factoring in this important functionality.

ESRP:  Microsoft strongly discourages using these results for benchmark-style comparisons.  And for good reason, as many storage vendors can’t resist the urge to do just that.  Yet given the wide latitude of Exchange configurations supported and lack of formal auditing procedures, Exchange architects truly need to read beyond the headlines and fully absorb the details of what was published before drawing conclusions comparing any two reports.  Nevertheless I’m a big fan of this program since it provides a meaningful indicator of various storage solutions for sizing purposes – if you do your homework.

TR-3521:  I’ve nicknamed this “the accidental benchmark” since we had no plans of publishing it ourselves.  Yet EMC’s clever work of Imagineering about 16 months ago prompted our performance engineering team to publish true results under this workload.  Ironically this also became the genesis of our formal SPC-1 efforts by inspiring our desire to add the highest possible level of transparency and credibility to this effort.

Avanade:  The strong momentum of our alliance with Avanade prompted them to formally test NetApp’s FAS arrays in order to help their own consultants size the storage component of their Exchange, SharePoint, SQL Server and other Microsoft Server Infrastructure solutions.  What they produced is one of the most impressive storage performance and functionality reports I have ever seen.

SPC-1:  Beepy covered it well today.  Industry firsts include usage of RAID-6 (and resulting dramatic advantage in usable capacity) in any independently audited storage performance benchmark, as well as the inclusion of multiple snapshot copies measured with the same objective scrutiny.  Perhaps the most controversial “industry first” is accepting Chuck Hollis’ invitation to benchmark the EMC CLARiiON and formally publishing the SPC-1 results after an independent audit.