Benchmarking – The art

So far I’ve been able to blog about the good and bad forms of Bench-Marketing. Now let’s spend some time on the right way to benchmark systems for maximum storage consultant, analyst, reseller and customer audience benefit.

Bench-Sizing

Storage Bench-Sizing is still a bit of an art, but also very close to being a science, since most vendors publish via many well-accepted and public processes. SPECsfs, SPC, ESRP and others are all good examples.  Bench-Sizing helps vendors highlight the practical strengths of their products in a credible manner. That implies adherence to at least the following criteria:

• Picking a high-volume product line to benchmark vs some exotic high-end of your product portfolio which only a precious few can afford
• Ensuring the system being tested is configured following published best-practices for production deployments, thereby …
• … avoiding “benchmark special” tuning of the system
• Picking a mainstream workload, representative of a majority of customer deployments
• Ensuring a neutral 3^rd party has reviewed or ideally audited the results for accuracy

Some competing storage vendors fulfill a few of the Bench-Sizing criteria above, but none do it as thoroughly or completely as NetApp. Practically all of the public benchmarks NetApp has published satisfy all of the criteria above. We do not bother with “benchmark special” configurations to win bragging rights in drag races. We don’t disable strong data protection options or avoid rich value-add features in the systems we report on. We simply run our storage solutions as customers would expect.

Most importantly, we always ensure someone other than our own engineers actually reviews the results for trustworthiness.  That may be a formal auditor, or perhaps a 3rd party verification authority.  This helps avoid nasty instances of Imagineered benchmarks that have no place in rational storage evaluation discussions.

This concludes my introductory series exposing Benchmarking the Game, the Art and the Alchemy. Always try to understand the nature of any benchmark you review with these definitions in mind and you’ll be sure to avoid wasting your time when it comes to actually using the system you’re reviewing.