NetApp raises the bar in SAN performance reporting

On Tuesday, January 29, 2008, NetApp published a couple of SPC-1 benchmark results  for the mid-range FAS3040 storage system.  With our highly efficient Snapshot(TM) feature on the FAS3040 with RAID-6 (RAID-DP) enabled, and our thin provisioning FlexVol architecture, the results clearly show that enabling data protection comes at a very modest cost in performance. The  features enabled on the NetApp storage are recommended in our best practice deployments and represent a realistic customer configuration. By using the independently audited, industry standard benchmark SPC-1 to report performance effects of these features, we hope to raise the level of conversation around storage system performance measurement and reporting.

Characterizing the performance impact of NetApp's Snapshot technology is the significant result - there is only a 3% drop in throughput from a base of 30,985 ops/s.

By way of comparison, as part of this study we measured a competitive system from EMC, the  CX3-40 system. In sharp contrast to the NetApp result, the EMC CX3-40 system configured in a standard mirrored RAID configuration - following EMC best  practices - suffers more than a 60% decrease in performance when their snapshots are enabled.

You can find more information about the results here.

The team at NetApp that did this work (thanks everyone!), and the product development group, are justifiably proud of these results. Such information on the costs and tradeoffs around standard storage management technologies is needed to make informed decisions about  storage deployments.

Benchmarks by their nature  tend to be controversial. But they are simply tools - some better, some worse - used to measure performance and compare two or more systems. A benchmark result is a combination of several factors. First, all benchmarks define a specific workload - a model, sometimes actually more often synthetic - to subject the test system to. Better benchmarks reflect attention to calibrating the workload with actual application workload results. The next factor is the configuration of the system under test - here is where a lot of controversy lies in benchmark results. "Tuning a configuration" to achieve good results is expected - it's when the tuning gets increasingly unrealistic - or forbid, benchmark specific - that things can get confusing. Which brings me to uniform run and reporting rules. Good benchmarks strive for useful comparisons by having run and reporting rules sufficient to allow others to reproduce the result. Finally,  peer review or auditing of the results prior to publication is required to maintain the integrity of their respective benchmarks. The SPEC SFS and SPC-1 benchmarks represent two of the more useful and rigorously defined benchmarks for storage performance measurement.

What is very cool about NetApp results like this is that we have striven to configure the system in much the same way as a customer would (following our best practice recommendations).  These results use much of our core feature set (RAID-DP, Flexible Volumes, Snapshot feature).  They are run using an industry standard workload, and audited by an independent auditor.  They are real.

The FAS 3040 result is the only SPC-1 published result using a RAID-6 implementation at this time. In fact, it's one of a small handful of published results using anything other than (the more costly) traditional RAID 1+0 mirror. NetApp's RAID-DP is an extremely efficient RAID 6 implementation that is enabled by default on our storage systems - including our StoreVault S300 product for small and medium businesses. (In the interest of full disclosure, I must say I am not only an employee of Network Appliance, but a fervent customer of one of our resellers and proud owner of an S300 - with a two year support contract:-) Further, this result is on our industry leading storage software stack that provides a Unified Storage Architecture (with the ability to support NAS, iSCSI and FC SAN access), and sophisticated thin provisioning capabilities that allow you to right size your LUNs and data containers for your application deployment. Because we have a single unified storage stack, it is possible to extrapolate these results to other deployments (say iSCSI or NAS connectivity) for a similar workload.

The other aspect of the reported result is the characterization of performance overhead when snapshots are enabled.  Snapshots, for online backups and application recovery points, are a commonly supported feature today in many storage offerings - but is usually implemented as a copy out operation  of unmodified data to a separate volume as opposed to NetApp's Snapshot in-place implementation. Snapshots were done more frequently on the NetApp FAS 3040 (every 15 minutes) vs. the EMC array (every 1 hour). Further, the NetApp  storage retained three snapshots rolling the oldest one off as a new one was created, versus one snapshot retained in the EMC measurement. Given the performance cost of the copy out snapshot mechanism in the CX-3 40 it was felt this was a more realistic frequency and depth.

Our belief is that the NetApp performance under SPC-1 load (NetApp result dropped only 3% vs. the nearly 60% performance drop in the measurement in the EMC result) reflects  that the Snapshot features in  Data ONTAP are more efficient because they  were designed into the product from the start as a fundamental feature of our solution.

So, what does all this mean to you? Realize that your mileage will vary - your workload will undoubtedly have different characteristics than that defined by SPC-1. But broadly speaking, the results provide apple-to-apple comparisons of two  competitive mid-range storage arrays and show the viability of an efficient RAID 6 solution vs. a standard RAID mirror configuration, and the dramatic performance overhead difference that you are likely to encounter with different snapshot architectures and implementations.

So what happens next?  There are a few additional aspects of these results that I will discuss in this blog in the near future. Longer term, we'll continue to improve our products, to make them more effective at solving more customers problems. And we'll publish benchmarks from time to time, allowing customers to understand the competitive context. Other critical storage features, such as remote mirroring and disaster recovery, bear characterization. We will publish when we have interesting things to say - and work with standard benchmarks and the organizations that develop them to encourage measurement methodologies around realistic customer configurations and concerns.

Comments

[MKent]

Results on realistic configs such as these are always helpfull in starting the wheat from chaff seperation process.

[Dimitris Krekoukias]

There have been rumors that when running multiple features in a NetApp box then performance suffers.

A realistic test would be to have OTHER apps sharing the array (on other spindles), as is usually the case.

Have the box do NAS, iSCSI AND FC (since that's when people typically choose NetApp - they want one box to do everything).

THEN show the performance delta as each feature is enabled.

Obviously hard to do and maintain the SPC results but it would be a worthwhile addendum.

D