Controversy: NetApp Outperforms EMC in SAN Database Benchmark

We just released benchmark results showing that our FAS storage systems outperform EMC’s CLARiiON on SAN database workloads. For details, see Brian Pawlowski’s blog. The quick summary is that a NetApp FAS3040 beat an EMC CX3 Model 40 on SPC-1, which is an industry standard benchmark that measures OLTP (Online Transaction Processing) performance. Our system cost less, had fewer disks, and beat the EMC by 24%. With snapshots enabled on both systems, NetApp was three times faster. Here’s the chart:

Why the controversy? EMC has never posted any SPC-1 results, so we had to run the benchmark ourselves. We did follow EMC’s best practice document for the CLARiiON, and we did have the Storage Performance Council independently audit the results. But still, the fact that we ran the tests ourselves caused concern. For instance, Chuck Hollis, the VP of Technology Alliances at EMC, raised questions about the integrity of the auditor, calling him “an infamous part-time ‘administrator’ for the Storage Performance Council” and saying that “It seems to be a part time gig for him to make a few bucks.” Chuck also raised questions about why we would do this: “The only reason you'd spend the money to buy the equipment and run the tests is to put your competitor in a bad light. I think most reasonable customers would figure that part out.”

It is fair to ask why we did this, so let me share our thinking:

First, to continue improving our SAN and database credibility. Given our NAS origins in the early 1990s, many customers and analysts have been skeptical of NetApp’s SAN capabilities. Results like this show today’s reality. Steve Duplessie at the Enterprise Strategy Group commented that “Netapp appears to have legit block performance, and shouldn't be dismissed because people (like me) presume it can't be true.” Chuck Hollis is a vocal skeptic of NetApp’s ability to play in SAN and database environments, so it shouldn’t surprise him that we want to refute his claims. (See here for a brief history of our benchmarking efforts over time.)

Second, to showcase our snapshot performance. Snapshots help customers improve backups and archive old data, and writable snapshots (FlexClones) let customers completely rethink their database test and development strategy. Unfortunately, snapshots in most storage systems are unusably slow. With NetApp, performance dropped only 3% with snapshots on. With EMC, performance dropped by a factor of three. I freely admit that our goal in focusing on snapshot performance was to “put our competitor in a bad light.” I think that’s fair because EMC’s snapshots really are painfully slow in real-world use. On the other hand, we expected our non-snapshot performance to be about the same, or maybe even a bit lower, given that our system is less expensive, has fewer disks, and uses RAID-6 instead of mirroring. Winning there was a pleasant surprise.

Third, because Chuck asked us to. In his blog entry on SPC, Chuck said: “We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.” I don’t promise always to follow Chuck’s advice, but I think it’s important to recognize good ideas no matter where they come from!

One key take away from this result is that turning on a simple feature like snapshots can radically change performance. Don’t let a bad experience with EMC’s snapshots scare you away from NetApp’s.

Let me close with a final word on benchmarks. Any honest vendor will agree that benchmark results are sometimes misleading, and that you should examine the details carefully. We believe that SPC-1 effectively simulates OLTP workloads, and we used real-world configurations based on each company’s own best practices documentation. But despite our best efforts, I stand by my argument in a previous blog entry that you should admire and respect great benchmark results, but also be careful.

 

 

The following SPC-1 results have been posted at www.storageperformance.org:

NetApp FAS3040 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00057

NetApp FAS3040 (with Snapshots):
http://www.storageperformance.org/results/benchmark_results_spc1#a00058

EMC CLARiiON CX3 Model 40 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00059

EMC CLARiiON CX3 Model 40 (with SnapView):
http://www.storageperformance.org/results/benchmark_results_spc1#a00060

All comparisons are current as of January 29, 2008.

 

Comments

[Taylor Allis]

Dave - bold move on NetApp's part. I think this kind of debate ultimately benefits the consumer - so thanks - Taylor

[Kevin Closson]

"Chuck said: “We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.”

...I think Chuck should read their EULA. I'd wager dollars-to-donuts there is a clear, concise stipulation in there against publicly disclosing benchmark results taken from an end-user licensed copy of the CLARiiON offering...but then, I'm just a gadfly.

[Ecio]

Hi Dave, i've had in the past the opportunity to use a Netapp FAS3020C and i appreciated it (so im not an EMC zealot). My question is, what do you think about Chuck's comments on this blogpost:
http://chucksblog.typepad.com/chucks_blog/2006/12/benchmarketing_.html
about the "What we found" part? (where Chuck says that you do test with system not fully loaded and that performance goes worse and worse days after days?)

unfortunately the emc doc that should explain this is not available...

[Mindaugas]

I felt strange reading this post. We have 3040 and it is slower than anything we have. Be it HP MSA1500, internal drives, HP EVA.

[Bostonian]

Dave NetApp's version of snapshots is a very creative hack that takes advantage of the Write Anywhere File System. When you perform a write somewhere else on the disk you simply retain the previous block and mark it as a snapshot. However, this causes the production file system to become fragmented. While this may not be a problem for CIFS and NFS data it can severely impact read performance on Databases such as SQL, Exchange, oracle.
www.storageguru.org

[Me]

Why do you ALWAYS compare yourself against EMC as if that is your only competition? There are many other storage vendors that you are competing against so why not also talk about those ALONG with EMC if you must.

[carrie]

Je vais devoir m'exprimer en français:

je vais personnellement suivre de très près votre société car j'ai longtemps travaillé pour EMC2 en France!

J'espère que tout ce que vous déclarez se vérifiera... A suivre!

[John]

You just forget to mention a crucial thing : you turned off the write cache on the EMC CX3-40 to run the benchmark ! Turn it on and do the bench again otherwise the results are meaningless.