Can a Lada Really Do 90mph?

Today's email question; I'm being asked how many read IOPS a NetApp system can do from cache. I know why the information is needed, and I know who the competition is, without reading any further.

It's HP.

A Jeremy Clarkson moment is upon me, I fear. This nonsense number has set me thinking about the Lada, an unloved and unlamented car that you could buy in the UK for the price of a bag of chips.  

The Lada promised much. Well, at least the brochure did. Glossier than a Lada's paintwork straight out of the factory, the brochure claimed that this brick-shaped, tank-like mode of transport could do 90 miles per hour. Even though it had the drag coefficient of an open parachute.

Brochure Benchmarks

Benchmarks are subjective things at the best of times, and there's all-round disagreement on what numbers are meaningful. Short of running the production workload for which you're buying the system, there's really no such thing as the ideal or perfect benchmark.

My colleague Val Bercovici covers benchmarking ups and downs in some detail; bench-marketing, imagineering and bench-sizing. There are some numbers he and I really can't understand as having any value, and this IOPS from cache number is one of them.

HP must be the only storage vendor in the universe that measures, publishes, promotes, uses or even cares about this number. Here's a line from the brochure for one of their models, the EVA4400 (others in the range are similarly described);

Maximum IOPS > 140,000 IO/sec*

It's the quoted "top speed", and it's the only performance figure quoted there. Something's not quite right with this figure, as it's a maximum 96 drives when fully loaded, and I calculate around 25k IOPS with fast FC disks.

This has to be IO from cache, surely. The asterisk points to a link in a footnote,

*More EVA4400 performance information is available at: www.hp.com/go/eva4400, Resource Library.

And after much searching around, a performance benchmark (not the external, auditable kind, you understand, but the HP's own figures).  Yes, it's a cache figure. Helpfully, HP inform us...

Because disk overload is a possible consequence of using cache performance numbers for capacity planning, HP best practices do not recommend this measurement.

[...]

When reporting performance numbers for an array, HP has found that the most useful numbers for customers are those associated with workload configuration and other characteristics that reflect actual business conditions.

I couldn't agree more.

Running Out of Puff

In reality, out of the showroom and on the road, the Lada was as slow as an asthmatic ant with heavy shopping. 90mph was only achievable in some very special circumstances. A well-known joke, referring to a popular method of disposal employed by frustrated owners of rubbish cars everywhere, goes as follows;

Q: Where are you if a Lada passes you at 90mph?
A: At the bottom of a cliff.

Before EVA lovers come banging on my door demanding blood, I want to make it crystal clear. I am not suggesting that the EVA4400 is anything like or anywhere near as bad as a Lada. Or that you should test it by throwing it off a cliff.

But just remember when you're browsing the brochure; it won't do 140,000 IOPS on the road. 

. 

Comments

[open systems storage guy]

It's fairly well documented that benchmarks perform differently in different workload situations. Most people either stress test devices for comparison under their own workloads, or use the generic workload agnostic SPC-2 (for IO/s) or SPC-1 (for MB/s).

SPC is contentious because many vendors build application level prefetching algorithms into their boxes, and SPC is specifically designed to prevent that from working. It is useful for a basic bake off though, and as far as I remember, HP posts benchmarks.

Where are your benchmarks? If you want to dispute this claim, post them at www.storageperformance.org.

[Geert]

What about:
http://www.storageperformance.org/results/benchmark_results_spc1/#netapp_spc1

Btw- where are HP's submissions for EVA...?
Wait a minute, here they are:
http://www.storageperformance.org/results/benchmark_results_spc1/#hp_spc1

But wait again; they're submitted in *2002* and *2003*, any more recent submissions? Nope. Now I wonder why.......

[Alex McDonald]

Hi OSSG

As Geert points out, the SPC benchmarks for NetApp are on the SPC website, along with the (very old) EVA benchmarks for systems HP no longer sell.

Lack of current SPC benchmarks is one issue, but that's not my complaint here.

It's the "top speed" figure HP quote. No-one can *ever* achieve that number "on the road" with an EVA4400.

1. It's not stated that the number represents IOPS from cache in the spec sheet.

2. It's not recommended by HP, but that isn't clear either until you get well through the performance document.

3. And when you get there, it says "Because disk overload is a possible consequence of using cache performance numbers for capacity planning..." Possible?!?!? So probable as to be a racing certainty is more accurate.

Weasely words and numbers.

Update: Hey, would you believe it, the website http://h18006.www1.hp.com/products/storageworks/eva4400/specs.html now has NO asterisk on the Maximum IOPS to point you at the performance document. This is it, plain and baldly stated;

-- Maximum IOPS > 140,000 IO/sec

No asterisk, no caveats. That's even *more* misleading than before.

[Val Bercovici]

Hi OSSG,

A couple of quick points:

1. SPC-1 is actually the one known for IOps whereas SPC-2 is the one geared for MB/sec.

2. NetApp has not only actively published our popular mid-range FAS3000 series extensively, but we've worked with the SPC-1 committee to push the boundaries of the benchmark to deliver even more valuable real-world info for customers to consider. Namely, we've added Thin Provisioning, Snapshots and extended-length (>3hrs) testing to our results.

More details here:
http://blogs.netapp.com/exposed/2008/01/a-brief-history.html

[Geert]

Btw, thinking of it - I may not be a math expert, but to my knowledge > means "more than". So how much "more" IOPs "than" 140,000 could a EVA4400 theoratically do? Like, 1 million? 2 million?

Besides all the valid points Val and Alex make, this figure just means, well eh... nothing, really.

[Lee Razo]

This video reveals the dark truth behind how HP achieved those benchmark results:

http://www.youtube.com/watch?v=fT8EqIQwgf8&session=P18yYD6nTwhIT3eJxSwu9jJTUMZOyo2hWqlexox7pB-C2DeaDXVyhfyljxKMvcDR4ZxkvBzRR1LALTj2yYfgBldko1m6_XBjh94PCnXwJQxe2yfSoeJqFokrpCWa_DvvbJhb8_UWE_VFrxpc-VvPopAkAcwEAyly5hrHGjkrWY-VKIZ9mO8mQN6GzU61DdFAnBE8LnGsTu_ZsqleToLSZkvSiDvsIxUIAfocgQIbxIJk-3Cjz_yISBXcmBuFjSNFUH_wQ5DzC-cSukKgiB8esX3Fy3TL-OE_qThqe7FiGUYgb54T9_AOtRQc1pCgqKeD