Lumpy Dell and Benchmark FUD

Edward "Lumpy" Stevens was a cricketer from the time when bowling was performed underarm or underhand, rather than the modern technique of roundarm bowling. Lumpy was famous for his ability to find lumps and bumps in the pitch, and consequently his bowling confused as the ball careered erratically towards the batsman.

For Honest Lumpy did allow
He ne'er would pitch but o'er a brow

Nowadays, cricket is a much more sophisticated game. Pitches are flatter, bowlers are faster, and umpires are on the watch for ball tampering and other activities that give the bowler an advantage. And underarm bowling is not part of the modern game of cricket.

So it’s time for the umpire to send Lumpy Dell back to the pavilion.

Dell Bowls Underarm

Every day, I get concerned sales and systems engineers emailing me about some document that our customers have been given by the competition.

“I attach a new document that Dell/EqualLogic is handing out to customers. Can you put together a response?”

The document is entitled

“Guide to comparing storage performance efficiency for business applications by leveraging Microsoft sizing tests”

In case anyone from Dell is reading this and needs a visual reminder, here’s the front page.

The contents are as long as the title, at 17 pages. But what is it about, and why would a sales colleague be looking for help? Dell’s words;

Microsoft wants to ensure systems are properly configured for running Exchange. The Exchange Solution Reviewed Program (ESRP) allows vendors to validate to Microsoft’s satisfaction sizing of storage performance and capacity for Exchange.

This, to be clear, is not one of Dell’s ESRP submissions; there’s nothing wrong them. This is a separate document that Dell give their customers.

The conclusions are going to be blindingly obvious, but I’ll repeat them anyway;

Microsoft-approved results of each vendor’s self-testing illustrate several advantages for EqualLogic:…

Comparisons Galore

It’s about everyone else’s ESRP submissions. This massive piece of marketing FUD directly compares and contrasts ESRP results from just about every vendor that’s ever published one.

EMC, HP, IBM, LeftHand, NetApp, Hitachi. In dense tables with helpful explanatory footnotes, the document details the lot. Dell again;

• IOPS per disk for most vendors
• Backup and log replay performance for most vendors
• Example results of each major vendor’s primary native SAN testing
• Examples of tier-2 SAN offerings

Then a point by point analysis of each submission and its shortcomings when compared to Dell/EqualLogic.

What a tragic waste of effort.

ESRP Is Not a Benchmark

Cheekily, Dell/EqualLogic’s FUD department have included a handy section called

Locating the key data inside ESRP reports

It is, of course, missing the one vital piece of information. Pick up any ESRP submission from any vendor; here’s one of Dell/EqualLogic’s. They all have the same boilerplate text. So pick up the link pay close attention to page 18. Here’s what it says;

The ESRP Storage program is not designed to be a benchmarking program. Its tests are not designed for achieving the maximum throughput for a given solution. Rather, they are focused on producing recommendations from vendors for the Exchange application. Therefore, the data presented in this document should not be used for direct comparisons among the solutions.

It’s Not Cricket

It might be a good idea for Lumpy Dell to re-read the rule book. No-one wants to be publically outed as underhand.

[updated 10 Oct; Skip (see comments) pointed out that this blog wasn’t very clear on which document I was criticizing. I’ve made clarifications.]

Comments

[Skip]

But isn't this what us customers want to see?

Benchmarks are all good and well in the lab but are rather meaningless in the real world.

Application X will never perform at the levels of Benchmark Test 123.

Rather than debunking another vendors results, perhaps it would be better to explain your own.

[Alex McDonald]

Skip, I see the confusion.

There's nothing wrong with Dell's ESRP submissions. There's nothing wrong with ours either. But they can't be compared and cotrasted side by side the way Dell has been doing.

The document I'm referring to is not the Dell ESRP document; this is a separate document they send to customers, of which I've shown two pages in the graphics. It's called

Guide to comparing storage performance efficiency for business applications by leveraging Microsoft sizing tests

It's sent to customers like you with the intent of misleading.

They're doing that by pretending that the ESRP submissions are benchmarks. They have taken several vendor's ESRP submissions, extracted parts of them into this document, and then compared and contrasted the results, mainly focused on performance.

Microsoft is quite clear on this. ESRP submissions "should not be used for direct comparisons among the solutions". Dell are doing exactly that, and are being underhand in using them this way.

We do explain our benchmarks and our ESRP submissions to customers. They are very valuable to customers in explaining how we might support example Exchange environments, and, along with best practices and assessment of specific customer needs, we can come up with an Exchange solution that works, both in terms of performance and cost.

I just wish Dell would explain theirs the same way, and stop sending out this extra and highly misleading document.

Hope this helps.

[Steven Schwartz - The SAN Technologist]

Not to nit pick, but WHERE on that document does it say DELL?

That is a legacy Equallogic paper, if you want to pick on pre-Dell Equallogic, but to "claim" this is still being done by Dell is just trying to throw FUD.

This would be just as similar as me FAULTING NetAPP for the Decru e-mail to Neoscale customers.

http://www.drunkendata.com/?p=853

[Steven Schwartz - The SAN Technologist]

Oh, also, I should say GOOD JOB! This was very good document dissection, even if it was a bit dated.

Pulling specific information out of a document and using it out of context! I really can't fault you for it, since I of course have done the same with NetApp Technical Reports!

[Alex McDonald]

Steven; are you absolutely sure you want to pursue this line? I can send you the Dell branded slides that go with this doc too if you want.

And where am I pulling information out of context, please? You have obviously found the document in your stash, so just refer me to the page number. If yours isn't dated Monday, June 30, 2008 6:28:00 PM, I'll send you a copy.

You're right on one count; the doc hasn't been rebranded from EqualLogic. But then neither have the ESRP documents been rebranded Dell.

[Steven Schwartz - The SAN Technologist]

Alex,

Which are we talking about? Just because someone saved the document you are referring to on June 30, 2008, that doesn't mean it was created/published on that date, nor in that form.

ESRP documents, have been on the Microsoft Website since they were created...and posted...prior to Dell acquisition. (ESRP v2.1 posted UNDER EQUALLOGIC NOT DELL:

* EqualLogic PS Series 65,000-User Storage Solution for Microsoft Exchange Server 2007 (Jan 22, 2008)

Which if we look at the close date of Dell's purchase, was prior to acquisition.

As for of out of context...posting 2 pages, doesn't give the whole picture of the document.

con·text (kän′tekst′)

noun

1. the parts of a sentence, paragraph, discourse, etc. immediately next to or surrounding a specified word or passage and determining its exact meaning to quote a remark out of context
2. the whole situation, background, or environment relevant to a particular event, personality, creation, etc.

I think I was pretty clear in what I meant by that statement, using the formal definition of CONTEXT.

[Steven Schwartz - The SAN Technologist]

Oh and in case, the date of acquisition closure comes into question:

http://www.eetindia.co.in/ART_8800500024_1800006_NT_ac1dea70.HTM

Cheers!

[Alex McDonald]

Steven;

The document was created Monday, June 30, 2008 6:28:00 PM. It's a copy of an older document that EqualLogic were pumping out, with updates for the Dell EqualLogic PS5000. I don't recall Dell reselling EqualLogic kit and calling it the PS5000. You tell me.

The Dell/EqualLogic slides match the document content.

ESRP documents are on your website. Microsoft's website points direct at the website www.equallogic.com, which I presume is now under Dell management.

If context still bothers you, and you want me to post you the whole doc, please give me the go-ahead and I'll do so.

I do this stuff for a living, btw. I'm 100% sure of my facts. Are you?

[Steven Schwartz - The SAN Technologist]

Ok, this has just become silly.

ESRP documents are all stored by the vendor who has created them.(thanks for bringing that into question). In this case, Equallogic.com hosts them, as when they were published, they were published by Equallogic Inc., yes we all know Dell now owns Equallogic Inc. so yes your statement about Dell would be correct, however, that doesn't change the time-line of reality, nor the publish date of those documents.

So am I 100% sure of "your facts"? Nope, they are "your facts".

Am I 100% sure of "my facts" yes. I have documents that have been distributed via e-mail and sales people, would I ever post them? NO WAY! why? Because unless they are publicly referenced, why would anyone trust me over the original vendor? They are exactly that FUD!

Also, nothing in the ERSP documents linked to by the Microsoft website claim any comparison.

The document you have screen captured and posted images of isn't a controlled document, and cannot be confirmed for authenticity or date of origin.

If you'll notice, anytime I've picked on a NetApp document I give the NetApp link for it, it is an official NetApp document.

If this document is a public document, linked to by Equallogic or Dell, post the link to the documents you are referring, so they can be examined by anyone.

Have a great weekend!

[Alex McDonald]

Steven;

Now it's not a controlled document, and Dell aren't responsible because it "cannot be confirmed for authenticity or date of origin". How lawyerly. And convenient.

But it's OK for your company to create and update this stuff, branded with a company logo that you own, and have your salesforce give customers any old rubbish they like under cover of a form of corporate blindness because "Its Not On An External Link."

Yeah, riiiight. Way to go, Dell.

It's cool by me, because it's your reputation. Get the sales staff at Dell to keep sending this out, and using the Dell branded powerpoint that has the same misleading story. Leave lots of copies behind.

Then every time I get either asked for my comments on the doc or ppt, I can point them straight here to read your weasel words.

I'd send you that link, btw, but I can't seem to get through to your FUD skunkworks CIFS share. You know where it is though.

I'm ignoring the rest, because you didn't bother to read what you wrote first time you commented.

[Skip181]

Well unfortunately I am not a rocket scientist.

Based upon my limited skill and insight, it seems to me put simply if I were deploying my Exchange on a SAN then based upon the raw material in the ESRP report then NetApp would need significantly more drives to support a similar number of users - isn't that correct?

I don't have time, patience nor the money to listen to a P.....g contest like the one you two have gotten into here...suffice to say a solution test scenario to me is far more intelligible to digest than an academic Benchmark.

Regards and goodluck

[Alex McDonald]

skip181; in that case, I'll keep it short.

No.

[Alex McDonald]

That was perhaps a little too brief :-)

As I said before, we explain our benchmarks and our ESRP submissions to customers. Then, along with best practices and assessment of specific customer needs and use of some sophisticated sizing tools, we come up with an Exchange solution that works, both in terms of performance and cost.

A solution that meets your requirements uses just the right number of disks.

[Val Bercovici]

Dell/EQL's competitive team amuses me to no end. Their tactics seem to continue to campaign for inclusion in the dictionary definition of "amateurish".

First it was comparing EQL Windows RAW LUN's to NTFS on NetApp. Yup, I'm sure many PS series customers buy into the simplicity of RAW LUN's! :)

Now it's comparing an ESRP report they pulled (which somehow got 174 illegitimate ops/disk) instead of the 159 ceiling they currently publish - against outdated NetApp results.

For those still counting, our FAS3170 submission for ESRP got a sterling 189 ops/disk - and as per Microsoft - comparing vendors in this manner using ESRP is USELESS.

So whenever a vendor pulls this type of invalid workload comparison stunt on you, kindly ask them to move on or show them the door. You'll be saving everyone a LOT of time and unnecessary anguish.

Dell once had the guts to publish SPC-1. There can only be one reason why they shy away from that much more legitimate comparison benchmark workload today. That didn't keep NetworkWorld from publishing the truth about relative performance:

http://www.networkworld.com/slideshows/2008/072808-test-iscsi.html
(See Slide 8 for the performance results)