An HP LeftHand Duplication Calculator

Our industry drives towards greater and greater storage efficiency. Green IT was the buzz in the media for a while, and although it's not getting the column inches it got last year, being efficient hasn't gone away as a business requirement.

It's a maturation effect; once the message is clear, media and CIO attention turns to other things, and the expectation is that the IT industry will get on with it and solve the problem. The storage industry has (with exceptions) risen to the challenge, with thin provisioning, virtualization of storage, larger and more energy efficient drives, and so on. Choosing the right storage system can make big savings.

• Smaller physical footprint
• Lower power and cooling
• Cheaper to buy and maintain

The list goes on. Deduplication on primary storage is the latest, and possibly greatest, contribution to really wringing every drop of space out of your storage system. Surprisingly (to me at any rate) some of these cost-saving measures have improved performance too. For example, deduplication and caching together works just as well on cheaper, slower and cooler disks as it does on expensive, hot and fast disks.

There are a number of efficiency calculators for primary storage out on the intertubes at the moment; one from NetApp, and one from EMC for NAS only -- which was originally entitled EMC Celerra Data Duplication. Although the title has been corrected, it still delivers the same poor results as before.

Leaving that aside for the moment, what the NetApp's calculator demonstrates quite clearly is that it's possible to store more data, more efficiently than ever before. Some in the storage industry don't appear to have got the message though.

HP's LeftHand.

I was confused by a video from Chris McCall, ex LeftHand Networks and now at HP. Two points about his video struck me as a little out of kilter in his positioning of LeftHand inside HP. LHN is now part of an HP "Unified Storage" division. Hold up! Unified? The "HP LeftHand P4000 SAN" is an iSCSI SAN system only. Doesn't sound terribly unified to me.

But the claim that really had me scratching my noddle was the claim that the LHN solution meets the needs of Green IT.

LHN uses something called network RAID over the top of storage nodes; they're HP servers running RAID5. Network RAID is nothing more than a fancy term for RAID5 synchronous mirroring, something that an LHN cluster needs to do because the individual storage nodes are SPOFs -- single points of failure. So copies of data on other nodes is essential if you ever expect to see your data again.

Let's cut to the chase. Here's the HP LeftHand Duplication Calculator. Unlike NetApp's space efficiency calculator, the LHN Duplication Calculator I've designed doesn't have any input fields or buttons. That's because, regardless of how small or how large your LHN SAN, it's always...

 

35% usable

Buy 10TB, get to use 3.5TB. That's about double the raw storage you'd need to buy from NetApp. No deduplication on an LHN system either, so you'll need to buy even more from HP.

LHN means more space, more power, more cooling, more cost per usable TB. Tell me, how can HP claim that this is green? 

.

[Updated 24June2009: corrected Chris McCall of HP's name]

Comments

[Lee Razo]

Alex,

I think the pie chart represents the usable capacity using Network RAID 2. What happens to the usable capacity in a multiple-node cluster with Network RAID 3 or 4?

[Alex McDonald]

nRAID3 (2 mirrors) is (70%/3) or 23%.
nRAID4 (3 mirrors) is (70%/4) or 17%.

nRAID2 will probably be the one of choice given such a dire set of options.

[marc farley]

What! No vomments from the LHN peeps out there? What's this world coming to?

BTW, I think you did a nice job with the analysis. My only question is what the 12% for rightsizing and overheads is for.

[Alex McDonald]

The %age was a guestimate based on 1TB SATA drives. Getting from the marketing numbers and adding in a fraction for checksums (this thing does checksums, yes?) and roughly comparing with NetApp, 12% seems about right. It could be a fraction higher or lower; it doesn't really make a great deal of diffference.

What's not shown is the metadata load for the filesystem. I can only guess at that; possibly 10% of the usable and network RAID, so the usable could be as low as 31% or so.

Let's say somewhere between 30% and 35%. Terrible.

HP are probably trying to work out how they hide this wart of an acquisition. Remember this "going, going, gone..." dribble from HP about NetApp usable space? Apparently

NetApp has a huge usable capacity issue in many environments that it tries desperately to hide but at the same time seems driven to confess as if subconsciously trying to purge some unresolved guilt.

It's difficult defending your position after that kind of nonsense.

[George Wagner]

Chris McCall (referred to by you as Chris Hall) never spoke of HP LeftHand being Green IT in traditional terms, but rather in terms of saving green (greenbacks).

There are many ways in which we save our customers dough. A couple of key ways are with reservation-less thin provisioning and an all inclusive feature-set. Customers surprisingly appreciate not being sold relevant features after they’ve already bought a solution.

I do love the ultimate red herring you served up, “Network RAID is nothing more than a fancy term for RAID5 synchronous mirroring, something that an LHN cluster needs to do because the individual storage nodes are SPOFs”. Come on. You know why we do what we do… It’s about storage clustering and/or scale-out storage. I believe NetApp has scale-out storage as well. It’s the high-end GX, right? Network RAID provides HA for volumes that require it within the cluster. This capability is built-in to HP Lefthand, thus saving a ton of green. I know, you do have HA features as well. They are MetroCluster and SyncMirror, add-ons that cost tens of thousands of dollars.

Most pragmatic administrators realize that a vendor’s “storage calculator” is typically built by the marketing dept, and oddly enough make their own products shine in comparison to others. I will spare your reader(s) our NetApp calculations in this response for the sake of their sanity, as I’m sure they’d rather investigate it for themselves.

There's more information on our blog regarding your misinformation. http://bit.ly/xyroz

[Alex McDonald]

@George

I've corrected Chris McCall's name in the blog. I'd encourage you to listen to his video again; start at 3:24 through the video I referenced and listen. Here's a transcript, as you appear to have developed a case of selective hearing.

Interviewer: Going into that... building on top of the power and cooling reductions and focussing on going green, which seems to be the big trend right now, can you give us a little insight into how that's taking place in LeftHand?

Chris: Yeah, sure, ... the biggest thing is it really comes down to capacity utilization. It's doing more with less, and you know, it's funny, a lot of people talk about going green, it's environmentally friendly.. you know, the way I think about it, the way most customers think about it, going green is really about saving money, it's about doing more with less, and yes, it takes less power and that's great for the environment, but really at the end of the day green means money that you're saving so it's all about capacity utilzation. And the Lefthand Architecture lends itself to extremely efficient utilization. We've got the integrated thin provisioning engine that we leverage for all of the different objects that we have in our storage. Not only for volumes but we have thin provisioning for snapshots, clones, remote copies, and everything uses that engine so we can achieve very high rates of capacity utilization and reduce the overall cost of the storage in your environment.

Lot's of mentions of "capacity utilization", but nary a mention of 35%.

"Network RAID provides HA for volumes that require it within the cluster." Sorry, George, it's got zero to do with HA. Because if I spread out a volume across two or more nodes, and I don't nRAID2 it, then when the SPOF node goes down, I loose all my volumes, all my snaps, clones and all my data. Everything on my entire LeftHand system. nRAID isn't optional; it's not about high availability, it's about protecting against data loss, a different issue altogether.

Your pricing model is nothing new. It's capacity based pricing; buy one brick, pay for the software once (I'm assuming you're not giving it away). Buy 10 bricks, pay 10 times. You're nickel and diming your customers.

Please don't spare my readers your NetApp usable calculations. Especially if the figure you come up with is anywhere north of 35%.

[Jim Haberkorn]

Hi Alex,

I couldn’t help but notice that you referred to a comment I made in regards the NetApp capacity guarantee program as ‘dribble’ and ‘nonsense’. Well, that dribble won the debate vs. you on my blog. Here is the link in case you’ve forgotten - http://www.communities.hp.com/online/blogs/datastorage/archive/2008/12/09/netapp-s-shining-moment-its-capacity-guarantee-program.aspx

However, if you really feel that my responses were not effective, or that your points carried the day, I would be happy to revisit the issue again on your blog.

Also, as long as you brought up the subject of usable capacity, maybe we can try once again to get to the bottom of this. Question: Does a NetApp filer degrade in performance as its usable capacity is used up?

I await your answer.

Best regards,

Jim

[Alex McDonald]

dribble: in various ball games, to move the ball, by repeated light kicks.

Which is exactly what you're doing; repeating the same stuff with light kicks, over and over and over...

Please, read this.. That covers your first point. Then read this. That covers your second point. Though I am willing to bet that you don't see it that way, and continue dribbling with the same ball, even though it's out of play.

Let's get back to the LeftHand 35% usable.

Why does Chris McCall think this is green and capacity efficent?And while you're at it, perhaps you can tell us; does the LeftHand filesystem degrade in performance as the 35% usable is filled up? How about a 48 hour longevity benchmark from LeftHand? Your ball.

[Jim Haberkorn]

Hi Alex,

I can see you want to continue the discussion on the NetApp 50% capacity guarantee program. Okay. I will post a response within a day both on your blog and on the blog you pointed me to as your response.

As to my second question. You didn’t answer it at all. You pointed me to a discussion that had ZERO to do with the question I asked you. Here is the question again: Does a NetApp filer degrade in performance as its usable capacity is used up?

I am still awaiting your response.

Best regards,

Jim

[Alex McDonald]

@Jim

How many links do you want? How about this one from Kostadis on WAFL performance under varying capacities.

And, no, I don't want to continue watching your handwaving on this question. You're employing the Chewbacca Defense.

LeftHand's 35% usable is the worst in the industry, by a long chalk. LeftHand SANs mean more space, more power, more cooling, more cost per usable TB than any other system out there. Tell me, how can HP claim that this is green?

I am still awaiting a response.

[Jim Haberkorn]

You and I have blogged before and surely you know by now that I am not derailed by your dismissiveness and derogatory comments, or by your pointing me to non-relevant blogs written by someone else.

Since you won’t answer the question, I will. NetApp filers absolutely do degrade in performance as capacity fills. This is proven by NetApp itself in a white paper currently on the NetApp website. It can be found at: http://media.netapp.com/documents/tr-3521.pdf . And this problem is closely linked to NetApp's usable capacity issues.

Readers can see the chart I’ll be talking about in this response by going to figure #3 on page 6 of the report mentioned above. In it you will see NetApp performance degrading by approximately 50% as capacity increases from 10% to 75%. And these are in tests conducted by NetApp engineers.

So, one question is: Why would NetApp engineering publish such a damaging performance result on its own website? The only answer I find plausible is that within the NetApp engineering community no one disputes the fact that NetApp filers degrade in performance as they fill up. And because of that they did not think twice about admitting it in a public forum. Now, the paper where the admission can be found is a NetApp response to an EMC attack on NetApp filer performance. The NetApp engineers who wrote the response were fixated on proving that they could get higher numbers out of their filer than EMC could get out of their CLARiiON. In their paper they chose not to address the significance of the fact that the NetApp filer performance curve did indeed track a significant downward arc as it used up free space. NetApp filer performance does indeed degrade as capacity is used up.

In any case, I bring this up to readers as a way of saying that despite what some NetApp marketeers would have you believe, NetApp competitors are not lying or making things up when they say that NetApp has serious performance and usable capacity issues in many environments. Here is what happens: Let’s say you are a NetApp customer and on the first day you fire up your NetApp filer, it can handle 25,000 users. (see the chart). Then, over time, as your filer fills up to 75% capacity, the number of users it can handle drops to 12,000. What will you do? Answer: You will purchase additional disks to increase the amount of free space. And there you have the beginning of a usable capacity issue.

Best regards,

Jim

[Alex McDonald]

@Jim

My apologies for not pointing out that there were a series of posts Kostadis made on the subject of NetApp performance and capacity. He's has written extensively on WAFL, and I pointed you at the start of the series. You may wish to read this further post from Kostadis on WAFL performance under varying capacities. It says;

A Better Than Real FiberChannel array like WAFL, will do very little work when the array has plenty of free space and therefore perform much better. A side effect, though, is that as the array free space shrinks, WAFL’s will have to do more work to find free space, and that more work will translate into lower performance. This is why as the storage utilization increases WAFL performance reaches the same level as other storage arrays.

I assume that the subject is now closed, as this comprehensive series of posts from Kostadis (a NetApp engineer with intimate knowledge of our systems) covers in far greater detail anything I or you could say on the subject.

In contrast to NetApp, there's a lack of technical papers and blogs on LeftHand systems for me to answer some of my questions. So perhaps you can help.

• Does the LeftHand filesystem degrade in performance as the 35% usable is filled up?
• LeftHand's 35% usable is the worst in the industry, by a long chalk. LeftHand SANs mean more space, more power, more cooling, more cost per usable TB than any other system out there. Tell me, how can HP claim that this is green?

I have other questions, but they can wait for the moment.

[John Spiers]

When using Network RAID 2 it protects you from multiple disk faults, complete array faults and site faults with auto failover and failback. NetApp can’t deliver this level of HA with auto failover and failback. Features like MetroCLuster give you data protection, but not HA, and at a lower capacity utilization than LeftHand. Can SnapMirror or MetroCluster automatically fail back, incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Ok, let’s talk capacity. All NetApp’s customers know NetApp’s storage utilization is below 50% when using best practices. But instead of re-hashing what everyone already knows, let’s do a simple calculation for a highly available multi-site SAN using MetroCluster (as a side note, you know Calvin is taking it easy on you with the MetroCluster pricing.)

Let’s say a customer has 10TB of NetApp raw storage at the primary site and they replicate that 10TB to a remote site for HA and disaster protection. Storage utilization is now 50% (10TB/20TB.) Take your 63% at both sites, and we won’t bother to include things like the space taken up for NetApp’s root volume and replication log files. 63% of 10TB leaves you 6.3TB of usable capacity replicated. This means you can create 6.3TB of data out of 20TB raw. That’s 31.5%. With LeftHand’s Network RAID level 2 you can split your SAN across 2 sites for a better HA solution and the customer’s utilization, according to you, is better than NetApp’s – 35%.

Now it’s time for some education:

Network RAID is set at the volume level. Not all volumes require the advance data protection level of Network RAID level 2, therefore utilization is typically much better if used at a single site. If you really want to get schooled let’s talk about dual-parity based network RAID and what that does for utilization. What we should really be talking about is cost and capacity utilization of NetApp GX vs. LeftHand, because that is the only product that comes close to LeftHand’s architecture. Does GX support block yet?

[Jim Haberkorn]

Hi Alex,

Okay, let’s step back and summarize. I take your response as a clear admission that NetApp filers do degrade in performance as they fill up. And according to data posted on the NetApp website, their performance drops by 50% as the filer goes from 10% to 75% capacity.

If you think that I am mis-stating your position, please speak up now.

Also, I would like to get your response to the point I made in my previous comment, i.e., that to recoup their previous performance, NetApp customers must buy more disks so as to increase free space back to where it was. True or false?

I’d also like to remind you that the reason why the question of usable capacity came up on your blog is because you brought it up. When you post comments attacking a competitor’s usable capacity, you have every right to expect a professional response in return – but you also have to expect that your competitors will question you on the same issue.

Best regards,

Jim

P.S. There are three HP LeftHand experts currently replying to you on your LeftHand questions. Please direct your LeftHand questions to them. I wish to address the other issues you brought up in your original blog, namely, usable capacity, and why you referred to my comments on the NetApp capacity guarantee program as ‘dribble’ and ‘nonsense’.

Best regards,

Jim

[Alex McDonald]

@Jim

Yes, I think you're mis-stating my position. This is the figure you should have been using (and I discussed this when I blogged about NetApp readsets and Dell's reference to figure 3 instead of figure 4), and it tells a different story;

Read the full paper for the details, but the lines to look at are the blue to 50%, followed by the green from 50% to 100%.

I'm flattered that it takes 3 LeftHand experts to address this network RAID capacity issue (although I only see George Wagner and John Spiers here), and I'll reply to the latest information shortly. John's made some interesting points.

[Alex McDonald]

@Jim

I spotted a second question, apologies for missing it. In answer to

I would like to get your response to the point I made in my previous comment, i.e., that to recoup their previous performance, NetApp customers must buy more disks so as to increase free space back to where it was. True or false?

the answer is either

1. obviously, buy more disk to increase spare capacity; that's true of any disk array, especially legacy arrays like the EVA
2. turn on NetApp deduplication and eliminate duplicate blocks (a free feature of our systems and unavailable from HP)
3. defragment the system (which kind of begs the question; does LHN have a defrag utiltiy?)

[Jim Haberkorn]

Hi Alex,

The question really is: Have you read your own white paper? And have you studied the graph you just posted?

But I must admit, I’m again, puzzled by your tactics. I suppose you think that people will look at the chart you posted and just assume that you wouldn’t be posting it if it contradicted your points. Alex, take a look at it closely. What is the graph saying?

If you really are going to try and pretend that that chart proves your points, then I will explain how to read it in my next response. However, at this point let me simply say that what the chart is saying is that depending on the level of concurrency, in tests conducted by NetApp engineers, a FAS 3050 will drop in performance from 25,000 users to either 11,000, or 13,000 as you fill up the array from 10% up to 75%. No other array that I know of suffers such a severe drop-off in perfomrance as free space is used up. And this NetApp phenomenon is at the root of its usable capacity issues.

Please do not continue to claim that that chart proves your point.

Best regards,

Jim

[Jim Haberkorn]

Hi Alex,

Now to address your second response just up above. In order:

1. Yes, Alex, if you add disks to any array, you will get extra space. But there is a big difference between adding disks for extra capacity and adding disks to restore performance. When it comes to performance, it is a common practice for customers to add disks to arrays to get INCREASED IOPS performance, but NetApp is the only storage vendor that has customers add disks to RESTORE performance.
2. Yes, there is room in this world for dedupe. In NetApp’s case the positive effect is severely diluted in most environments because of the very issues we are discussing in this blog. The NetApp technology takes away usable capacity with one hand and then with dedupe gives some of it back.
3. Defrag?? Why do you need a defrag tool on a NetApp filer? Care to explain?

Best regards,

Jim

[Alex McDonald]

@Jim

I stand by my point, and repeat;

A Better Than Real FiberChannel array like WAFL, will do very little work when the array has plenty of free space and therefore perform much better. A side effect, though, is that as the array free space shrinks, WAFL’s will have to do more work to find free space, and that more work will translate into lower performance. This is why as the storage utilization increases WAFL performance reaches the same level as other storage arrays.

I've provided as much information and evidence (NetApp blogs, reports and external 3rd party reports and benchmarks) as I have; much more information than you've ever given on LeftHand (or any other HP array for that matter). That you remain unconvinced is not my problem, since you are unlikely to ever use a NetApp system the way my customers do every day of the week, and therefore you won't see the benefits of its rich feature set.

I'd be grateful if you could continue your train of thought on an HP blog. I probably won't have much to say apart from correcting your more egregious errors as and when they arise. In responding to you here, I'm beginning to repeat myself, and it's not something I want to continue doing.

[Jim Haberkorn]

Hi Alex,

As promised, since you brought up the NetApp 50% dedupe capacity guarantee program again, here are my objections – again:
1) The program compares the competitor’s RAID-1 against NetApp’s RAID-DP. If you do the math, (14+2 RAID stripe vs. RAID-1) that means that of the 50% guarantee 43% is achieved due to forcing a comparison vs. RAID-1. Just to make it clear: What I’m saying here is that of the capacity advantage implied by NetApp to be the result of dedupe, 86% is achieved simply due to RAID without any dedupe.
2) NetApp excludes competitors’ RAID-5 from the comparison because it is not robust enough.
3) It also excludes competitors’ RAID-DP because it’s not fast enough. Are you getting the picture here?
4) It requires that 90% of the deduped data be extremely dedupe friendly. Here is a cut and paste from the NetApp program document: “There must be no more than 10% of the following data types under the program: images and graphics, XML, database data, Microsoft Exchange data, and encrypted data. Excludes workloads with high-performance requirements that require spindles; to be determined by SE/PS during sizing.” That doesn’t leave much, does it?
5) Bottom line: Strip away the self-serving RAID-1 requirement, and NetApp is only willing to guarantee customers a 7% capacity advantage over non-deduped arrays even while deduping a blatantly dedupe-friendly data set.
6) Finally, even with all these caveats, NetApp further insures it will never lose money on this program by requiring customers to purchase extra professional services before they can qualify.
7) Conclusion: This is a marketing program designed to hide something. The question is ‘What’?

If you dispute my facts of the program, please speak up. The program details can be found here: http://media.netapp.com/documents/wp-7053.pdf
Best regards,

Jim

[Just a Storage Guy]

@ Alex-

I think where you're losing Jim is easy- He's making the assumption that customers are losing performance- He isn't understanding that no one (NetApp or its Partners) size systems based on the left hand side of that graph, and then need to *add* spindles to restore performance. The sizing is based on the right side of the graph, and any of that *extra* performance from low utilization is a happy accident.

@Jim
You point about the guarantee is well taken. We in the SI community feel the percentage guaranteed should be higher or the stipulations lessened. We see anywhere from 70-95%(!!) de-duplication on VMs in customers environments. Just because they are only guaranteeing 50% doesn't mean you won't see more. YMMV.

[Mike Riley]

As a NetApp SE, personally, I hope that Jim stays on his stump about this "degrades over as file systems fill up" stuff. It keeps my customer conversations brief and light-hearted. I get to show grainy black and white photos of Bigfoot and Nessy. I even sketched out an X-Files episode with Jim hunched over his laptop under a WAFL poster that says "I want to believe." They all must be true because I put them into print myself and posted them on the Internet.

Now, for 3rd party validation, I could transmit this to the fillings in Oliver Stone's head and hope he makes a docu-drama on WAFL decay OR I can simply look at the market. (Apparently Reynold's aluminum foil not only locks in freshness but locks out logical technical proofs from NetApp engineers and 3rd party benchmarks). If you look at the market then you really have only two choices: a) NetApp sales and marketing can repeatedly fool an ever-expanding pool of customers over the past 17 years or, b) we can actually do what we say we're going to do. With customers such as BT running on average at 80% utilization rates, it really makes you wonder why they would buy more? Glutton for punishment or merely co-conspirators? Hmmmm.

Whether or not WAFL works and works well has already been proven. How it works still seems to escape the grassy knoll crowd and, you know, that's O.K. You'll never convince them. At some point, it just works to your advantage. Think about it. We can put out a conservative guarantee that says we can cut storage costs by 50% for virtual server environments. The grassy knoll crowd looks at storage savings of 50% and comes to the conclusion that we must be hiding something? Really?

O.K. all I can say is, guys, keep pumping these X-Files out. Blog hard on them and stay committed. The truth is out there.

[Jim Haberkorn]

Hello Storage Guy,

Thanks for weighing in. My original point wasn’t particularly complicated. I just wanted Alex to admit what the NetApp chart already proved, i.e., that NetApp performance does decrease, dropping by about 50%, as the filer fills from 10% to 75%. Now, some drop can be accounted for simply by the head stroke lengthening as the disk fills up. But a 50% drop in overall performance at 75% capacity?

Now, your point is a reasonable one in defense of your technology, i.e., that this NetApp drop in performance can be compensated for if the storage system is properly configured; that is, if it is configured based on the spindles in the array being filled to a reasonable capacity. Then the customer will have no surprises and no room to complain. I happen to agree with that.

But here’s the problem in real life: All of us storage vendors are locked in competitive battles vs. each other. If a NetApp sales team knows they can double their IOPS by configuring their system at 10% capacity instead of 75%, what is likely to happen in a competitive situation where performance is the deciding factor? Answer: Human nature being what it is, the sales team will perform their demo, POC, or benchmark at the configuration level that will get them the best performance and win the deal. And then the customer is exposed to being surprised later on as he fills the disks with data and the performance drops. And then he has to buy more disks to increase the free-space, or buy a more powerful filer.

This is not an uncommon phenomenon with NetApp in block environments. If you go to a NetApp community blog where folks aren’t pressured to defend themselves in front of critics, and you pose the question, “My filer is dropping in performance, what shall I do?”, invariably the first diagnostic questions are around how much free space do you have.

For Mike: Your sarcasm was duly noted.

For Alex: I'm on holiday next week, so I'll be signing off now. No hard feelings.

Best regards,

Jim

[Calvin Zito]

I asked a storage architect from our HP LeftHand solutions team to jump into the discussion about capacity utilization and he's posted something on our blog: http://bit.ly/lvt5B. The moral of the story is when Alex says "it's always" (as he did in this post when referring to his "LeftHand capacity calculator"), file away the blog post as FUD and move on to a topic that's worth discussing.

[John Martin]

You state "If a NetApp sales team knows they can double their IOPS by configuring their system at 10% capacity instead of 75%, what is likely to happen in a competitive situation where performance is the deciding factor? Answer: Human nature being what it is, the sales team will perform their demo, POC, or benchmark at the configuration level that will get them the best performance and win the deal"

Speaking as someone who faces these pressures on a regular basis, I can say that your "Human Nature" argument is fallacious. Reps who try that kind of thing dont last long at NetApp, actually, they dont even get hired. We can pick and choose our candidates, and trust me, we are very very picky. We've got some very good (not to mention conservative) sizing tools which we use all the time when putting configurations together. Maybe things are different at HP, or maybe your confusing us with some other vendor.

As a case in point our recent public benchmarks like the SPC-1 are all done at higher capacity utlisation than other vendors publishing at the same level, and well over the theoretical 50% capacity utilisation of RAID-10, or nRAIDwhatever.

The other part that you miss is that we arent under the same pressure as our competitors because we have a great technical advantage when it comes to performance, to whit .. on a spindle for spindle basis at the same usable capacity, NetApp arrays easily outperform our competitors.

You'll note this on the graph above (i.e. even at our worst we were still better than EMC), and we proved this yet again during our SPC-1 submission with the 3040 vs CX40. HP havent put anything up recently (I wonder why ?), but if you take a look at all the old published Exchange 2003 ESRP figures.

NetApp FAS3040 achieved 140 IOPS per 15K RPM Drive

EMC CX3-40 achieved 105 IOPS per 15K RPM Drive

Hitachi AMS-1000 achieved 90 IOPS per 15K RPM Drive

HP EVA 8000 achieved 71 IOPS per 15K RPM Drive

EVA starts bad and stays bad, is that a great selling proposition :-( I’d love to see where Lefthand would have come in this list

One final thing, since that graph was published (almost three years ago), we've released a number of performance enhancing features that flatten out that top line significantly.

If you or anyone else is interested in our free space reccomendations for high performance databases (i.e. the thing everyone else uses RAID-10 for), check out the excellent paper by Steve Daniel on our public website tr-3647. To save you some time, we perform better than most equivalent setups up to about the 80% utilisation mark, which is 30% better than everyone else becuase they have to use RAID-10 (max 50% utilisation) for those workloads (80%-50%=30%)

[John]

@John

Taken from your own FAS3040 ESRP submission Page 17.

"The ESRP program is not designed to be a benchmarking program; tests are
not designed to measure the maximum throughput for a given solution.
Rather, it is focused on producing recommendations from vendors for
supported Exchange configurations. The data presented in this document
should not be used for direct comparisons among the solutions."

Given the rather widely differing requirement being addressed in each ESRP report and the subsequent differences in configurations your statement on IOPS is misleading at best.

[Geert]

John,

Care to put out a SPC-1 benchmark for EVA or LHN instead? We're very curious......

[Alex McDonald]

@John

Yes, it's misleading. But as John Martin points out, it's the only data available out there to defend against HP's continued and baseless assertions about NetApp performance and capacity.

Where are HP's benchmarks for any of its storage products?

[john]

I know I'm about to get tag teamed by the FanBoi's but holding up SPC as some kind of independent proof of performance is a bit rich from the company who took 5+ years to post their first SPC-1 benchmark, only EMC has managed to hold out longer. Intially I kind of welcomed SPC, but over time it's just been turned into a bit of a farce, and that impression wasn't really helped by your own submission and worst still SPC's acceptance of a competitors results. Yes it was quite funny to watch EMC squirm, but when you took a step back, it really was a very cheap shot by all involved including SPC.

[Alex McDonald]

@John

So it's OK that HP's marketing department does the numbers? You need to suggest an alternative.

[John]

@Alex,

Not sure what you're getting at. I called out the bogus use of the MS ESRP submissions as a benchmark, which you've already admitted was misleading. Given the title and content of this article I'm struggling to feel any sympathy. You called them out and they responded. On the point of marketting doing the numbers, well I'm not sure given the recent guarantee programs your in a position to complain.

[Alex McDonald]

@John

The ESRP program, as you pointed out and I agreed, is not a comparative benchmark. SPC is a comparative benchmark, but HP don't do them, so we're kind of stuck here. I was asking what alternative to a benchmark you might recommend.

Benchmarks are imperfect, but they're what we as an industry have. That HP refuse to undertake them, but keep harping on about imagined flaws in NetApp performance, is unacceptable to me (and others). The only HP numbers that are available are HP's own marketing figures; for an example, see here. I'll certainly keep calling them out on that.

I'm sorry, I don't understand your reference to the guarantee program.

[John]

@Alex,

I don't quite agree on the SPC being a comparitive benchmark, there are workload controls but very few controls on the submitted configuration. Witness some of the science projects put forward by some of the vendors. Many of these will perform like stink but will be unmanageable at best and downright unreliable at worst.

Here's a good publicly available whitepaper on EVA4400 performance, right off the product homepage, outlines the test setup, all the numbers are there including response times.

http://h20195.www2.hp.com/v2/GetDocument.aspx?docname=4aa1-8473enw

Not quite SPC but probably slightly more honest and comprehensive than results tweaked for SPC bragging rights.

There've been numerous discussion looking at replacing SPC for just these reasons, but everyone has their own agenda and I don't see this being fixed anytime soon. Until then I'll treat SPC results with the same level of healthy suspicion with which I treat Vendor marketing numbers & blogs.

I just can't think why you don't understand the marketing reference ;-)

[Alex McDonald]

Thanks for the link. On page 5 it says

Benchmarking procedures:
• RAID 1—16, 381818-MB Virtual Disk
• RAID 5 and 16, 610910-MB Virtual Disk

How many and what size were the LUNs? What was the effective VRAID group size on the RAID5?

I still don't get your reference to the guarantee program!

[John]

Well I'll put this up for you to tear down but going on the figures in the paper it's a single Disk Group consisting of 96x146GB-15K Drives, about 12.3TB usable after right sizing, sparing, etc.

16 x 381GB VDisks = 6.1TB + Vraid1 = 12.2TB
16 x 610GB Vdisks = 9.7TB + Vraid5 = 12.2TB

"Wrote to all of the capacity before making the measurements."

Like I said fairly comprehensive, with the aim of the paper being to "Build trust by using testing procedures that, within normal engineering constraints, customers can verify and repeat at their place of business."

i.e not a benchmark.

I know you get it, but we'll drop it.