When folk tell me that the chance to travel with my work must be wonderful, I just grit my teeth and smile politely. Let me tell you, it's far from wonderful.
I'm newly returned from a bout of traveling, nearly 3 weeks and 3 locations all told. Spending entire weeks away from home in business hotels, with daylight hours in bat caves, evenings eating overpriced and unhealthy food, capped by 12,000 miles of weekend air travel in conditions that animal welfare organisations would condemn as unsuitable -- that isn't my idea of a company paid-for holiday.
To cap it all, it was my birthday last weekend, on Sunday 31st May. I'd like to thank my colleagues that bought me a beer or two at the bar of the Santa Clara Hyatt. It helped me assuage the pain of being away from home. Much appreciated, even though my memories of the evening are a little sketchy.
I'm delighted to be back now, and was looking forward to my normal schedule of work. And would you believe it, I'm being asked (yet again) about HP's latest anti-NetApp nonsense doing the rounds.
Here's a screen shot of the latest dipstickery that passes for critical analysis at HP. (Yes, the slide says it's internal use, but it's publicly available on the web and it's being given to HP partners and prospective customers, probably with a "sshhh, don't tell anyone about this, but I've got this internal report, and it says...")
I'm not sure what NetApp report is being referred to here as there are several Cost of Ownership reports we've done over the years (search on Mercer or Wyman for example), and I can't find the report that matches with page numbers in the HP slide deck. The latest is here. Anyone at HP care to tell me if that's the right one?
Whichever HP used, the slide contains more than a few errors.
- We never recommend RAID-4, because RAID-DP at 14 data plus 2 parity is far more reliable. RAID-DP is more reliable than RAID-1, the most reliable and performant of the non-dual parity schemes. With HP you get to choose; reliability (RAID-6 for truly performance sapping protection) or RAID-1 for performance and protection at serious capacity cost; RAID-5 if you like your risks and performance about equally divided. With NetApp, you get superior performance, capacity and protection from RAID-DP.
- NetApp space reservation has been consistently and deliberately misunderstood by HP. And others.
- You choose snapshots or full clones on HP, and the choice is a tough one. HP EVA snapshots perform badly and consume little space; clones perform well, but consume lots of space. NetApp snapshots and clones don't have that problem.
- There is no cost for an NDMP licence on a NetApp system. And there never has been. Ever.
Conclusion; If HP only understood NetApp's technology...
When your HP sales rep slides this across your desk and suggests that HP's products are wonderful, think business travel. Then just grit your teeth and smile politely.
.
Alex,
Do you have any accurate benchmarks reflecting the the truly performance sapping Raid 6 protection ? or is this just an assumption ?
My observation on the misunderstandings around the need for space reservation seems to be largely down to Netapp first denying the problem existed. Then stating the documentation referencing this as a best practice was out of date. You should look closer to home on this one.
Again benchmark on the snapshot usage ? You also forget to mention that Netapp clones are read writable snapshots which will protect you from a logical failure but not a physical failure of the source volume. So although real clones have a capacity penalty they also have availability and recoverability advantages.
You conveniently skirted the one about comparing a single filer to a dual controller array.
Regardless of the other claims there is a cost for an NDMP license. Which you will need to purchase from your chosen backup vendor. In some cases you'll pay per TB and in others you'll maybe need two if your clustering files. Something not needed on a non prorietary system.
Posted by: john | June 09, 2009 at 02:35 AM
John
No, there’s no benchmark information for Vraid-6 on the EVA, because there are no HP or independent benchmarks for the EVA at all. The EVA Best Practices Whitepaper at http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-0914ENW.pdf page 21 says;
No mention of Vraid6, although it’s supported, and given that it operates with extra parity, it must be down in performance on Vraid5. And as Vraid5 isn’t exactly going to set the heather alight (except fo those unusual sequential write workloads) in comparison with Vraid1, then performance sapping seems a reasonable description.
On space reservation, this “competitive analysis” from HP was dated June 2009. The NetApp document was updated in January 2009 (http://media.netapp.com/documents/tr-3483.pdf). That’s enough time for HP to update this piece of fluff, surely?
On clones, you say;
Yes, HP’s Vraid0 or Vraid5 might need this, I can see that. HP's clones also have performance problems. Your scenario of “physical failure of the source volume” is what NetApp's RAID-DP is protecting against.
If someone will point me at the report, I’ll address the single filer/dual controller issue. Although, as it refers to archiving, the HP solution of dual controllers (the only type available) seems overkill and adds to the HP cost without really adding much. I suspect that might be the reasoning.
Lastly, the testing was comparing an HP SAN vs a NetApp SAN. Why NDMP would be relevant is beyond me, as NDMP is for NAS backup.
"Something not required on a non-proprietary system?" Really? Since when was an HP EVA non-proprietary?
Posted by: Alex McDonald | June 09, 2009 at 01:06 PM
One comment to add to this comment from John:
"You also forget to mention that Netapp clones are read writable snapshots which will protect you from a logical failure but not a physical failure of the source volume. So although real clones have a capacity penalty they also have availability and recoverability advantages."
This is incorrect. There is a little thing called "clone split" which you can run at any time you wish.
This means that your clone is a writeable snapshot only for as long as you need it to be (like in a dev/test environment for example). At the moment of your choosing you can decide to turn it into a full copy clone, just like an EVA. Or (unlike an EVA) you could even just delete it and instantly create a whole new one - or 15 - instantly and without any full copying at all. That's efficiency.
Posted by: Lee Razo | June 11, 2009 at 02:45 PM
Hey Alex,
I guess NetApp doesn't have standards of business conduct or maybe you don't? HP Restricted means its an HP internal slide (and can be showed to others only under non-disclosure). This along with your PR team stretching the truth in your press releases (http://bit.ly/meyi4) should have customers questioning NetApp.
Posted by: Calvin Zito | June 14, 2009 at 06:27 PM
@Calvin
A little reading of my piece (it clearly says "it's publicly available on the web") and some research would have led you to this copy on the internet. If you want to maintain the illusion that this is not in the public domain and that you don't point your potential customers at it, then get it off the internet.
As to our PR, where you say (my bold and link)
Gary Zasman, NetApp's Practice Director for Apps Integration, is the founder of the SNIA DIM. He has documentation from SNIA that shows that he was the original founder of this group; he then recruited other companies to participate. Go ask SNIA.Yes, we have standards, both business and technical. NetApp are very active in the standards area, a task that, while generally unrewarded, benefits all. We've a blog devoted to it that you may find useful.
Posted by: Alex McDonald | June 15, 2009 at 02:05 AM
Bad excuse for spreading a private slide publicly on your blog Alex - kind of like telling the judge when you pass counterfeit money that somebody gave it to you when you know it's counterfeit. It's still your bad.
The SNIA press release says IBM and your company founded the SIG. I guess you should talk to SNIA if you thought you should be given exclusive foundership of the SIG.
Posted by: Calvin ZIto | June 15, 2009 at 05:31 AM
@Calvin
Some of the slides have "HP Internal Use Only" on them, in which case you've a bigger breach of security than you'd like, or, as I believe, a pretty cavalier, sloppy and inconsistent attitude to the legal aspects.
Your loose definition of private -- that is, when it suits you -- is what's causing you indigestion. Declaring something as only available under NDA (non disclosure agreement) simply because you mark it that way doesn't make it subject to an NDA.
And in case you don't get this point, to have a valid NDA for something, it needs to be a secret. You can’t guard and keep secret something people already know and that you leaked; what secret are you protecting?
The only "secret" I can see in this presentation are the bits that are wildly wrong, because the rest is information that's publically available. And that's the bits I'm refuting, the rubbish that you tell your partners and customers. In "secret".
If you feel your NDAs are being abused, and that you actually have something in that document that justifies an NDA, wag your finger at your NDA reseller that leaked this to a customer and put it up on his website, or the NDA customer that leaked this, or the author of the slides that sloppily mixed internal and partner only slides.
The SNIA press release is correct, and it doesn't say how the SIG was formed. So is NetApp's press release correct, and it points out that we initiated it. What exactly is your point?
And I'm still waiting for someone from HP to point me at the analyst report in question.
Posted by: Alex McDonald | June 15, 2009 at 06:25 AM
Hi Alex,
Remember me? I’m the guy you had the debate with over the Mercer cost of ownership white paper back in November on the HP blogsite - the same white paper referenced in the slide you posted above. (For those interested in reading the original discussion, here it is: http://bit.ly/12Svs8 )
Funny – you had no problem figuring out which white paper I was talking about back then. To jog your memory, here is one of my comments about the ‘fairness rules’ that the NetApp/Mercer white paper violated…and your response on my blog.
============================
Jim: "Fairness rule #3 – no fair comparing apples to elephants - you can't, I mean in a COO study you JUST CAN'T compare solutions where one uses snapshots and the other uses full-copy clones."
Alex: Another fair point. Do you advise the use of snapshots on EVA instead?
===========================
Is this starting to look familiar? And, yes, the slide is mine.
Now, as to the ‘errors’ you found in the slide above:
1.We never recommend RAID-4. RESPONSE: Back in 2006 when the white paper was first published you frequently did. (Just do a Google search on: NetApp, RAID 4, 2006). But it’s a moot point. In a cost-of-ownership white paper, it makes no difference whether you use RAID-4 with a 7+1 stripe or RAID-6 with a 14+2 stripe – the usable capacity works out the same. But here’s the point: How can you publish a 22 page storage array cost-of-ownership white paper and not mention what RAID levels are being compared?
2.NetApp space reservation has been consistently and deliberately misunderstood by HP and others. RESPONSE: I assume the others you are referring to are NetApp customers. In any case, you and I already had a long debate on this subject a few months ago on the HP blog. I made my points. Your points were…well, would you care to have that debate again, this time on your blog?
3.You choose snapshots or full-clones on HP, and the choice is a tough one. RESPONSE: You’re trying to change the subject. The issue on the slide is not whether you should use snaps or clones – the issue is why would a storage-savvy marketing organization compare the capacity used by their snaps against the capacity used by their competitors’ full-copy clones and then not make that clear in their COO white paper.
4.There is no cost for an NDMP license on a NetApp system and there never has been. Ever. RESPONSE: Yes, NetApp doesn’t charge for NDMP licenses – I thought everyone knew that – but the backup vendors who sell the software sure do. The last time I looked, the backup vendors were not charities. Bottom line: The back-up software vendors do charge extra for NDMP and it is an extra over-and-above cost for customers in NetApp environments and should be considered in any reasonable cost-of-ownership white paper.
I did notice your response to John (above) where you said that NDMP was not relevant because the white paper was talking about SANs. On the contrary: The paper only used the competitor SANs but for NetApp it compared SAN and NFS (page 9, figure 5). Also on page 7, diagram 3 of the white paper you’ll notice it shows either a SAN or a LAN switch between the NetApp filers and the database servers. And in case there is any doubt whether NDMP is used to back up databases: http://eval.symantec.com/mktginfo/products/White_Papers/Data_Protection/nbu6_NetApp_Oracle_wp_3394.pdf
Finally, when I show the slide to audiences I never say, ‘Sshh, don’t tell anyone about this’. I say, “You signed a non-disclosure agreement. Honor it.”
Regards, Jim Haberkorn
Posted by: Jim Haberkorn | June 15, 2009 at 07:02 AM
This feels like team tag blogging :-)
Thanks for responding, and thanks for the memory jog on your blog. (Btw, did anyone get dinner in Zurich?)
I'm sorry someone saw fit to breach your NDA. But you really can't point the finger at me as the guilty party here. You need to look at what you're NDAing and who you trust with this, and whether it really needs to be. That I can't help you with.
Posted by: Alex McDonald | June 15, 2009 at 07:59 AM
Hi Alex,
No Alex, no one has claimed your dinner yet. I’m still waiting for you in Zurich.
And no hard feelings from me personally about posting my slide. We’re all big boys here.
But since you were kind enough to post my slide on your blog, it’s only fair that I explain how I use it. And I’m going to try and say this as tactfully as I can – because I may be having dinner with you someday. Here it is: It’s too bad that whoever sent you the slide wasn’t able to send you the laugh track that goes along with it, because I assure you, when I go through that slide the only jaws not laughing are the ones hitting the ground. Here is what the NetApp white paper did:
1.Compared the number of disks required for its snapshots against the disks required for its competitors full copy clones.
2.Tried to hide that fact by referring to the clones as ‘snapshot equivalent technology’ – a name it made up just for the occasion.
3.Further, tried to hide it by posting a diagram where all the competitor solutions were lumped together in a box with the word ‘snapshot’ prominently displayed.
4.Then forgot to mention that it was comparing its parity RAID with an 87.5% usable capacity against its competitor arrays with RAID-1 and 50% usable capacity. Now, why would a company forget to mention that?
5.Tried to stack the deck even further, by throwing in a cheap, single-controller archive solution with multiple single-points-of-failure against its competitors fully-redundant arrays.
6.Also, in this objective, carefully researched study based on ‘actual customers’, this 22-page paper worked out that the EVA, CLARiiON, and Symmetrix used the exact same capacity down to the hundredth of a TB even though they all have different RAID stripe sizes and the CLARiiON uses vault drives, and the Symmetrix uses far more cache. Now, how’s that for a coincidence!!!
7.But here’s the one that really cracks me up: Even though NetApp admits it has aggregate, volume, and LUN space reservations requirements that other vendors don’t have, it still insists that it has better usable capacity. In fact, the paper claims that it is not 10% or even 20% better than its competitors, but that it is in fact 100% more space efficient than EMC and HP. Where NetApp customers would only need to purchase 5.6TB, the paper claimed that the EVA, CLARiiON, and Symmetrix customers all had to buy the exact same 11.65TB. And to get the same performance as NetApp they would have had to purchase 7x more storage – 2.2TB vs. the exact same 15.4TB. One reason I personally find this performance claim so funny is that here are HP and EMC, the two biggest SAN vendors in the world, with arrays designed for blocks from the ground up, and then here is NetApp with a solution that builds LUNs on top of a filesystem, and NetApp will still claim it’s faster. And not just a little faster, but practically an order of magnitude faster. You gotta love the chutzpah.
So, the way I use the slide is to first go through it in detail and review what NetApp is claiming about itself in the area of cost-of-ownership. And then I show how NetApp uses basically the same tactics in defending its space reservation, performance, capacity guarantee, and usable capacity claims. It’s a pretty simple argument.
Hope that helps,
Jim Haberkorn
Posted by: Jim Haberkorn | June 16, 2009 at 08:12 AM
That's a seriously long reply, Jim. Here's a short and pretty simple rebuttal; NetApp's Storage Efficiency blog.
On the "NetApp will still claim it’s faster" part, why don't you prove that the HP EVA that's "designed for blocks from the ground up" is faster? We're faster than the equivalent EMC box that was designed for blocks from the ground up, and that was RAID-DP versus RAID-10 too.
Posted by: Alex McDonald | June 17, 2009 at 05:05 AM
I thought they had.
http://www.communities.hp.com/online/blogs/datastorage/archive/2008/12/04/making-sense-of-wafl-part-4.aspx
Posted by: john | June 17, 2009 at 09:33 AM
Are you serious? That's not a benchmark, it's a set of homebrew tests that Karl Dohm of HP ran to embaress Pat Cimprich of Avanade (which he he rebutted here).
Posted by: Alex McDonald | June 17, 2009 at 09:45 AM
Hi Alex,
Okay, let’s review the situation:
1.Alex posts competitor’s confidential slide on his blog and boldly describes it as nonsensical and inaccurate.
2.Jim, the slide creator, calls him on it.
3.Alex hems and haws, changes the subject, points people to non-relevant information, stumbles around and can’t seem to come up with anything factual in regards his earlier bold comments.
But, I’m still puzzled as to what possessed you to post my slide on your blog when eight months ago, on my blog ( http://bit.ly/12Svs8 ), you already conceded the key points it made. To refresh your memory, here is a cut and paste from my blog showing the ‘fairness points’ I said the NetApp cost-of-ownership white paper violated along with your responses.
Jim: "Fairness rule #1 - you have to make it clear what is being compared. No leaving out RAID levels."
Alex: OK, what RAID levels would you like to compare RAID-DP with? RAID-10? Why is it important? We've only got the one RAID level.
Jim: "Fairness rule #2 - no making up your own confusing names as a substitute for common industry terms - you can't refer to your competitors' full copy clones in the configuration table as ‘snapshot equivalent functionality'. "
Alex: Fair point. See below.
Jim: "Fairness rule #3 - no fair comparing apples to elephants - you can't, I mean in a COO study you JUST CAN'T compare solutions where one uses snapshots and the other uses full-copy clones."
Alex: Another fair point. Do you advise the use of snapshots on EVA instead?
Whether you intended it or not, you have now publicly conceded my points a second time - this time on your own blog.
But I'm still scratching my head as to why you brought it up again.
I’m out of here.
Regards,
Jim
Posted by: Jim Haberkorn | June 18, 2009 at 10:47 PM
@Jim
Conceded what? I suggested you had a fair point, and then you hemmed and hawed an answer to the question I posed. The bit you missed in bold.
It depends? Depends on what? Care to answer properly or are you really leaving and taking your toys with you?
I keep bringing this up because I keep seeing your stuff; customers bring it to me, asking for point by point rebuttals of it. The good stuff (and there is some) is overwhelmed by the volume of unsubstantiated assertions, conspiracy theories and kack handed attempts at (mis)understanding NetApp technology.
The problem is, it's printed on HP paper, variously marked as confidential, NDA, internal (hah!), so customers give it some credence. That's why I keep bringing it up.
Posted by: Alex McDonald | June 19, 2009 at 02:08 AM
Alex,
"Alex: Another fair point. Do you advise the use of snapshots on EVA instead?"
"Jim: it depends on what the customer is trying to achieve."
Jims response seems fair, the reality is that Netapp and any other vendor should be asking this question of the customer. Snapshots may not always be the most appropriate technology dependent on the usage scenario. That's why you have the "Clone Split" command, yes ?
What if the data you're snapping needs to have high bandwidth backup, an integrity check or reporting run against the system, what if you just need to move the LUN to another tier ? any of these use cases has the potential to impact the source Lun's performance due to the sharing of blocks. As such snapshot wouldn't be the most appropriate technology regardless of vendor.
Taking the moral high ground around the competitive slides is a bit rich, I get to see Netapp, EMC, Equalogic and HP competitive papers handed to customers on a regular basis. All the vendors do it and Netapp seem to have as active a competitive attack team as any of the worst offendors out there.
Posted by: John | June 19, 2009 at 04:17 AM
John
Jim's response is only needed because taking a NetApp snapshot is pretty much performance & capacity overhead free. So is cloning from a snapshot (thin cloning). I want to know from Jim -- what's wrong with an EVA snapshot; why does it depend? NetApp users just take them without the "depends" clause. I'll answer it, since you and Jim seem unable to do so;
Taking an EVA snapshot has serious performance implications.
Why on earth would a full clone be better for high speed backup? I want to get at my data as fast as possible; if the blocks are shared (or duplicates), I'm reading fewer blocks, and not moving the heads between a full copy and the original data either. Or wasting time, IOs and space copying it out.
As for moving data between tiers, you're confusing a screwdriver with a hammer. Take a copy, cloned or otherwise.
Lastly, this is defence, not attack. And if you find anything -- and I mean anything -- that's unclear, indefensible or simply wrong in NetApp's material, please let me know. And if you see customers getting NDA materials (I think you're a reseller, yes?) it must be as part of your business. You're breaching NetApp, EMC, EqualLogic and HP restrictions on their materials if you or your colleagues do so.
Posted by: Alex McDonald | June 19, 2009 at 05:05 AM
Alex,
In a backup environment I may not want a the backup stream load impacting my application I/O. If I'm sharing blocks with the backup stream then I'll incur head contention and latency on my application over potentially a long period of time. I do have customers with these kinds of concerns which the various methods of cloning and mirroring can and do solve.
I'm not sure what you're implying reagrding breaching NDA's, but I didn't say I was providing the information to customers, so how am I or my company breaking NDA's ? If a customer chooses to show me something a vendor or any other reseller has left them with, then I'm merely a witness to someone else breaching NDA. I'm not the one posting or bringing to the fore another companies internal documentation and then crying foul.
Personally I try to avoid getting into FUD as it only leads to confusion for the customer and leaves a bad taste for all involved. Whenever possible I'll ask the customer to set such documentation aside and then attempt to get to the heart of the requirement.
Since no single solution or vendor fits all requirements, regardless of the hype. If I don't have the expertise on the particular product then I'll call in a colleague who does.
Posted by: John | June 19, 2009 at 06:41 AM
I'm coming across as irritable, for which my apologies, as I let Jim H get under my skin. I should know better!
Your points are well made. Yes, there's a diference between a snapshot, thin clone and full clone, and each has its use case. It would have been useful for Jim to explain why our use of a full clone on the EVA was mistaken, and how an EVA snapshot would have been just as good, but I suppose that was too much to expect.
The request for correction when we're wrong still stands; we pride ourselves on having accurate and verifiable information.
Thanks, and again, my apologies for sounding off like Groundskeeper Willy.
Posted by: Alex McDonald | June 19, 2009 at 07:39 AM
@John - you say;
"In a backup environment I may not want a the backup stream load impacting my application I/O. If I'm sharing blocks with the backup stream then I'll incur head contention and latency on my application over potentially a long period of time. I do have customers with these kinds of concerns which the various methods of cloning and mirroring can and do solve."
This is exactly where thin replication D2D backup (aka SnapVault) comes in play. This totally eliminates any backup load from the production environment by simply moving and storing changed blocks only. Every update REPRESENTS a FULL copy of the data, while only storing a fraction!
Now add virtual, or thin, cloning (aka FlexClone) to the mix and re-purpose that data making it useful instead of just having it sit on (virtual) tapes.
Now THAT is Storage Efficiency, resulting in lower costs and improved utilization - while enabling business breakthroughs.
But don't take it from us, take it from a real world customer - http://www.netapp.com/us/company/news/news_rel_20061024.html:
"Oracle's utilization rate is very high," said Bert Dollahite, IT Tools and Automation Services at Oracle. "NetApp storage and data management solutions have been a key part of the equation that has helped us lower our total IT and management costs, and enhance productivity."
So with all the hand waving and bending of the truth at a micro-level (=storage box) you guys are clearly showing you don't understand what it is all about.
But by all means, please stick to your traditional methods, we well educate your customers on how business issues (like archaic tape backups) can be solved....
Posted by: Geert | June 19, 2009 at 10:39 AM
@ Geert,
I'm sure Netapp will be grateful for the advertising, but snapvault's functionality aint exactly unique. There's a raft of competing products I could name that can do the same job, even MS has one now.
BTW neither did I claim a clone was the only way to achieve this goal, I used the clone as an example to make the point that despite the cool aid, a snap is not functionally equivalent to a clone regardless of vendor implementation.
Despite all your handwaving, the marketing release you pointed me to is nothing more than well erm a marketing release! Search long enough on any of the tier 1 vendors sites and you'll find virtually the same story in one guise or another. Back scratching between Oracle and storage vendors has been going on as long as I've been working in storage.
I've lost interest at this point, but please be my guest and feel free to intiate, sorry meant educate my customers.
Posted by: John | June 20, 2009 at 02:04 AM
C'mon John, you can't simply pick up your ball and leave the playground because you don't like the direction in which this discussion is going - only Chuck is entitled to do so, or at least that's what he thinks... ;-)
If you can point me to a single website that has a SnapVault equivalent - that works exactly the same across ALL PROTOCOLS, like CIFS, NFS, iSCSI and FCP (or FCoE for that matter) and across ALL SYSTEMS, low-end to high-end and disk tiers and on which you can SPACE EFFICIENTLY clone that backup data (so without copying a single bit) and has a significant number of real world implementations - then I'll surrender. Until then, you're not dismissed... ;-)
And what do you mean "marketing release? This is a voluntary testimonial by a real world customer, with a real name, that simply testified they achieve very high utilization, which is exactly the opposite of what you and Jim and all those FUDders are constantly blaming us of...
Ever heard of a system running at over 400% utilization, John. Ask the guy. Or are do you want to call him a liar?
And btw - this press release and thus customer testimonial is already almost 3 years old so our Storage Efficiency is going on already for quite a while. It's in our DNA. Accept it...
Posted by: Geert | June 20, 2009 at 10:02 AM