NetApp - The Missing Shade of Blue

Random Rocks And Benchmarks

“People understand contests. You take a bunch of kids throwing rocks at random and people look askance, but if you go and hold a rock-throwing contest -- people understand that.” (Don Murray)

And that, in a nutshell, is the origin of most competitive sports. What starts as "Hey, I can chuck this boulder through that window over there -- betcha you can't!" develops through time a set of rules, referees, measuring devices, a governing body (sometimes two or three) and the sport of rock-chucking may even get recognised by many, played by many and make it into the Olympics. Like spear throwing (the javelin) or blasting birds out of the sky with a shotgun (clay pigeon shooting), rock-chucking has matured into shot-putting.

Just like benchmarks. Originally used to demonstrate some feature of your system in all its one-trick glory (HP IOPS from cache is a classic example of how to do this), modern industry benchmarks make the attempt to allow some real-world comparison between competitors. The relationship to real life is debatable, but the rules by which they operate are carefully designed to inject some aspects of realism to comparative claims.

Benchmarks are intended to be a repeatable test of a set of skills. But, as with all competitive sports, sometimes there's the kid who starts his benchmarking career hanging round on street corners stoning passers-by for entertainment, or hoisting up the nearest large rock at hand and heaving it through the closest plate-glass window.

And the new kid is over at HP, using a non-benchmark as a benchmark, and generally lobbing rocks around in all directions. I'm not going to dissect the post in detail, because others have and will continue to do that. There's one little paragraph I want to focus on, because it demonstrates one of my pet peeves; benchmark intuition.

Now things were starting to make sense.   We were seeing the same sort of decay curve as shown in the IOMeter results posted in Making Sense of WAFL - Part 4.    Every time the test is run, the random component of the Jetstress database accesses fragment the LUN further and the throughput numbers get worse.  An array like EMC CX or HP EVA wont undergo this sort of decay curve since these arrays do not have internal WAFL-fragmentation problems like the FAS does.

The non-benchmark is Microsoft's ESRP, and the tester's intuitive assumption is that WAFL fragments; hence the tester's intuitive assumption that this is the source of the diminishing throughput numbers.

Bzzzt. Big fail.

Let's allow the first intuition; let's allow, for the sake of this demonstration, that "WAFL fragments your data". Here's a simple example to demonstrate why his intuition is wrong on fragmentation being the source of the problem. Exchange 2007 generates small random IOs (and that's the JetStress that HP are using in their test). The table below has 5 columns to demonstrate why small random IO works just as well (or badly, depending on your take) on randomly laid out data as sequentially laid out data.

• Random Placement: I've place 100 blocks randomly. The numbers have been generated from www.random.org. Slot 1 is block 67, slot 2 is block 19 and so on.
• Random Requested Block: this is meant to simulate IO requests from Exchange; again, drawn from a different run from www.random.org.
• Matching Block (Random Placement): this is where the requested block actually lives, So asking for block 80 requires a visit to slot 22, and so on.
• Seek Distance (Random Placement): this is the effective seek distance between requested random blocks. After we visit slot 22 (for block 80), we need to visit slot 58 (for block 37), requiring a seek of 36 slots.
• Seek Distance (Sequential Placement): this is the effective seek distance between requested sequential blocks. After we visit slot 80 (for block 80), we need to visit slot 37 (for block 37), requiring a seek of 43 slots.

(I'm having difficulty uploading the spreadsheet to TypePad, so when I get it fixed you'll be able to "Click to download the whole spreadsheet". Not yet though.)

Random Placement	Random Requested Block	Matching Block (Random Placement)	Seek Distance (Random Placement)	Seek Distance (Sequential Placement)
67	80	22	0	0
19	37	58	36	43
75	18	61	3	19
23	26	53	8	8
85	57	63	10	31
59	100	14	49	43
14	59	6	8	41
...	...	...	...	...
SUM			3269	3322

Hey, look at that! The sequentially laid out data takes more slot seeks than the randomly laid out data! Try it yourself, replace the 100 numbers in the first two columns with random numbers, and check the seek distance sum.

On average, they will be equal. In fact, if the requested blocks are random, it doesn't matter how the data is laid out. Intuition fail.

Here's the professional sport of benchmarking, which HP don't take part in (still being, as it were, at the rock-throwing stage);

• NetApp SPC Benchmarks (no HP here)
• SFS2008 NFS benchmarks (HP missing again)

And, of course, the official ESRP results (and, just as a reminder, these aren't benchmarks)

• Microsoft ESRP results

Having failed at shot-putting, perhaps HP might want to pick another sport for their talented testers. Like nude football.

[updated to correct some borked links and a typo].

IBRIX Falls to HP

Today, HP announced that it was going to buy IBRIX, a small NAS scale-out software technology. HP have partnered with them for some time, so the move was probably a natural one.

A quick analysis; here are my initial thoughts.

• PolyServe (aka Enterprise File Services) is dead. Dead as the Norwegian Blue. It always looked a bit like it was nailed to its perch, but this, if any was needed, is confirmation that doing scale-out well is hard, and that HP picked a loser the first time round.
  
• This lets HP sell more servers; IBRIX is software only, and what better than another acquisition that lets them close out IBM and Dell in favour of HP tin.
  
• Cloud thinking must be in there somewhere. A scale-out NAS is always going to be preferable over a bunch of strung together aging EVAs.
  
• What is confusing is the HP StorageWorks product range. It encompasses everything from DAS to NAS to SAN in multiple flavours; a veritable Tutti Frutti of storage. But then, it's all "designed" to sell servers from what I can see. Either it runs on an HP server, or it needs an HP server to front it off.
  
• Lastly, and I think you'd have to agree on this one; it's an interesting time to be in storage.

Where and what (and who) next? Your guess on this one is as good as mine.

.

LeftHand Quorums and Split Brains

There's been a bit of a discussion going on between John Spiers (HP LeftHand) and myself over a number of issues I raised about a LeftHand SAN's storage efficiency. The comments moved on to talk about HA (high availability) and data reliability, and John raised a number of questions that I thought deserved a longer answer.

John Martin of NetApp (who also blogs on NetApp's Storage Efficiency blog) has kindly provided me with more detail, and rather than post this as a comment, I thought it worthy (again) of a blog in its own right. Thanks to John for the responses.

John Spiers' points are in blue (and I think I've accurately captured them, but they've been lightly [edited] for context). They're also slightly out of order.

In summary, I think there's a need for greater clarity on LeftHand best practices. Everyone, including the user quoted below, appears to be operating in the dark. As I said in a previous post, we've had to do a little"reading between the lines" and work from first principles. If any of this is wrong, please let me know, and I'll correct it.

[Update 09July: Many LeftHand manuals, including best practices, appeared in late June/July, but weren't there when I first looked in late May/early June. Much reading to do!]

---

NetApp can't deliver this level of HA with auto failover and failback [compared to a LeftHand SAN]

It depends on what you mean by "auto failback". If you mean failback initiated without permission or authorization from a responsible human being, then you'd be correct. If you mean an automated process with minimal user intervention, then you're wrong. From a NetApp perspective, and that of most storage and systems professionals I've talked to when discussing failback requirements, automated and uncontrolled failback is usually judged to be a bad idea.

NetApp also have an excellent product in addition to MetroCluster, called MultiStore. This provides similar kinds of functionality (automated failover and failback) over standard IP connections.

Can SnapMirror or MetroCluster [...] incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Yes it can. As soon as the administrator believes it to be safe, the "failback" process is initiated, and an automatic incremental rebuild and resync provides seamless failback.

[quoting from a NetApp document] "Mirrored active/active configurations do not provide the capability to fail over to the partner node if one node is completely lost. For example, if power is lost to one entire node, including its storage, you cannot fail over to the other node. For this capability, use a MetroCluster"

[quotes sections about manual cluster failover and prevention of “split brain” and JS then says] LeftHand has distributed quorum management that eliminates all possibilities of a “split brain”. This allows at least one site to operate and then automatically resync the other sites when they come back online.

This is an interesting quote, which when taken out of its correct context (a practice commonly called "quote mining") makes it sound much worse than when it's put in context.

1. It should be noted that this is for a VERY unusual configuration, one that I’ve never seen go into production at any site. The comment is under the heading “Mirrored Active/Active Configurations”. With this, NetApp's Syncmirror is added to a standard active/active configuration to provide a second level of mirroring across the disks, but without a MetroCluster license for local (or stretch) functionality (suitable for distances of around 500m).
2. Should a customer decides that his data required the extra protection of Syncmirror (i.e. RAID 6+1), then we would recommend the extra resilience provided for in this configuration by using MetroCluster.

The reason for this behavior is to avoid a “split brain” scenario. There is no way of automatically detecting the difference between a total failure of a system, or the the failure of all forms of communication between them. It's this that causes "split brain"; two or more running systems thinking the other has failed, when only the communication between them has failed.

This applies to every conceivable cluster configuration. That includes quorum disks, proxy nodes and other cluster node failure detection techniques; they are all effectively forms of communication between nodes.

Distributed quorum management requires at least three nodes. It is not possible to have a quorum based on two nodes.

I’m not letting you brush this one under the rug. MetroCluster and SyncMirror don’t provide the same level of availability that is inherent in LeftHand’s base SAN/iQ software offering, and LeftHand requires no additional equipment.

That's interesting, because here's a LeftHand user experience;

"So at work we decided to go with a Lefthand Implementation of iSCSI, I am rather unhappy to find out that with only 2 units you have to run a virtual machine to complete the Quorum for redundancy. I am not happy to find out that in reality you need 3 appliances to complete a Quorum for management and to ensure that you have redundancy and that everything is available."

You should have been clearer about comparing a three or more node site vs. a two node site. This HA "no additional equipment solution" is growing legs.

And while I’ll give kudos for an n-way implementation, unlike the NetApp documentation you quote, you don’t address how you handle what happens when there is a complete loss of communication only from the primary site to all other sites. None of the other nodes are able to detect whether the site is a smoldering pile of rubble, or if it’s just incommunicado, and if you make the assumption that lack of communication is a trigger for failover, then you have a recipe for split brain.

The manual “declared disaster” approach is the safest way of dealing with this. If a customer believes they have a foolproof way of detecting a true site failure, then its trivially easy to integrate this by automating MetroCluster failover (or failback for that matter)

Yes, [HP's LeftHand] Network RAID 2 can sustain any random 5 disk failure. In fact, a 4 array system configured in Network RAID level 2 can sustain up to 2 complete array failures, and up to 6 disk failures in each of the remaining 2 arrays, and all at the same time.

Hmmm... Picture courtesy of HP's LeftHand P4000 brochure 4AA2-5247ENW, April 2009.

Let's start with nodes. I have two copies of my data for nRAID2, and four arrays. If any two nodes fail, then I lose data. Which two arrays can I blow away in this four node system, and not lose data in my logical volume? With this simplistic 4 block LUN, 1 and 3 or 2 and 4. Unless you can arrange your node failures in advance, I'd suggest that is one node protection.

And the statement that I can survive any random 5 disk failure is equally implausible given the LeftHand diagram above. Two disk failures in node 1 in the same RAID5 group would kill node 1. A further two disk failures in a RAID5 group kill node 2, and we now have two node failures which can't be survived. The answer for nRAID2 is 3 random disk failures, not 5.

I'm not even going to try and work out what "up to 6 disk failures in the remaining 2 arrays" really means, although I'll bet the words "random" and "quorum" don't feature heavily. 

Of course, this all assumes the HP LeftHand diagram above is correct, and not just a graphic designer's interpretation of the facts. The diagram below for the terrible usable capacity of a LeftHand system for 1 node failure protection (split brained or otherwise) is correct. I've had it double checked by HP.

LeftHand nRAID2 means 35% usable capacity

.

An HP LeftHand Triplication Calculator

Part 1 of my investigation of LeftHand's claim to save money, given its less than stellar 35% usable from raw disk space, generated a number of interesting replies from HP. One is worthy of more analysis, but as it's a bit big for a comment, I've taken the liberty of extracting John Spier's reply to me. John was the former CTO of LeftHand Networks prior to its acquisition by HP.

There's a lack of information on how LeftHand does its stuff publicly available, so I've had to do a little"reading between the lines" and work from first principles. If I've got any of this wrong, please let me know, and I'll correct it. I've added a running commentary to John's comment; my apologies for breaking it up, but it's all here (in blue for clarity).

When using Network RAID 2 it protects you from multiple disk faults, complete array faults and site faults with auto failover and failback. NetApp can’t deliver this level of HA with auto failover and failback. Features like MetroCLuster give you data protection, but not HA, and at a lower capacity utilization than LeftHand. Can SnapMirror or MetroCluster automatically fail back, incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Then you'd be wrong; that's exactly what MetroCluster is about. Except the auto failback; that's just adding a disaster on top of a disaster. But I digress, and that's the subject for another post. 

It's worth pointing out before I analyse this claim that I originally thought that LeftHand had come up with a new paradigm with its network RAID; that it provided both data protection and high availability built on commodity tin. I was wrong; if it was that easy, we'd have done it. But NetApp's 15 years experience in doing this stuff has taught us otherwise.

First, I need to explain what a NetApp cluster is all about; then we can compare and contrast, and ask some questions.

NetApp Data Protection and HA

NetApp uses NVRAM (non volatile RAM) and transaction logging to capture writes to disk. All writes are acknowledged before they get to disk, but only after they've been logged. This means that when a single controller (non-HA) fails, the data we said to the server or client that we wrote, but stored in NVRAM, is replayed to the disks when the system comes back up. That way, we guarantee what we promised when we said we'd written the data. The data is both consistent and durable.

In a cluster or HA solution, we ensure cache coherency; the contents of the NVRAM on controller 1 are mirrored to controller 2. That way, when controller 1 goes down, controller 2 can replay controller 1's writes, and take over its workload, as the disks are addressable from both controllers. Again, the data is both consistent and durable; and we've made it HA, without downtime to the application. It carries on running.

If controller 2 now fails (or both fail together), we're still consistent and durable; see above for the single controller case.

Lastly, SATA drives in particular can suffer from "lost writes". Every drive has a cache where it stores data to be written. This is separate from any other protected cache, for instance NetApp's NVRAM.

As soon as an IO hits this buffer, the drive acknowledges the write. But blocks can subsequently be written in the wrong place, or not written at all, especially if there's a disk failure between acknowledgement and the physical write. 

Because NetApp has the ability to control both the RAID and the file system, Data ONTAP 7G provides the unique ability to catch errors such as this and recover. Along with a block checksum, ONTAP also stores WAFL metadata (the inode # of a file containing the block) that provide the ability to verify the validity of a block being read. If the block being read does not match what WAFL expects, the data gets reconstructed ensuring that your data is both consistent and durable.

NetApp goes to extraordinary lengths to protect your data.

Here's the issue. I can't see this level of protection in a LeftHand SAN.

LeftHand Data Protection and HA

LeftHand systems are built from commodity servers and use a battery backed RAID controller to provide a log of writes to disk. This means that when a single node (non-HA) fails, the data in the cache is replayed to the disks when the system comes back up.  

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

In a cluster, and using nRAID2, the IO is copied to a second node, the same scenario as in the single node case is played out. Effectively, cache coherency is provided by mirroring the data to a second (or third, or fourth) node across the network, which is slower and adds to latency.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

With nRAID2, when a node fails, you want to guarantee your writes because the other node(s) are now holding a single copy of your data. That nRAID2 is now no more than RAID5 striped across one or more nodes, writing data through disk cache.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

The choices to protect your data?

• Turn on write-through (turn off the disk cache and force IO straight to disk). On all disks on all nodes.
• Choose nRAID3 so you have a second mirror.

The first option causes huge IO performance problems; drives that are forced to write directly perform very badly indeed.

The second option of nRAID3 is the only alternative.

And every read is still fraught with danger. You may read a block from node A but get a completely different data from node B for the same block request -- because there's no guarantee of protection against lost writes on any of the nodes.

The LeftHand Triplication Calculator

Ok, let’s talk capacity. All NetApp’s customers know NetApp’s storage utilization is below 50% when using best practices.

NetApp's best practices are here. See page 20, section 7.4 Best Practice Configurations. This is the same old HP tap dancing.

But instead of re-hashing what everyone already knows, let’s do a simple calculation for a highly available multi-site SAN using MetroCluster (as a side note, you know Calvin is taking it easy on you with the MetroCluster pricing.)

Let’s say a customer has 10TB of NetApp raw storage at the primary site and they replicate that 10TB to a remote site for HA and disaster protection. Storage utilization is now 50% (10TB/20TB.) Take your 63% at both sites, and we won’t bother to include things like the space taken up for NetApp’s root volume and replication log files. 63% of 10TB leaves you 6.3TB of usable capacity replicated. This means you can create 6.3TB of data out of 20TB raw. That’s 31.5%.

With LeftHand’s Network RAID level 2 you can split your SAN across 2 sites for a better HA solution and the customer’s utilization, according to you, is better than NetApp’s – 35%.

Except it isn't anywhere near MetroCluster in terms of HA and data protection -- in fact it's nowhere near a single controller NetApp system in terms of data protection.

LeftHand is now down to 24% usable for an inferior solution.

24% usable

The Rest

Now it’s time for some education:

Network RAID is set at the volume level. Not all volumes require the advance data protection level of Network RAID level 2, therefore utilization is typically much better if used at a single site.

At which point, you now have a RAID5 solution with no lost write protection (or you turn on write-through on every disk and suffer huge IO penalties). Really, there's no point. Might as well buy a cheap Linux server and be done with it.

If you really want to get schooled let’s talk about dual-parity based network RAID and what that does for utilization. What we should really be talking about is cost and capacity utilization of NetApp GX vs. LeftHand, because that is the only product that comes close to LeftHand’s architecture. Does GX support block yet?

No, let's focus first on my claim that LHN means more space, more power, more cooling, more cost per usable TB for an inferior solution. Address that, and then you get bragging rights.

An HP LeftHand Duplication Calculator

Our industry drives towards greater and greater storage efficiency. Green IT was the buzz in the media for a while, and although it's not getting the column inches it got last year, being efficient hasn't gone away as a business requirement.

It's a maturation effect; once the message is clear, media and CIO attention turns to other things, and the expectation is that the IT industry will get on with it and solve the problem. The storage industry has (with exceptions) risen to the challenge, with thin provisioning, virtualization of storage, larger and more energy efficient drives, and so on. Choosing the right storage system can make big savings.

• Smaller physical footprint
• Lower power and cooling
• Cheaper to buy and maintain

The list goes on. Deduplication on primary storage is the latest, and possibly greatest, contribution to really wringing every drop of space out of your storage system. Surprisingly (to me at any rate) some of these cost-saving measures have improved performance too. For example, deduplication and caching together works just as well on cheaper, slower and cooler disks as it does on expensive, hot and fast disks.

There are a number of efficiency calculators for primary storage out on the intertubes at the moment; one from NetApp, and one from EMC for NAS only -- which was originally entitled EMC Celerra Data Duplication. Although the title has been corrected, it still delivers the same poor results as before.

Leaving that aside for the moment, what the NetApp's calculator demonstrates quite clearly is that it's possible to store more data, more efficiently than ever before. Some in the storage industry don't appear to have got the message though.

HP's LeftHand.

I was confused by a video from Chris McCall, ex LeftHand Networks and now at HP. Two points about his video struck me as a little out of kilter in his positioning of LeftHand inside HP. LHN is now part of an HP "Unified Storage" division. Hold up! Unified? The "HP LeftHand P4000 SAN" is an iSCSI SAN system only. Doesn't sound terribly unified to me.

But the claim that really had me scratching my noddle was the claim that the LHN solution meets the needs of Green IT.

LHN uses something called network RAID over the top of storage nodes; they're HP servers running RAID5. Network RAID is nothing more than a fancy term for RAID5 synchronous mirroring, something that an LHN cluster needs to do because the individual storage nodes are SPOFs -- single points of failure. So copies of data on other nodes is essential if you ever expect to see your data again.

Let's cut to the chase. Here's the HP LeftHand Duplication Calculator. Unlike NetApp's space efficiency calculator, the LHN Duplication Calculator I've designed doesn't have any input fields or buttons. That's because, regardless of how small or how large your LHN SAN, it's always...

 

35% usable

Buy 10TB, get to use 3.5TB. That's about double the raw storage you'd need to buy from NetApp. No deduplication on an LHN system either, so you'll need to buy even more from HP.

LHN means more space, more power, more cooling, more cost per usable TB. Tell me, how can HP claim that this is green? 

.

[Updated 24June2009: corrected Chris McCall of HP's name]

Grit Your Teeth and Smile Politely

When folk tell me that the chance to travel with my work must be wonderful, I just grit my teeth and smile politely. Let me tell you, it's far from wonderful.

I'm newly returned from a bout of traveling, nearly 3 weeks and 3 locations all told. Spending entire weeks away from home in business hotels, with daylight hours in bat caves, evenings eating overpriced and unhealthy food, capped by 12,000 miles of weekend air travel in conditions that animal welfare organisations would condemn as unsuitable -- that isn't my idea of a company paid-for holiday. 

To cap it all, it was my birthday last weekend, on Sunday 31st May. I'd like to thank my colleagues that bought me a beer or two at the bar of the Santa Clara Hyatt. It helped me assuage the pain of being away from home. Much appreciated, even though my memories of the evening are a little sketchy.

I'm delighted to be back now, and was looking forward to my normal schedule of work. And would you believe it, I'm being asked (yet again) about HP's latest anti-NetApp nonsense doing the rounds.

Here's a screen shot of the latest dipstickery that passes for critical analysis at HP. (Yes, the slide says it's internal use, but it's publicly available on the web and it's being given to HP partners and prospective customers, probably with a "sshhh, don't tell anyone about this, but I've got this internal report, and it says...")

I'm not sure what NetApp report is being referred to here as there are several Cost of Ownership reports we've done over the years (search on Mercer or Wyman for example), and I can't find the report that matches with page numbers in the HP slide deck. The latest is here. Anyone at HP care to tell me if that's the right one?

Whichever HP used, the slide contains more than a few errors.

• We never recommend RAID-4, because RAID-DP at 14 data plus 2 parity is far more reliable. RAID-DP is more reliable than RAID-1, the most reliable and performant of the non-dual parity schemes. With HP you get to choose; reliability (RAID-6 for truly performance sapping protection) or RAID-1 for performance and protection at serious capacity cost; RAID-5 if you like your risks and performance about equally divided. With NetApp, you get superior performance, capacity and protection from RAID-DP.
• NetApp space reservation has been consistently and deliberately misunderstood by HP. And others.
• You choose snapshots or full clones on HP, and the choice is a tough one. HP EVA snapshots perform badly and consume little space; clones perform well, but consume lots of space. NetApp snapshots and clones don't have that problem.
• There is no cost for an NDMP licence on a NetApp system. And there never has been. Ever.

Conclusion; If HP only understood NetApp's technology...

When your HP sales rep slides this across your desk and suggests that HP's products are wonderful, think business travel. Then just grit your teeth and smile politely.

.

WALL·E, It's Time To Dump EVA

I'm a big fan of Pixar, and I loved WALL·E. I even went and bought the DVD, I'm that smitten by this work of genius.

Here's the challenge that Pixar rose to, and that I think summarises the greatness of this film; it's a cartoon about WALL·E, a robot that falls in love with another robot, EVA. Yes, I'm a child when it comes to this kind of stuff, but that's some trick Pixar pull off. Reality twice removed; a cartoon about robots and love. Brilliant.

I can't love HP's EVA though. And I don't think I'm going to like the latest EVAs either. When they arrive, that is.

For months I've been hearing about the next range of HP EVA StorageWorks systems. Last year, the EVA4400 replaced the EVA4100. Then the EVA6400 and EVA8400 were supposed to replace the EVA6100 and EVA8100 respectively.

"So where does this leave the EVA6100?" asks Chet Jacobs, way back in April 2008.

It's a good question, and if anyone should know, he should; he's Chief StorageWorks Evangelist at HP. 2H2008 has come and gone, and there's been nothing about the missing boxes since then.

Can we expect a launch of an extension to the EVA range any day soon now? Probably, but I think you'll have to wait until March 10th.

So if you're a customer of HP and you're about to buy in to the HP vision of storage with an EVAx100 system (something I wouldn't recommend, but you know that already), I'd be looking to ask a lot of hard questions of your HP and HP channel sales people.

Like; So where does this leave me when I buy an EVA6100 or EVA8100?

• Is it field upgradable at no expense to me when the new model comes out? Will I have to take an outage to do the upgrade?
• What kind of performance improvement does the new range offer?
• What kind of huge discount do I get for buying yesterday's tin?
• Am I still stuck with FC connectivity only? No iSCSI? No dedupe? No NAS? No smarts at all? Are you guys kidding?

Don't be a wally (in the British sense). HP are no Pixar, and it's time to end the love affair. Dump the EVA now.

.

Because We Have To...

I spend too much time worrying. Far too much. I get concerned that our competitors are going to find out our Big Secret. A colleague has asked me to respond to a competitive analysis of NetApp's deduplication, and I can't get past this paragraph. The stress is killing me.

"So, we ask again, why is NetApp the only vendor to provide dedupe on the primary volume. Answer: because they have to."

NetApp provide leading edge technology solutions, a list so long I won't post it here; for that, our corporate website does a great job. But I still worry.

NetApp has tens of thousands of customers using these technologies to their considerable advantage, including deduplication. That's from measured data we collect from customer systems (using AutoSupport). And yet I still worry.

NetApp has a Virtualization Space Guarantee program, now extended to include other virtualization products including Microsoft's Hyper-V and Citrix' XENServer.

Ulcers. The fear is gnawing away at me. The Big Secret, our dirty laundry. I can't take the stress of this, so it's time to confess.

...it [NetApp] tries desperately to hide but at the same time seems driven to confess as if subconsciously trying to purge some unresolved guilt. 

This isn't my usual shrink speaking. This is an HP competitive storage analyst I'm quoting, but he's good, no? He's so right, that's where the stress is coming from; hiding the Big Secret.

It's time to come clean. I confess. We only provide deduplication on primary volumes because we have to.

Confession made, the guilt is gone. What a relief! I feel better already.

.

HP and Death by a Thousand VMs

Over at HP's community storage blog, HP appear to be attempting to psychoanalyze NetApp. Strange, I know, but it seems to be a tactic that keeps them happy. First, HP's Karl Dohm on WAFL; and now Jim Haberkorn on NetApp usable space and our VMware 50% Space Guarantee.

Where to start? Jim's first post and now this latest post are so long and full of hearsay, straw men and misrepresentations that it's a difficult one.

There are many, many questions that Jim asks, and I could make this an exceedingly long reply indeed. Instead, read some of the comments; I've replied (I think!) to most of the substantive points and refuted them, so they're not worth repeating here.

One is worth a mention, though. Here's Jim;

I think you can justifiably conclude that NetApp's usable capacity woes span all sorts of applications.

I could have just said "we don't have a problem" and left it there, but I actually went to some effort last time EMC accused us of this, so I said "we don't have a problem, here's the proof". The response;

NetApp has a huge usable capacity issue in many environments that it tries desperately to hide but at the same time seems driven to confess as if subconsciously trying to purge some unresolved guilt.

Very Freudian. We're apparently in denial. Need more HP evidence of NetApp in denial? Yes, it's our VMware 50% Space Guarantee

Ah, to have one in the first place. Why is this guarantee around at all? Nick Triantos sums up the whys and wherefores. Worth a read, especially to see the impressive numbers of customers and the kind of %age savings they are getting.

Meanwhile, EMC and HP are team tagging at the moment, presumably on the principle that "your a friend of mine if your enemy is my enemy". Cool, really, considering that HP had a hissy fit with EMC on the subject of usable space just recently. How short the memory, how time flies.

Anyhow, EMC and HP have kissed and made up, and they're in agreement. We still have space issues according to the good doctors at HP and EMC, with a bit of a diversionary marketing gimmick thrown in for good measure.

So how does NetApp hide all this?  Well, one way is if they have a small SAN running on the same filer as a large NAS.  With all the free space running around no one really tracks whether the SAN is being a free space hog.   

So we're still a basket case running around when it comes to those space issues, apparently.

Listen, stop playing doctors and nurses and get used to it; it's a good idea. Everyone except HP and EMC thinks it's a good idea. Gartner in a recent report on our guarantee says;

During the research phase of your next storage purchasing cycle, ask each vendor if it offers a capacity savings or utilization guarantee.

Gosh. That's a "No" for EMC and HP then. Go psychoanalyze Gartner, why don't you.

Time for me to don the white coat and stethoscope. Here's a list of psychoses the HP MSA, EVA and XP suffer from;

• RAID-5 which just isn't good enough 
• No evidence of any RAID-6 on the EVA
• HP's VMware best practices that recommend VRAID-1 (mirroring) for performance across all their platforms
• VMware over iSCSI on the EVA is not available without expensive FC/iSCSI bridges
• VMware over NFS is not available on any of the mid to high end storage offerings
• The XP has no VMware certification [Update 5Jan2009; thanks to a sharp eyed reader that pointed out thet the XP is VMWare certified.]  
• None of the systems has displayed any signs of deduplication
• Snapshots impact performance (it's that copy-on-first-write problem again)
• Lack of effective thin provisioning (the EVA's "dynamic provisioning" is actually "chubby provisioning" and anyhow it doesn't work with VMware)

Why do you need these features to remain sane in the VMware world? If there's only one link you click on in this post, make it this next one. Try getting this kind of value on any HP storage system. This is why you'll never ever see a space usage guarantee on VMware from HP on their current range of storage.

An excellent example of low cost operation is captured in this video where we host 5,440 VDI clients in 10 GB of storage and just 32 Volumes.

5,440 VDI clients in 10 GB of NetApp storage. And NetApp has got problems? I think we're fine, thanks.

But not for HP storage. This is death by a thousand VMs. And HP had better not start denying it, or we'll have grounds for thinking they've a guilty secret and got something to hide.

.

HP Takes Right Hand of LeftHand

$360m iSCSI Dowry

HP has wooed and won LeftHand Networks in deal reputedly worth $360m.

Hmmmm….

Let’s see if HP can make more of this purchase than their PolyServe acquisition. It’s certainly a poke in the eye for Dell. Their purchase of EqualLogic cost them a whopping $1.4billion to get their mits on an iSCSI solution.

When will these guys learn? It’s all about unified storage, not about bodging up hybrid monsters from the parts bin.

.