NetApp - The Missing Shade of Blue

An HP LeftHand Triplication Calculator

Part 1 of my investigation of LeftHand's claim to save money, given its less than stellar 35% usable from raw disk space, generated a number of interesting replies from HP. One is worthy of more analysis, but as it's a bit big for a comment, I've taken the liberty of extracting John Spier's reply to me. John was the former CTO of LeftHand Networks prior to its acquisition by HP.

There's a lack of information on how LeftHand does its stuff publicly available, so I've had to do a little"reading between the lines" and work from first principles. If I've got any of this wrong, please let me know, and I'll correct it. I've added a running commentary to John's comment; my apologies for breaking it up, but it's all here (in blue for clarity).

When using Network RAID 2 it protects you from multiple disk faults, complete array faults and site faults with auto failover and failback. NetApp can’t deliver this level of HA with auto failover and failback. Features like MetroCLuster give you data protection, but not HA, and at a lower capacity utilization than LeftHand. Can SnapMirror or MetroCluster automatically fail back, incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Then you'd be wrong; that's exactly what MetroCluster is about. Except the auto failback; that's just adding a disaster on top of a disaster. But I digress, and that's the subject for another post. 

It's worth pointing out before I analyse this claim that I originally thought that LeftHand had come up with a new paradigm with its network RAID; that it provided both data protection and high availability built on commodity tin. I was wrong; if it was that easy, we'd have done it. But NetApp's 15 years experience in doing this stuff has taught us otherwise.

First, I need to explain what a NetApp cluster is all about; then we can compare and contrast, and ask some questions.

NetApp Data Protection and HA

NetApp uses NVRAM (non volatile RAM) and transaction logging to capture writes to disk. All writes are acknowledged before they get to disk, but only after they've been logged. This means that when a single controller (non-HA) fails, the data we said to the server or client that we wrote, but stored in NVRAM, is replayed to the disks when the system comes back up. That way, we guarantee what we promised when we said we'd written the data. The data is both consistent and durable.

In a cluster or HA solution, we ensure cache coherency; the contents of the NVRAM on controller 1 are mirrored to controller 2. That way, when controller 1 goes down, controller 2 can replay controller 1's writes, and take over its workload, as the disks are addressable from both controllers. Again, the data is both consistent and durable; and we've made it HA, without downtime to the application. It carries on running.

If controller 2 now fails (or both fail together), we're still consistent and durable; see above for the single controller case.

Lastly, SATA drives in particular can suffer from "lost writes". Every drive has a cache where it stores data to be written. This is separate from any other protected cache, for instance NetApp's NVRAM.

As soon as an IO hits this buffer, the drive acknowledges the write. But blocks can subsequently be written in the wrong place, or not written at all, especially if there's a disk failure between acknowledgement and the physical write. 

Because NetApp has the ability to control both the RAID and the file system, Data ONTAP 7G provides the unique ability to catch errors such as this and recover. Along with a block checksum, ONTAP also stores WAFL metadata (the inode # of a file containing the block) that provide the ability to verify the validity of a block being read. If the block being read does not match what WAFL expects, the data gets reconstructed ensuring that your data is both consistent and durable.

NetApp goes to extraordinary lengths to protect your data.

Here's the issue. I can't see this level of protection in a LeftHand SAN.

LeftHand Data Protection and HA

LeftHand systems are built from commodity servers and use a battery backed RAID controller to provide a log of writes to disk. This means that when a single node (non-HA) fails, the data in the cache is replayed to the disks when the system comes back up.  

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

In a cluster, and using nRAID2, the IO is copied to a second node, the same scenario as in the single node case is played out. Effectively, cache coherency is provided by mirroring the data to a second (or third, or fourth) node across the network, which is slower and adds to latency.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

With nRAID2, when a node fails, you want to guarantee your writes because the other node(s) are now holding a single copy of your data. That nRAID2 is now no more than RAID5 striped across one or more nodes, writing data through disk cache.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

The choices to protect your data?

• Turn on write-through (turn off the disk cache and force IO straight to disk). On all disks on all nodes.
• Choose nRAID3 so you have a second mirror.

The first option causes huge IO performance problems; drives that are forced to write directly perform very badly indeed.

The second option of nRAID3 is the only alternative.

And every read is still fraught with danger. You may read a block from node A but get a completely different data from node B for the same block request -- because there's no guarantee of protection against lost writes on any of the nodes.

The LeftHand Triplication Calculator

Ok, let’s talk capacity. All NetApp’s customers know NetApp’s storage utilization is below 50% when using best practices.

NetApp's best practices are here. See page 20, section 7.4 Best Practice Configurations. This is the same old HP tap dancing.

But instead of re-hashing what everyone already knows, let’s do a simple calculation for a highly available multi-site SAN using MetroCluster (as a side note, you know Calvin is taking it easy on you with the MetroCluster pricing.)

Let’s say a customer has 10TB of NetApp raw storage at the primary site and they replicate that 10TB to a remote site for HA and disaster protection. Storage utilization is now 50% (10TB/20TB.) Take your 63% at both sites, and we won’t bother to include things like the space taken up for NetApp’s root volume and replication log files. 63% of 10TB leaves you 6.3TB of usable capacity replicated. This means you can create 6.3TB of data out of 20TB raw. That’s 31.5%.

With LeftHand’s Network RAID level 2 you can split your SAN across 2 sites for a better HA solution and the customer’s utilization, according to you, is better than NetApp’s – 35%.

Except it isn't anywhere near MetroCluster in terms of HA and data protection -- in fact it's nowhere near a single controller NetApp system in terms of data protection.

LeftHand is now down to 24% usable for an inferior solution.

24% usable

The Rest

Now it’s time for some education:

Network RAID is set at the volume level. Not all volumes require the advance data protection level of Network RAID level 2, therefore utilization is typically much better if used at a single site.

At which point, you now have a RAID5 solution with no lost write protection (or you turn on write-through on every disk and suffer huge IO penalties). Really, there's no point. Might as well buy a cheap Linux server and be done with it.

If you really want to get schooled let’s talk about dual-parity based network RAID and what that does for utilization. What we should really be talking about is cost and capacity utilization of NetApp GX vs. LeftHand, because that is the only product that comes close to LeftHand’s architecture. Does GX support block yet?

No, let's focus first on my claim that LHN means more space, more power, more cooling, more cost per usable TB for an inferior solution. Address that, and then you get bragging rights.

An HP LeftHand Duplication Calculator

Our industry drives towards greater and greater storage efficiency. Green IT was the buzz in the media for a while, and although it's not getting the column inches it got last year, being efficient hasn't gone away as a business requirement.

It's a maturation effect; once the message is clear, media and CIO attention turns to other things, and the expectation is that the IT industry will get on with it and solve the problem. The storage industry has (with exceptions) risen to the challenge, with thin provisioning, virtualization of storage, larger and more energy efficient drives, and so on. Choosing the right storage system can make big savings.

• Smaller physical footprint
• Lower power and cooling
• Cheaper to buy and maintain

The list goes on. Deduplication on primary storage is the latest, and possibly greatest, contribution to really wringing every drop of space out of your storage system. Surprisingly (to me at any rate) some of these cost-saving measures have improved performance too. For example, deduplication and caching together works just as well on cheaper, slower and cooler disks as it does on expensive, hot and fast disks.

There are a number of efficiency calculators for primary storage out on the intertubes at the moment; one from NetApp, and one from EMC for NAS only -- which was originally entitled EMC Celerra Data Duplication. Although the title has been corrected, it still delivers the same poor results as before.

Leaving that aside for the moment, what the NetApp's calculator demonstrates quite clearly is that it's possible to store more data, more efficiently than ever before. Some in the storage industry don't appear to have got the message though.

HP's LeftHand.

I was confused by a video from Chris McCall, ex LeftHand Networks and now at HP. Two points about his video struck me as a little out of kilter in his positioning of LeftHand inside HP. LHN is now part of an HP "Unified Storage" division. Hold up! Unified? The "HP LeftHand P4000 SAN" is an iSCSI SAN system only. Doesn't sound terribly unified to me.

But the claim that really had me scratching my noddle was the claim that the LHN solution meets the needs of Green IT.

LHN uses something called network RAID over the top of storage nodes; they're HP servers running RAID5. Network RAID is nothing more than a fancy term for RAID5 synchronous mirroring, something that an LHN cluster needs to do because the individual storage nodes are SPOFs -- single points of failure. So copies of data on other nodes is essential if you ever expect to see your data again.

Let's cut to the chase. Here's the HP LeftHand Duplication Calculator. Unlike NetApp's space efficiency calculator, the LHN Duplication Calculator I've designed doesn't have any input fields or buttons. That's because, regardless of how small or how large your LHN SAN, it's always...

 

35% usable

Buy 10TB, get to use 3.5TB. That's about double the raw storage you'd need to buy from NetApp. No deduplication on an LHN system either, so you'll need to buy even more from HP.

LHN means more space, more power, more cooling, more cost per usable TB. Tell me, how can HP claim that this is green? 

.

[Updated 24June2009: corrected Chris McCall of HP's name]

Venezuelan Beaver Cheese

The EMC blogosphere is full of "Why I Work For EMC!" posts at the moment, and to be honest, it's a concerted strategy to present EMC as the coolest and bestest employer in the tech industry. Ever.

I'm not in any doubt as to why, and if you've been following the press recently, I suspect you aren't in any doubt either. EMC bloggers are supporting the central thesis of Uncle Joe's open letter to Data Domain employees, that EMC is one of those rare things, a company that's coolest and best and biggest, the undisputed leader of companies, a world-class, #1 company.

With one voice, they're crying "I love it here! And so will you."  Which leads the inquisitive (actually, in my case, paranoid) amongst us to wonder if this is an orchestrated attempt to provide a bit of a public shove and push to the idea that corporate culture at EMC is just fine and dandy, and that there nothing to fear from being acquired. As an echo and reinforcement of Joe Tucci's message, the blogs I’ve seen definitely toe the line.

Whatever the agenda, what are they saying about why they love EMC so? There's a lot of high-falutin' sentiments on display; some personal observations about what it's like to work there, how to get promoted, where to wander round corridors talking to engineers about cool stuff, advice on what it takes to prosper, enjoy, fit in, etc etc.

The EMC they describe is like Venezuelan Beaver Cheese.

I love cheese. I've eaten a huge amount and variety in my time, from the humble Cheddar to the exotic ash covered Saint Maure. But if it was possible, the one I would really want to taste is the very rare Venezuelan Beaver Cheese. Some would have you believe that it's more than just a cheese; that it's the coolest and best and biggest cheese, the undisputed leader of cheeses, a world-class, #1 cheese.

Unfortunately, Venezuelan Beaver Cheese isn't just rare, it doesn't exist at all outside of a Monty Python sketch. Just like the picture of working at EMC we're getting from these blogs. I don't believe they're being honest with themselves.

EMC is probably more like a ripe Vieux Boulogne, a cheese that’s an acquired taste best washed down with strong drink. Co-incidentally, it has the unenviable reputation for being the world's smelliest.

---

Update 15June2009: In the "I wish I'd said that" category; Jay Kidd's blog post on EMC's bid for Data Domain.

Grit Your Teeth and Smile Politely

When folk tell me that the chance to travel with my work must be wonderful, I just grit my teeth and smile politely. Let me tell you, it's far from wonderful.

I'm newly returned from a bout of traveling, nearly 3 weeks and 3 locations all told. Spending entire weeks away from home in business hotels, with daylight hours in bat caves, evenings eating overpriced and unhealthy food, capped by 12,000 miles of weekend air travel in conditions that animal welfare organisations would condemn as unsuitable -- that isn't my idea of a company paid-for holiday. 

To cap it all, it was my birthday last weekend, on Sunday 31st May. I'd like to thank my colleagues that bought me a beer or two at the bar of the Santa Clara Hyatt. It helped me assuage the pain of being away from home. Much appreciated, even though my memories of the evening are a little sketchy.

I'm delighted to be back now, and was looking forward to my normal schedule of work. And would you believe it, I'm being asked (yet again) about HP's latest anti-NetApp nonsense doing the rounds.

Here's a screen shot of the latest dipstickery that passes for critical analysis at HP. (Yes, the slide says it's internal use, but it's publicly available on the web and it's being given to HP partners and prospective customers, probably with a "sshhh, don't tell anyone about this, but I've got this internal report, and it says...")

I'm not sure what NetApp report is being referred to here as there are several Cost of Ownership reports we've done over the years (search on Mercer or Wyman for example), and I can't find the report that matches with page numbers in the HP slide deck. The latest is here. Anyone at HP care to tell me if that's the right one?

Whichever HP used, the slide contains more than a few errors.

• We never recommend RAID-4, because RAID-DP at 14 data plus 2 parity is far more reliable. RAID-DP is more reliable than RAID-1, the most reliable and performant of the non-dual parity schemes. With HP you get to choose; reliability (RAID-6 for truly performance sapping protection) or RAID-1 for performance and protection at serious capacity cost; RAID-5 if you like your risks and performance about equally divided. With NetApp, you get superior performance, capacity and protection from RAID-DP.
• NetApp space reservation has been consistently and deliberately misunderstood by HP. And others.
• You choose snapshots or full clones on HP, and the choice is a tough one. HP EVA snapshots perform badly and consume little space; clones perform well, but consume lots of space. NetApp snapshots and clones don't have that problem.
• There is no cost for an NDMP licence on a NetApp system. And there never has been. Ever.

Conclusion; If HP only understood NetApp's technology...

When your HP sales rep slides this across your desk and suggests that HP's products are wonderful, think business travel. Then just grit your teeth and smile politely.

.