NetApp - The Missing Shade of Blue

Random Rocks And Benchmarks

“People understand contests. You take a bunch of kids throwing rocks at random and people look askance, but if you go and hold a rock-throwing contest -- people understand that.” (Don Murray)

And that, in a nutshell, is the origin of most competitive sports. What starts as "Hey, I can chuck this boulder through that window over there -- betcha you can't!" develops through time a set of rules, referees, measuring devices, a governing body (sometimes two or three) and the sport of rock-chucking may even get recognised by many, played by many and make it into the Olympics. Like spear throwing (the javelin) or blasting birds out of the sky with a shotgun (clay pigeon shooting), rock-chucking has matured into shot-putting.

Just like benchmarks. Originally used to demonstrate some feature of your system in all its one-trick glory (HP IOPS from cache is a classic example of how to do this), modern industry benchmarks make the attempt to allow some real-world comparison between competitors. The relationship to real life is debatable, but the rules by which they operate are carefully designed to inject some aspects of realism to comparative claims.

Benchmarks are intended to be a repeatable test of a set of skills. But, as with all competitive sports, sometimes there's the kid who starts his benchmarking career hanging round on street corners stoning passers-by for entertainment, or hoisting up the nearest large rock at hand and heaving it through the closest plate-glass window.

And the new kid is over at HP, using a non-benchmark as a benchmark, and generally lobbing rocks around in all directions. I'm not going to dissect the post in detail, because others have and will continue to do that. There's one little paragraph I want to focus on, because it demonstrates one of my pet peeves; benchmark intuition.

Now things were starting to make sense.   We were seeing the same sort of decay curve as shown in the IOMeter results posted in Making Sense of WAFL - Part 4.    Every time the test is run, the random component of the Jetstress database accesses fragment the LUN further and the throughput numbers get worse.  An array like EMC CX or HP EVA wont undergo this sort of decay curve since these arrays do not have internal WAFL-fragmentation problems like the FAS does.

The non-benchmark is Microsoft's ESRP, and the tester's intuitive assumption is that WAFL fragments; hence the tester's intuitive assumption that this is the source of the diminishing throughput numbers.

Bzzzt. Big fail.

Let's allow the first intuition; let's allow, for the sake of this demonstration, that "WAFL fragments your data". Here's a simple example to demonstrate why his intuition is wrong on fragmentation being the source of the problem. Exchange 2007 generates small random IOs (and that's the JetStress that HP are using in their test). The table below has 5 columns to demonstrate why small random IO works just as well (or badly, depending on your take) on randomly laid out data as sequentially laid out data.

• Random Placement: I've place 100 blocks randomly. The numbers have been generated from www.random.org. Slot 1 is block 67, slot 2 is block 19 and so on.
• Random Requested Block: this is meant to simulate IO requests from Exchange; again, drawn from a different run from www.random.org.
• Matching Block (Random Placement): this is where the requested block actually lives, So asking for block 80 requires a visit to slot 22, and so on.
• Seek Distance (Random Placement): this is the effective seek distance between requested random blocks. After we visit slot 22 (for block 80), we need to visit slot 58 (for block 37), requiring a seek of 36 slots.
• Seek Distance (Sequential Placement): this is the effective seek distance between requested sequential blocks. After we visit slot 80 (for block 80), we need to visit slot 37 (for block 37), requiring a seek of 43 slots.

(I'm having difficulty uploading the spreadsheet to TypePad, so when I get it fixed you'll be able to "Click to download the whole spreadsheet". Not yet though.)

Random Placement	Random Requested Block	Matching Block (Random Placement)	Seek Distance (Random Placement)	Seek Distance (Sequential Placement)
67	80	22	0	0
19	37	58	36	43
75	18	61	3	19
23	26	53	8	8
85	57	63	10	31
59	100	14	49	43
14	59	6	8	41
...	...	...	...	...
SUM			3269	3322

Hey, look at that! The sequentially laid out data takes more slot seeks than the randomly laid out data! Try it yourself, replace the 100 numbers in the first two columns with random numbers, and check the seek distance sum.

On average, they will be equal. In fact, if the requested blocks are random, it doesn't matter how the data is laid out. Intuition fail.

Here's the professional sport of benchmarking, which HP don't take part in (still being, as it were, at the rock-throwing stage);

• NetApp SPC Benchmarks (no HP here)
• SFS2008 NFS benchmarks (HP missing again)

And, of course, the official ESRP results (and, just as a reminder, these aren't benchmarks)

• Microsoft ESRP results

Having failed at shot-putting, perhaps HP might want to pick another sport for their talented testers. Like nude football.

[updated to correct some borked links and a typo].

Free Cheese from EMC

EMC's Barry Burke seems mighty pleased to announce that Virtual Provisioning (what everyone else calls thin provisioning) is now "free" (at least on the V-Max and DMX4 storage arrays but not on the CLARiiON).

I'm having a remarkably similar argument with HP folks on their assertion that the software on their LeftHand storage arrays is "free", even though there's a big difference in the price of similarly specified servers and the P4000 based on the same tin. The same with Sun; their "free" software on the the Sun 7210 array costs a lot. The system is based on a Sun X4540, which is much cheaper.

And Barry's challenge "We'll see if others follow suit" line is just so, well, yesterday's news. There are any number of storage vendors that have thin provisioning at no additional cost, including NetApp FAS systems; since 2004 to be precise.

What Barry means by free is at no additional cost, and it's an important difference. And I don't  think buyers are fooled by this use of the word. It's the total cost that you need to look at, including the free. He eventually gets there; right in the last-but-one paragraph where it's "all at no extra charge".

Free? There's always free cheese in mouse traps, but the mice there aren't best pleased.

.

LeftHand Quorums and Split Brains

There's been a bit of a discussion going on between John Spiers (HP LeftHand) and myself over a number of issues I raised about a LeftHand SAN's storage efficiency. The comments moved on to talk about HA (high availability) and data reliability, and John raised a number of questions that I thought deserved a longer answer.

John Martin of NetApp (who also blogs on NetApp's Storage Efficiency blog) has kindly provided me with more detail, and rather than post this as a comment, I thought it worthy (again) of a blog in its own right. Thanks to John for the responses.

John Spiers' points are in blue (and I think I've accurately captured them, but they've been lightly [edited] for context). They're also slightly out of order.

In summary, I think there's a need for greater clarity on LeftHand best practices. Everyone, including the user quoted below, appears to be operating in the dark. As I said in a previous post, we've had to do a little"reading between the lines" and work from first principles. If any of this is wrong, please let me know, and I'll correct it.

[Update 09July: Many LeftHand manuals, including best practices, appeared in late June/July, but weren't there when I first looked in late May/early June. Much reading to do!]

---

NetApp can't deliver this level of HA with auto failover and failback [compared to a LeftHand SAN]

It depends on what you mean by "auto failback". If you mean failback initiated without permission or authorization from a responsible human being, then you'd be correct. If you mean an automated process with minimal user intervention, then you're wrong. From a NetApp perspective, and that of most storage and systems professionals I've talked to when discussing failback requirements, automated and uncontrolled failback is usually judged to be a bad idea.

NetApp also have an excellent product in addition to MetroCluster, called MultiStore. This provides similar kinds of functionality (automated failover and failback) over standard IP connections.

Can SnapMirror or MetroCluster [...] incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Yes it can. As soon as the administrator believes it to be safe, the "failback" process is initiated, and an automatic incremental rebuild and resync provides seamless failback.

[quoting from a NetApp document] "Mirrored active/active configurations do not provide the capability to fail over to the partner node if one node is completely lost. For example, if power is lost to one entire node, including its storage, you cannot fail over to the other node. For this capability, use a MetroCluster"

[quotes sections about manual cluster failover and prevention of “split brain” and JS then says] LeftHand has distributed quorum management that eliminates all possibilities of a “split brain”. This allows at least one site to operate and then automatically resync the other sites when they come back online.

This is an interesting quote, which when taken out of its correct context (a practice commonly called "quote mining") makes it sound much worse than when it's put in context.

1. It should be noted that this is for a VERY unusual configuration, one that I’ve never seen go into production at any site. The comment is under the heading “Mirrored Active/Active Configurations”. With this, NetApp's Syncmirror is added to a standard active/active configuration to provide a second level of mirroring across the disks, but without a MetroCluster license for local (or stretch) functionality (suitable for distances of around 500m).
2. Should a customer decides that his data required the extra protection of Syncmirror (i.e. RAID 6+1), then we would recommend the extra resilience provided for in this configuration by using MetroCluster.

The reason for this behavior is to avoid a “split brain” scenario. There is no way of automatically detecting the difference between a total failure of a system, or the the failure of all forms of communication between them. It's this that causes "split brain"; two or more running systems thinking the other has failed, when only the communication between them has failed.

This applies to every conceivable cluster configuration. That includes quorum disks, proxy nodes and other cluster node failure detection techniques; they are all effectively forms of communication between nodes.

Distributed quorum management requires at least three nodes. It is not possible to have a quorum based on two nodes.

I’m not letting you brush this one under the rug. MetroCluster and SyncMirror don’t provide the same level of availability that is inherent in LeftHand’s base SAN/iQ software offering, and LeftHand requires no additional equipment.

That's interesting, because here's a LeftHand user experience;

"So at work we decided to go with a Lefthand Implementation of iSCSI, I am rather unhappy to find out that with only 2 units you have to run a virtual machine to complete the Quorum for redundancy. I am not happy to find out that in reality you need 3 appliances to complete a Quorum for management and to ensure that you have redundancy and that everything is available."

You should have been clearer about comparing a three or more node site vs. a two node site. This HA "no additional equipment solution" is growing legs.

And while I’ll give kudos for an n-way implementation, unlike the NetApp documentation you quote, you don’t address how you handle what happens when there is a complete loss of communication only from the primary site to all other sites. None of the other nodes are able to detect whether the site is a smoldering pile of rubble, or if it’s just incommunicado, and if you make the assumption that lack of communication is a trigger for failover, then you have a recipe for split brain.

The manual “declared disaster” approach is the safest way of dealing with this. If a customer believes they have a foolproof way of detecting a true site failure, then its trivially easy to integrate this by automating MetroCluster failover (or failback for that matter)

Yes, [HP's LeftHand] Network RAID 2 can sustain any random 5 disk failure. In fact, a 4 array system configured in Network RAID level 2 can sustain up to 2 complete array failures, and up to 6 disk failures in each of the remaining 2 arrays, and all at the same time.

Hmmm... Picture courtesy of HP's LeftHand P4000 brochure 4AA2-5247ENW, April 2009.

Let's start with nodes. I have two copies of my data for nRAID2, and four arrays. If any two nodes fail, then I lose data. Which two arrays can I blow away in this four node system, and not lose data in my logical volume? With this simplistic 4 block LUN, 1 and 3 or 2 and 4. Unless you can arrange your node failures in advance, I'd suggest that is one node protection.

And the statement that I can survive any random 5 disk failure is equally implausible given the LeftHand diagram above. Two disk failures in node 1 in the same RAID5 group would kill node 1. A further two disk failures in a RAID5 group kill node 2, and we now have two node failures which can't be survived. The answer for nRAID2 is 3 random disk failures, not 5.

I'm not even going to try and work out what "up to 6 disk failures in the remaining 2 arrays" really means, although I'll bet the words "random" and "quorum" don't feature heavily. 

Of course, this all assumes the HP LeftHand diagram above is correct, and not just a graphic designer's interpretation of the facts. The diagram below for the terrible usable capacity of a LeftHand system for 1 node failure protection (split brained or otherwise) is correct. I've had it double checked by HP.

LeftHand nRAID2 means 35% usable capacity

.

An HP LeftHand Triplication Calculator

Part 1 of my investigation of LeftHand's claim to save money, given its less than stellar 35% usable from raw disk space, generated a number of interesting replies from HP. One is worthy of more analysis, but as it's a bit big for a comment, I've taken the liberty of extracting John Spier's reply to me. John was the former CTO of LeftHand Networks prior to its acquisition by HP.

There's a lack of information on how LeftHand does its stuff publicly available, so I've had to do a little"reading between the lines" and work from first principles. If I've got any of this wrong, please let me know, and I'll correct it. I've added a running commentary to John's comment; my apologies for breaking it up, but it's all here (in blue for clarity).

When using Network RAID 2 it protects you from multiple disk faults, complete array faults and site faults with auto failover and failback. NetApp can’t deliver this level of HA with auto failover and failback. Features like MetroCLuster give you data protection, but not HA, and at a lower capacity utilization than LeftHand. Can SnapMirror or MetroCluster automatically fail back, incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Then you'd be wrong; that's exactly what MetroCluster is about. Except the auto failback; that's just adding a disaster on top of a disaster. But I digress, and that's the subject for another post. 

It's worth pointing out before I analyse this claim that I originally thought that LeftHand had come up with a new paradigm with its network RAID; that it provided both data protection and high availability built on commodity tin. I was wrong; if it was that easy, we'd have done it. But NetApp's 15 years experience in doing this stuff has taught us otherwise.

First, I need to explain what a NetApp cluster is all about; then we can compare and contrast, and ask some questions.

NetApp Data Protection and HA

NetApp uses NVRAM (non volatile RAM) and transaction logging to capture writes to disk. All writes are acknowledged before they get to disk, but only after they've been logged. This means that when a single controller (non-HA) fails, the data we said to the server or client that we wrote, but stored in NVRAM, is replayed to the disks when the system comes back up. That way, we guarantee what we promised when we said we'd written the data. The data is both consistent and durable.

In a cluster or HA solution, we ensure cache coherency; the contents of the NVRAM on controller 1 are mirrored to controller 2. That way, when controller 1 goes down, controller 2 can replay controller 1's writes, and take over its workload, as the disks are addressable from both controllers. Again, the data is both consistent and durable; and we've made it HA, without downtime to the application. It carries on running.

If controller 2 now fails (or both fail together), we're still consistent and durable; see above for the single controller case.

Lastly, SATA drives in particular can suffer from "lost writes". Every drive has a cache where it stores data to be written. This is separate from any other protected cache, for instance NetApp's NVRAM.

As soon as an IO hits this buffer, the drive acknowledges the write. But blocks can subsequently be written in the wrong place, or not written at all, especially if there's a disk failure between acknowledgement and the physical write. 

Because NetApp has the ability to control both the RAID and the file system, Data ONTAP 7G provides the unique ability to catch errors such as this and recover. Along with a block checksum, ONTAP also stores WAFL metadata (the inode # of a file containing the block) that provide the ability to verify the validity of a block being read. If the block being read does not match what WAFL expects, the data gets reconstructed ensuring that your data is both consistent and durable.

NetApp goes to extraordinary lengths to protect your data.

Here's the issue. I can't see this level of protection in a LeftHand SAN.

LeftHand Data Protection and HA

LeftHand systems are built from commodity servers and use a battery backed RAID controller to provide a log of writes to disk. This means that when a single node (non-HA) fails, the data in the cache is replayed to the disks when the system comes back up.  

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

In a cluster, and using nRAID2, the IO is copied to a second node, the same scenario as in the single node case is played out. Effectively, cache coherency is provided by mirroring the data to a second (or third, or fourth) node across the network, which is slower and adds to latency.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

With nRAID2, when a node fails, you want to guarantee your writes because the other node(s) are now holding a single copy of your data. That nRAID2 is now no more than RAID5 striped across one or more nodes, writing data through disk cache.

But what about lost writes? Do LeftHand SANs provided protection against failure to write data correctly from the disk's cache?

The choices to protect your data?

• Turn on write-through (turn off the disk cache and force IO straight to disk). On all disks on all nodes.
• Choose nRAID3 so you have a second mirror.

The first option causes huge IO performance problems; drives that are forced to write directly perform very badly indeed.

The second option of nRAID3 is the only alternative.

And every read is still fraught with danger. You may read a block from node A but get a completely different data from node B for the same block request -- because there's no guarantee of protection against lost writes on any of the nodes.

The LeftHand Triplication Calculator

Ok, let’s talk capacity. All NetApp’s customers know NetApp’s storage utilization is below 50% when using best practices.

NetApp's best practices are here. See page 20, section 7.4 Best Practice Configurations. This is the same old HP tap dancing.

But instead of re-hashing what everyone already knows, let’s do a simple calculation for a highly available multi-site SAN using MetroCluster (as a side note, you know Calvin is taking it easy on you with the MetroCluster pricing.)

Let’s say a customer has 10TB of NetApp raw storage at the primary site and they replicate that 10TB to a remote site for HA and disaster protection. Storage utilization is now 50% (10TB/20TB.) Take your 63% at both sites, and we won’t bother to include things like the space taken up for NetApp’s root volume and replication log files. 63% of 10TB leaves you 6.3TB of usable capacity replicated. This means you can create 6.3TB of data out of 20TB raw. That’s 31.5%.

With LeftHand’s Network RAID level 2 you can split your SAN across 2 sites for a better HA solution and the customer’s utilization, according to you, is better than NetApp’s – 35%.

Except it isn't anywhere near MetroCluster in terms of HA and data protection -- in fact it's nowhere near a single controller NetApp system in terms of data protection.

LeftHand is now down to 24% usable for an inferior solution.

24% usable

The Rest

Now it’s time for some education:

Network RAID is set at the volume level. Not all volumes require the advance data protection level of Network RAID level 2, therefore utilization is typically much better if used at a single site.

At which point, you now have a RAID5 solution with no lost write protection (or you turn on write-through on every disk and suffer huge IO penalties). Really, there's no point. Might as well buy a cheap Linux server and be done with it.

If you really want to get schooled let’s talk about dual-parity based network RAID and what that does for utilization. What we should really be talking about is cost and capacity utilization of NetApp GX vs. LeftHand, because that is the only product that comes close to LeftHand’s architecture. Does GX support block yet?

No, let's focus first on my claim that LHN means more space, more power, more cooling, more cost per usable TB for an inferior solution. Address that, and then you get bragging rights.

An HP LeftHand Duplication Calculator

Our industry drives towards greater and greater storage efficiency. Green IT was the buzz in the media for a while, and although it's not getting the column inches it got last year, being efficient hasn't gone away as a business requirement.

It's a maturation effect; once the message is clear, media and CIO attention turns to other things, and the expectation is that the IT industry will get on with it and solve the problem. The storage industry has (with exceptions) risen to the challenge, with thin provisioning, virtualization of storage, larger and more energy efficient drives, and so on. Choosing the right storage system can make big savings.

• Smaller physical footprint
• Lower power and cooling
• Cheaper to buy and maintain

The list goes on. Deduplication on primary storage is the latest, and possibly greatest, contribution to really wringing every drop of space out of your storage system. Surprisingly (to me at any rate) some of these cost-saving measures have improved performance too. For example, deduplication and caching together works just as well on cheaper, slower and cooler disks as it does on expensive, hot and fast disks.

There are a number of efficiency calculators for primary storage out on the intertubes at the moment; one from NetApp, and one from EMC for NAS only -- which was originally entitled EMC Celerra Data Duplication. Although the title has been corrected, it still delivers the same poor results as before.

Leaving that aside for the moment, what the NetApp's calculator demonstrates quite clearly is that it's possible to store more data, more efficiently than ever before. Some in the storage industry don't appear to have got the message though.

HP's LeftHand.

I was confused by a video from Chris McCall, ex LeftHand Networks and now at HP. Two points about his video struck me as a little out of kilter in his positioning of LeftHand inside HP. LHN is now part of an HP "Unified Storage" division. Hold up! Unified? The "HP LeftHand P4000 SAN" is an iSCSI SAN system only. Doesn't sound terribly unified to me.

But the claim that really had me scratching my noddle was the claim that the LHN solution meets the needs of Green IT.

LHN uses something called network RAID over the top of storage nodes; they're HP servers running RAID5. Network RAID is nothing more than a fancy term for RAID5 synchronous mirroring, something that an LHN cluster needs to do because the individual storage nodes are SPOFs -- single points of failure. So copies of data on other nodes is essential if you ever expect to see your data again.

Let's cut to the chase. Here's the HP LeftHand Duplication Calculator. Unlike NetApp's space efficiency calculator, the LHN Duplication Calculator I've designed doesn't have any input fields or buttons. That's because, regardless of how small or how large your LHN SAN, it's always...

 

35% usable

Buy 10TB, get to use 3.5TB. That's about double the raw storage you'd need to buy from NetApp. No deduplication on an LHN system either, so you'll need to buy even more from HP.

LHN means more space, more power, more cooling, more cost per usable TB. Tell me, how can HP claim that this is green? 

.

[Updated 24June2009: corrected Chris McCall of HP's name]

Grit Your Teeth and Smile Politely

When folk tell me that the chance to travel with my work must be wonderful, I just grit my teeth and smile politely. Let me tell you, it's far from wonderful.

I'm newly returned from a bout of traveling, nearly 3 weeks and 3 locations all told. Spending entire weeks away from home in business hotels, with daylight hours in bat caves, evenings eating overpriced and unhealthy food, capped by 12,000 miles of weekend air travel in conditions that animal welfare organisations would condemn as unsuitable -- that isn't my idea of a company paid-for holiday. 

To cap it all, it was my birthday last weekend, on Sunday 31st May. I'd like to thank my colleagues that bought me a beer or two at the bar of the Santa Clara Hyatt. It helped me assuage the pain of being away from home. Much appreciated, even though my memories of the evening are a little sketchy.

I'm delighted to be back now, and was looking forward to my normal schedule of work. And would you believe it, I'm being asked (yet again) about HP's latest anti-NetApp nonsense doing the rounds.

Here's a screen shot of the latest dipstickery that passes for critical analysis at HP. (Yes, the slide says it's internal use, but it's publicly available on the web and it's being given to HP partners and prospective customers, probably with a "sshhh, don't tell anyone about this, but I've got this internal report, and it says...")

I'm not sure what NetApp report is being referred to here as there are several Cost of Ownership reports we've done over the years (search on Mercer or Wyman for example), and I can't find the report that matches with page numbers in the HP slide deck. The latest is here. Anyone at HP care to tell me if that's the right one?

Whichever HP used, the slide contains more than a few errors.

• We never recommend RAID-4, because RAID-DP at 14 data plus 2 parity is far more reliable. RAID-DP is more reliable than RAID-1, the most reliable and performant of the non-dual parity schemes. With HP you get to choose; reliability (RAID-6 for truly performance sapping protection) or RAID-1 for performance and protection at serious capacity cost; RAID-5 if you like your risks and performance about equally divided. With NetApp, you get superior performance, capacity and protection from RAID-DP.
• NetApp space reservation has been consistently and deliberately misunderstood by HP. And others.
• You choose snapshots or full clones on HP, and the choice is a tough one. HP EVA snapshots perform badly and consume little space; clones perform well, but consume lots of space. NetApp snapshots and clones don't have that problem.
• There is no cost for an NDMP licence on a NetApp system. And there never has been. Ever.

Conclusion; If HP only understood NetApp's technology...

When your HP sales rep slides this across your desk and suggests that HP's products are wonderful, think business travel. Then just grit your teeth and smile politely.

.

Pay More Tax and Retire at 70

The last time I felt overwhelmed by big numbers was when UK's Chancellor of the Exchequer Alastair Darling made his last budget announcement, and he declared that there was a borrowing requirement of some 175 billion for the public sector economy. That's an eye-wateringly large number, much, much larger than any previous government's borrowing requirement.

As Senator Everett Dirkson once noted, "A billion here, a billion there, pretty soon it adds up to real money." We're in real money territory now.

And we (at least us poor unfortunates resident in the UK) will be paying for this debt for years; some estimate until 2020 or beyond. Others reckon the debt is so large, it needs higher taxes now and we'll not be retiring until we're much older.

If you live in the UK, Darling's budget means you've no choice. Pay more tax and plan on retiring at 70. Either that, or it's emigrate somewhere a little less financially punishing.

EMC have exactly the same attitude to budgeting as the UK government.

In his latest blog on the Symmetrix V-Max, Chuck Hollis is in big numbers mood, Dirkson- and Darling-sized big numbers. Go take a look and report back here on how you feel. I was surprised -- nay, shocked, I tell you -- at the bazillions of zeroes required to tell us what EMC did in developing the V-Max.

Having your eyeballs seared by all these digits makes it difficult to get at the detail, but one number did catch my eye, because Chuck beats on it. The amount of money spent on developing the V-Max.

A cool $200 million in staff costs alone, never mind the 1.5 acres of air conditioned machine room, or the 135,000 disks, or the total electrical output of a small country  required to bring this project to fruition. I suspect the real costs were nearer $400 or even $500 million if that's the staffing figure alone.

Gadzooks, but that's a lot of money. So who pays? 

The sales of the Symmetrix DMX have been on a steep downward trajectory for some time. That's because consumers aren't buying it in the same volumes because they don't have the need (their business has shrunk) or the money isn't there (they used to be a bank). The lack of growth is generally true of the entire IT industry, but the "big iron" has taken more than it's fair share of the beating. And the V-Max is big iron -- just look at all those numbers!

So my bet is on anyone that buys anything from EMC -- CLARiiON, Celerra, Centera, DMX, or even V-Max, Documentum, RSA, maintenance, support etc -- will pay the premium of getting EMC some of their money back, given EMC's less than stellar financial performance recently. Because the C in EMC doesn't stand for Charity, that's for sure. And the E doesn't look like it stands for Economical.

If you bought into EMC, the V-Max budget means you've no choice. Pay more tax and plan on retiring at 70. Either that, or you might want to consider migrating to something a little less financially punishing.

.

Add Lightness

My regular readers (5 or so) will now know that I'm the proud possessor of a Caterham 7, a direct descendant of Colin Chapman's iconic flying machine, the 50 year old Lotus 7.

Constructed from a tubular space frame chassis, clad in as little bodywork as is possible, and with small and light engines, the fundamentals of a car that was said to be “too fast to race” hasn’t changed in half a century. Chapman held that:

"Adding power makes you faster on the straights. Subtracting weight makes you faster everywhere."

He was one of the great engineering geniuses of the race and motor industry.

While doing a bit of research on cloud architectures, I stumbled across an email archived on an internal engineering wiki, sent by Dave Hitz way back in 1997. The discussion was about adding a specific feature to a NetApp product, and the upsides and the downsides of doing so. (I've edited Dave's missive for brevity.)

Hey -- I love philosophical controversies, so I'm going to dive right in!

... One of the most important questions you can ask about an appliance is: Should we add this feature?

... The potential drawbacks include:

• The feature decreases one or more of "fast, simple, reliable."
• Testing the feature (in every new release FOREVER) slows down our release cycle and hurts time-to-market.
• Documenting the new feature makes the manual thicker and harder to use.
• The feature has disadvantages that are hard to explain, and customers who try to use the feature may get burned.
• Very few customers actually need the feature, reducing benefit.
• The feature is expensive to implement...

So how are my little car's design and Hitz's email connected?

I'm not a engineer of either motor or software variety, but I see the similarities. What Hitz is identifying are the same engineering principles as espoused by Chapman, applied here to NetApp's software. As Einstein said (yes, it's rent-a-quote week here at ShadeOfBlue Towers);

"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction."

When I look around at the creeping featuritis that rival storage systems employ, the scale and size of the moving parts, and the horrendous complexity of the solutions they sell, I'm sure that the competition are employing intelligent fools. They certainly show little appetite to move in the opposite direction. I'll not bore you with examples, as I'm sure you know who they are and what systems and solutions they sell.

That's what struck me about NetApp when I joined NetApp some 4 or so years ago, and strikes me now. Our solutions are no more complex than they need to be; fast, simple, reliable are still the hallmarks.

But there is one item I'd like to add to Dave's email; I believe there's an extra quality that shines thorough in all our solutions and proposals. Colin Chapman, in possibly his most famous quote, said; 

"Simplify. Then add lightness."

That's the feature I think we have in abundance; added lightness. So when you're buying storage solutions -- heck, if you're buying anything -- keep Dave's list in mind for your next RFP/RFQ/RFI, And then add in this; 

In the proposed solution, please identify where you added lightness.

---

Update 8May: Steve Klinkner has an excellent series of posts on adding lightness. I've just read them (sorry Steve, I sometimes don't have enough time to keep up!), and this is one of the proofs that we take this "fast, simple, reliable" goal very seriously indeed. Well worth a read to see the kind of thinking that goes into the development of our software.

.

Please Squeeze Me (Part 3)

In two previous blog entries I explored EMC's late entry into the deduplication market with the EMC Celerra Data Deduplication product.

I used the contents of my laptop to demonstrate that getting to 35% savings was a bit of a squeeze using EMC's technique for deduplicating.Perhaps it wasn't a good representative sample of the kind of data you might typically see on home directories, and that it might be better with real life user data.

Nothing like real life data for making a point.

So here are some real numbers from a real user. I've used anonymous customer data before, in a usable space analysis. Using some advanced technology (in this case, NetApp's Information Server 1200) I can make a much more accurate assessment of the capabilities of EMC's latest but not so greatest. In fact, for customers looking to see how much EMC deduplication can't save them, I can strongly recommend its analytic capabilities.

The conclusion? 30% deduplication on the Celerra, with some generous assumptions. See below the line for the gory details.

The real figures will be much worse, and look out for all these other caveats.

EMC dedupe:

• is for Celerra only
• disables MPFS 
• does not support VMware (VMDKs)
• does not support LUNs (iSCSI or FC SAN)
• is only done at a file level.
• is targeted at only infrequently accessed (inactive) files
• only works on file sizes less than 200MB
• compression only works on file sizes greater than 24K
• has a performance impact in reading deduped and compressed files.
• restricts backups and restores to full volumes 
• means file level restores have to use SavVols and CoFR (copy-on-first-write) snapshots that impact performance

What will it look like on your data? 20% space savings? Less? The EMC calculator doesn't work on actual data, just %age estimates by file type. Give it try.

NetApp provides two calculators. A deduplication calculator, like the EMC tool (except that NetApp's deduplication works across all primary data, not just NAS files), and SSET, a space saving estimation tool that uses actual data on your system. Actual results are within +/-5% of the space savings that SSET predicts.

I still think it looks like all the world to me like EMC are squeezing its customers, not your data. And, missing in action; V-Max deduplication?

---

The Details

EMC's "Deduplication"

Let's recap. For home directories, EMC's deduplication uses the following rules for "cold" or not frequently accessed files that pass filtering;

• If the file is less than 24K, don't compress, only deduplicate
• If the file is >200MB, don't compress or deduplicate; leave it alone
• Otherwise, compress and deduplicate

For "cold", I'm using a figure of 90% of the data; and I'm also assuming that all files between 0K and 250MB are compressible by 50%. That will make the EMC numbers much better than are achievable in practice. In reality, YMMV. You might see a smaller or larger %age of files as cold, and for certain, not all files will compress by 50%; many will not compress at all.

But as you'll see, my original 35% was generous.

Large Files Still Dominate

As I noted before, large files may be few in number, but the represent the bulk of the data. Here's a sample from a user system. This file system contains 20 million files in just under 13TB; it's CIFS home directory data. (Click to see full sized).

Small files make up the bulk of the raw file count, and the large files in excess of 250MB are very few in number (less than 0.03% of the total file count).

But those large files over 250MB make up over 41% of the used space.

That leaves 59% of the data for EMC to deduplicate (well, compress first and deduplicate second).

Duplicates

Duplicate files are interesting. There are very few duplicates for files in excess of 50MB, and the savings from deduplicating those show a 1% decrease. So not deduplicating them has very little effect.

Files smaller than 50MB show an average of 13% duplicate space savings, although over 30% of files are duplicate by count. Here, it's medium sized files; most of them make up the duplicates. Small sized files are rarely duplicates. I'll err on the side of safety and use 10%.

At least EMC are deduplicating the right files!

Active Files

In terms of active files, 35% of the space was made up of active PSTs. EMC's deduplication doesn't do active files, and again, that's a big chunk of data we can't address. But most of them are in excess of 250MB and are in the "large files" category, so to remove them would be to double count, so I haven't.

The Results

Now we have some raw figures, let's go compress and deduplicate.

Taking our 13TB file system, we have

1. 13TB total, but only 60% or 7.8TB is addressable files <250MB
2. 90% of the 7.8TB is cold, giving 7.0TB to compress and deduplicate
3. With 50% compression, 7.0TB compresses to 3.5TB
4. 3.5TB file-level deduplicates by 13% to 3.1TB
5. There's a total saving of 7.0TB - 3.1TB, or 3.9TB
6. 13TB - 3.9TB is a 30% saving

A 30% Saving Based On Real Data

Here were my working figures and assumptions, and I was generous.

• For files <250MB
• 90% of your data is cold
• 13% of your data is duplicated
• All files compress by 50%

If we use the numbers EMC provide from their whitepaper (Achieving Storage Efficiency through EMC Celerra Data Deduplication, Jan 2009);

• For files <200MB
• 10% of your data is duplicated
• Files that compress do so between 40-50%

.

A Hornet Watching EMC

It's been launched, the party's over, the champagne bottles popped and drunk. The much-hyped V-Max is here.

As a professional EMC watcher, I have to congratulate EMC on keeping the launch under raps. It made a significant difference from Joe's usual chatty style on discussing futures (think Atmos, where the eventual official launch was a bit of a damp squib as the details were well known before lift off).

And StorageZilla was obviously in a straitjacket in a backroom of the EMC manufacturing plant in Cork, labeled up as due for release only after the event started.

The reaction to all this has been fairly muted, though. Val B had a blog up 24 hours before the bar opened, HDS have a couple of blogs on the subject -- one a whole list of questions which I thought pretty reasonable -- and 3Par did a rather odd guerilla piece. Some of the press had pieces; and no-longer-the-independent Steve Foskett summarised the whole thing for us. Lots of EMC!

All in all, given the scale of the hype, I thought the response fairly low key. Perhaps a day later we're all still a bit hungover from the party to work out -- is it any good?

That's why I haven't managed to digest the entirety of the announcement. There is rather a lot to take in, with a good deal of it marketing hooplah and broohah that needs to be sifted out before I can see clearly the direction EMC are headed. I can't speak for others, especially users, but that's where I am right now. Reading and trying to make connections.

And although Barry Burke breathlessly hyped blogged his little heart out on the subject of the technology, so far I'm having a hard time of connecting it back to reality and what you actually do with this stuff.

So I'm still digging through what EMC veep Dave Donatelli described at the launch as as the VMA and V-Max "twofer". (That's a word that's not well understood outside of the US, by the way. He may also not be aware that it also refers to attaching two light bulbs to a dimmer. Someone ought to tell him.)

Eventually, I suspect that although some to most of the V-Max and VMA details will turn out to be less impressive than at first sight, there will no doubt be some novel technologies that I can get my teeth into.

One thing I'm positive on; the reaction to the huge launch party wasn't big enough for EMC. Stirrer-in-Chief for EMC has his big stick out and is furiously trying to generate some reaction, mainly by poking a bunch of hornets who really weren't interested in stinging EMC for executing a fabby marketing launch.

Just give us a bit of time to clear our heads and dig through the mountains of discarded streamers and ribbons. When we've something to say, don't worry; we'll say it.

Here's one for starters; there are issues of when you can get a lot of this technology. There seem to be a number of futures, like FAST, that when subtracted out from the announcement, make for a much smaller ripple than EMC might claim in the pond that is the storage industry.

And am I surprised at this kind of pre-announcement by EMC? Not really, although EMC just hate it when we appear to do it to them.

Hypocrites! Bzzzzz... we're coming after you....

.