NetApp - The Missing Shade of Blue

Random Rocks And Benchmarks

“People understand contests. You take a bunch of kids throwing rocks at random and people look askance, but if you go and hold a rock-throwing contest -- people understand that.” (Don Murray)

And that, in a nutshell, is the origin of most competitive sports. What starts as "Hey, I can chuck this boulder through that window over there -- betcha you can't!" develops through time a set of rules, referees, measuring devices, a governing body (sometimes two or three) and the sport of rock-chucking may even get recognised by many, played by many and make it into the Olympics. Like spear throwing (the javelin) or blasting birds out of the sky with a shotgun (clay pigeon shooting), rock-chucking has matured into shot-putting.

Just like benchmarks. Originally used to demonstrate some feature of your system in all its one-trick glory (HP IOPS from cache is a classic example of how to do this), modern industry benchmarks make the attempt to allow some real-world comparison between competitors. The relationship to real life is debatable, but the rules by which they operate are carefully designed to inject some aspects of realism to comparative claims.

Benchmarks are intended to be a repeatable test of a set of skills. But, as with all competitive sports, sometimes there's the kid who starts his benchmarking career hanging round on street corners stoning passers-by for entertainment, or hoisting up the nearest large rock at hand and heaving it through the closest plate-glass window.

And the new kid is over at HP, using a non-benchmark as a benchmark, and generally lobbing rocks around in all directions. I'm not going to dissect the post in detail, because others have and will continue to do that. There's one little paragraph I want to focus on, because it demonstrates one of my pet peeves; benchmark intuition.

Now things were starting to make sense.   We were seeing the same sort of decay curve as shown in the IOMeter results posted in Making Sense of WAFL - Part 4.    Every time the test is run, the random component of the Jetstress database accesses fragment the LUN further and the throughput numbers get worse.  An array like EMC CX or HP EVA wont undergo this sort of decay curve since these arrays do not have internal WAFL-fragmentation problems like the FAS does.

The non-benchmark is Microsoft's ESRP, and the tester's intuitive assumption is that WAFL fragments; hence the tester's intuitive assumption that this is the source of the diminishing throughput numbers.

Bzzzt. Big fail.

Let's allow the first intuition; let's allow, for the sake of this demonstration, that "WAFL fragments your data". Here's a simple example to demonstrate why his intuition is wrong on fragmentation being the source of the problem. Exchange 2007 generates small random IOs (and that's the JetStress that HP are using in their test). The table below has 5 columns to demonstrate why small random IO works just as well (or badly, depending on your take) on randomly laid out data as sequentially laid out data.

• Random Placement: I've place 100 blocks randomly. The numbers have been generated from www.random.org. Slot 1 is block 67, slot 2 is block 19 and so on.
• Random Requested Block: this is meant to simulate IO requests from Exchange; again, drawn from a different run from www.random.org.
• Matching Block (Random Placement): this is where the requested block actually lives, So asking for block 80 requires a visit to slot 22, and so on.
• Seek Distance (Random Placement): this is the effective seek distance between requested random blocks. After we visit slot 22 (for block 80), we need to visit slot 58 (for block 37), requiring a seek of 36 slots.
• Seek Distance (Sequential Placement): this is the effective seek distance between requested sequential blocks. After we visit slot 80 (for block 80), we need to visit slot 37 (for block 37), requiring a seek of 43 slots.

(I'm having difficulty uploading the spreadsheet to TypePad, so when I get it fixed you'll be able to "Click to download the whole spreadsheet". Not yet though.)

Random Placement	Random Requested Block	Matching Block (Random Placement)	Seek Distance (Random Placement)	Seek Distance (Sequential Placement)
67	80	22	0	0
19	37	58	36	43
75	18	61	3	19
23	26	53	8	8
85	57	63	10	31
59	100	14	49	43
14	59	6	8	41
...	...	...	...	...
SUM			3269	3322

Hey, look at that! The sequentially laid out data takes more slot seeks than the randomly laid out data! Try it yourself, replace the 100 numbers in the first two columns with random numbers, and check the seek distance sum.

On average, they will be equal. In fact, if the requested blocks are random, it doesn't matter how the data is laid out. Intuition fail.

Here's the professional sport of benchmarking, which HP don't take part in (still being, as it were, at the rock-throwing stage);

• NetApp SPC Benchmarks (no HP here)
• SFS2008 NFS benchmarks (HP missing again)

And, of course, the official ESRP results (and, just as a reminder, these aren't benchmarks)

• Microsoft ESRP results

Having failed at shot-putting, perhaps HP might want to pick another sport for their talented testers. Like nude football.

[updated to correct some borked links and a typo].

I'm Sending Chuck a Bluey

My youngest daughter is going out with a professional soldier. That's tough for her; soldiers in the UK Armed Forces are often on active duty, and he's served several long tours of duty in Afghanistan already, something that she finds difficult to handle emotionally when he's way.

During his last tour, she wanted to send him a bluey. It's the Services equivalent of a telegram; you write your bluey online, it gets sent and printed locally, and most if not all blueys are delivered in 24 hours, even to servicemen right on the front line. It's good for morale, and the service is excellent.

Her first look at the website resulted in tears. When she looked at the information to get the letter to him, she discovered that all she knew was his name, his rank, and that he was in Afghanistan. No other information, and although she thought she knew his regiment, she wasn't sure.

Dad to the rescue. The boyfriend's name is unusual. Very unusual indeed, with a first name from Greek antiquity, and a surname that appears four times in the local telephone directory -- and they're all family members. This guy has a name you'd never forget; a name, like Roman Rock (not his real name, because he's still on active service). If there's another person in the UK with the same name, far less in the Army, I'll eat my shorts. So I calmed her down, and helped here address her bluey, to Roman Rock. That was all the information we could fill in.

It got to him in less than 12 hours. We didn't know where he was located, but the Army did, because he had a unique name.

Chuck Hollis had an interesting blog on filling out blueys. Well, sort of; on what he sees as the death of the filesystem. The topic struck me as interesting; The Future Doesn't Have A File System.  As usual, but with good reason, I disagree.

Object-based information stores are different than filesystems in several important ways. 

First, you use a token or other uniform identifier to get your information.  File systems imply location, tokens don't -- no such thing as a broken link or a moved file system.  Not to mention, tokens can uniquely identify gazillions of information objects.

Disagree; filenames don't imply location. They're just names; metadata that allows you to uniquely identify the data you wish to access. This is a problem that the internet created, and that the internet solved; see later.  

Second, they have the ability to associate all sorts of metadata with the object itself.  As the information object goes, its metadata travels with it.  A very useful property indeed.

And that differs how? You can associate metadata with a filename -- if you can do it for an "object", you can do it with a file. (Even old-fashioned filesystems keep things like last access time, size, and the name itself is often used to give further, human readable clarification, such as .html or .doc; although it may lie and we may care not to use it for such, that doesn't change to point of associating metadata with the name-as-a-handle.)

Third, the ability to hang metadata off the object gives us the ability to create all sorts of useful policies and services around the information without having to put everything in some sort of database or repository.

Double eh? If this (it's a GUID, or globally unique identifier);

c2f41010-65b3-11d1-a29f-00aa00c14882

doesn't require a repository or a database for its metadata, I'll eat my shorts. Again.

The internet solved this problem a good while ago. See RFC1737, RFC2141 and RFC3986. The essence of the RFCs; part is a name (like Roman Rock) and part is a location (like Afghanistan). The part that is the name doesn't tell you where the file is located; but the part that describes the location can be completely absent.

An example is in order. To demonstrate how far things have changed since Chuck banged away with Unix pipes and vi in the1900s, this returns a file;

http://blogs.netapp.com/shadeofblue/2009/08/poetry-corner.html

Interestingly, the file doesn't exist until you ask for it, because it's dynamically generated from parts. There's no directory shadeofblue, 2009, or 08, and no file poetry-corner.html either. And (here's a clue how far this goes) it doesn't live on a server at netapp.com either. It's all name and no location, and it works across the entire internet, not just inside a single object store.

Filesystems aren't the problem here; it's an attempt to make Atmos relevant. I think I'll send a bluey to Chuck and let him know; The Future Doesn't Have an Atmos.

 

---

I'm off to VMworld next week in San Fransisico. It's my first time at a VMworld conference, and I'm not quite sure what to expect.  But I do hope to meet lots of interesting people I've never met before while I'm out there. If you recognize me (yes, I look like the photograph), please introduce yourself. And no, I don't bite!

LeftHand Quorums and Split Brains

There's been a bit of a discussion going on between John Spiers (HP LeftHand) and myself over a number of issues I raised about a LeftHand SAN's storage efficiency. The comments moved on to talk about HA (high availability) and data reliability, and John raised a number of questions that I thought deserved a longer answer.

John Martin of NetApp (who also blogs on NetApp's Storage Efficiency blog) has kindly provided me with more detail, and rather than post this as a comment, I thought it worthy (again) of a blog in its own right. Thanks to John for the responses.

John Spiers' points are in blue (and I think I've accurately captured them, but they've been lightly [edited] for context). They're also slightly out of order.

In summary, I think there's a need for greater clarity on LeftHand best practices. Everyone, including the user quoted below, appears to be operating in the dark. As I said in a previous post, we've had to do a little"reading between the lines" and work from first principles. If any of this is wrong, please let me know, and I'll correct it.

[Update 09July: Many LeftHand manuals, including best practices, appeared in late June/July, but weren't there when I first looked in late May/early June. Much reading to do!]

---

NetApp can't deliver this level of HA with auto failover and failback [compared to a LeftHand SAN]

It depends on what you mean by "auto failback". If you mean failback initiated without permission or authorization from a responsible human being, then you'd be correct. If you mean an automated process with minimal user intervention, then you're wrong. From a NetApp perspective, and that of most storage and systems professionals I've talked to when discussing failback requirements, automated and uncontrolled failback is usually judged to be a bad idea.

NetApp also have an excellent product in addition to MetroCluster, called MultiStore. This provides similar kinds of functionality (automated failover and failback) over standard IP connections.

Can SnapMirror or MetroCluster [...] incrementally rebuild the primary site, while maintaining application state and data integrity – i.e. RPO=0 and 100% uptime? I didn’t think so.

Yes it can. As soon as the administrator believes it to be safe, the "failback" process is initiated, and an automatic incremental rebuild and resync provides seamless failback.

[quoting from a NetApp document] "Mirrored active/active configurations do not provide the capability to fail over to the partner node if one node is completely lost. For example, if power is lost to one entire node, including its storage, you cannot fail over to the other node. For this capability, use a MetroCluster"

[quotes sections about manual cluster failover and prevention of “split brain” and JS then says] LeftHand has distributed quorum management that eliminates all possibilities of a “split brain”. This allows at least one site to operate and then automatically resync the other sites when they come back online.

This is an interesting quote, which when taken out of its correct context (a practice commonly called "quote mining") makes it sound much worse than when it's put in context.

1. It should be noted that this is for a VERY unusual configuration, one that I’ve never seen go into production at any site. The comment is under the heading “Mirrored Active/Active Configurations”. With this, NetApp's Syncmirror is added to a standard active/active configuration to provide a second level of mirroring across the disks, but without a MetroCluster license for local (or stretch) functionality (suitable for distances of around 500m).
2. Should a customer decides that his data required the extra protection of Syncmirror (i.e. RAID 6+1), then we would recommend the extra resilience provided for in this configuration by using MetroCluster.

The reason for this behavior is to avoid a “split brain” scenario. There is no way of automatically detecting the difference between a total failure of a system, or the the failure of all forms of communication between them. It's this that causes "split brain"; two or more running systems thinking the other has failed, when only the communication between them has failed.

This applies to every conceivable cluster configuration. That includes quorum disks, proxy nodes and other cluster node failure detection techniques; they are all effectively forms of communication between nodes.

Distributed quorum management requires at least three nodes. It is not possible to have a quorum based on two nodes.

I’m not letting you brush this one under the rug. MetroCluster and SyncMirror don’t provide the same level of availability that is inherent in LeftHand’s base SAN/iQ software offering, and LeftHand requires no additional equipment.

That's interesting, because here's a LeftHand user experience;

"So at work we decided to go with a Lefthand Implementation of iSCSI, I am rather unhappy to find out that with only 2 units you have to run a virtual machine to complete the Quorum for redundancy. I am not happy to find out that in reality you need 3 appliances to complete a Quorum for management and to ensure that you have redundancy and that everything is available."

You should have been clearer about comparing a three or more node site vs. a two node site. This HA "no additional equipment solution" is growing legs.

And while I’ll give kudos for an n-way implementation, unlike the NetApp documentation you quote, you don’t address how you handle what happens when there is a complete loss of communication only from the primary site to all other sites. None of the other nodes are able to detect whether the site is a smoldering pile of rubble, or if it’s just incommunicado, and if you make the assumption that lack of communication is a trigger for failover, then you have a recipe for split brain.

The manual “declared disaster” approach is the safest way of dealing with this. If a customer believes they have a foolproof way of detecting a true site failure, then its trivially easy to integrate this by automating MetroCluster failover (or failback for that matter)

Yes, [HP's LeftHand] Network RAID 2 can sustain any random 5 disk failure. In fact, a 4 array system configured in Network RAID level 2 can sustain up to 2 complete array failures, and up to 6 disk failures in each of the remaining 2 arrays, and all at the same time.

Hmmm... Picture courtesy of HP's LeftHand P4000 brochure 4AA2-5247ENW, April 2009.

Let's start with nodes. I have two copies of my data for nRAID2, and four arrays. If any two nodes fail, then I lose data. Which two arrays can I blow away in this four node system, and not lose data in my logical volume? With this simplistic 4 block LUN, 1 and 3 or 2 and 4. Unless you can arrange your node failures in advance, I'd suggest that is one node protection.

And the statement that I can survive any random 5 disk failure is equally implausible given the LeftHand diagram above. Two disk failures in node 1 in the same RAID5 group would kill node 1. A further two disk failures in a RAID5 group kill node 2, and we now have two node failures which can't be survived. The answer for nRAID2 is 3 random disk failures, not 5.

I'm not even going to try and work out what "up to 6 disk failures in the remaining 2 arrays" really means, although I'll bet the words "random" and "quorum" don't feature heavily. 

Of course, this all assumes the HP LeftHand diagram above is correct, and not just a graphic designer's interpretation of the facts. The diagram below for the terrible usable capacity of a LeftHand system for 1 node failure protection (split brained or otherwise) is correct. I've had it double checked by HP.

LeftHand nRAID2 means 35% usable capacity

.

Venezuelan Beaver Cheese

The EMC blogosphere is full of "Why I Work For EMC!" posts at the moment, and to be honest, it's a concerted strategy to present EMC as the coolest and bestest employer in the tech industry. Ever.

I'm not in any doubt as to why, and if you've been following the press recently, I suspect you aren't in any doubt either. EMC bloggers are supporting the central thesis of Uncle Joe's open letter to Data Domain employees, that EMC is one of those rare things, a company that's coolest and best and biggest, the undisputed leader of companies, a world-class, #1 company.

With one voice, they're crying "I love it here! And so will you."  Which leads the inquisitive (actually, in my case, paranoid) amongst us to wonder if this is an orchestrated attempt to provide a bit of a public shove and push to the idea that corporate culture at EMC is just fine and dandy, and that there nothing to fear from being acquired. As an echo and reinforcement of Joe Tucci's message, the blogs I’ve seen definitely toe the line.

Whatever the agenda, what are they saying about why they love EMC so? There's a lot of high-falutin' sentiments on display; some personal observations about what it's like to work there, how to get promoted, where to wander round corridors talking to engineers about cool stuff, advice on what it takes to prosper, enjoy, fit in, etc etc.

The EMC they describe is like Venezuelan Beaver Cheese.

I love cheese. I've eaten a huge amount and variety in my time, from the humble Cheddar to the exotic ash covered Saint Maure. But if it was possible, the one I would really want to taste is the very rare Venezuelan Beaver Cheese. Some would have you believe that it's more than just a cheese; that it's the coolest and best and biggest cheese, the undisputed leader of cheeses, a world-class, #1 cheese.

Unfortunately, Venezuelan Beaver Cheese isn't just rare, it doesn't exist at all outside of a Monty Python sketch. Just like the picture of working at EMC we're getting from these blogs. I don't believe they're being honest with themselves.

EMC is probably more like a ripe Vieux Boulogne, a cheese that’s an acquired taste best washed down with strong drink. Co-incidentally, it has the unenviable reputation for being the world's smelliest.

---

Update 15June2009: In the "I wish I'd said that" category; Jay Kidd's blog post on EMC's bid for Data Domain.

Grit Your Teeth and Smile Politely

When folk tell me that the chance to travel with my work must be wonderful, I just grit my teeth and smile politely. Let me tell you, it's far from wonderful.

I'm newly returned from a bout of traveling, nearly 3 weeks and 3 locations all told. Spending entire weeks away from home in business hotels, with daylight hours in bat caves, evenings eating overpriced and unhealthy food, capped by 12,000 miles of weekend air travel in conditions that animal welfare organisations would condemn as unsuitable -- that isn't my idea of a company paid-for holiday. 

To cap it all, it was my birthday last weekend, on Sunday 31st May. I'd like to thank my colleagues that bought me a beer or two at the bar of the Santa Clara Hyatt. It helped me assuage the pain of being away from home. Much appreciated, even though my memories of the evening are a little sketchy.

I'm delighted to be back now, and was looking forward to my normal schedule of work. And would you believe it, I'm being asked (yet again) about HP's latest anti-NetApp nonsense doing the rounds.

Here's a screen shot of the latest dipstickery that passes for critical analysis at HP. (Yes, the slide says it's internal use, but it's publicly available on the web and it's being given to HP partners and prospective customers, probably with a "sshhh, don't tell anyone about this, but I've got this internal report, and it says...")

I'm not sure what NetApp report is being referred to here as there are several Cost of Ownership reports we've done over the years (search on Mercer or Wyman for example), and I can't find the report that matches with page numbers in the HP slide deck. The latest is here. Anyone at HP care to tell me if that's the right one?

Whichever HP used, the slide contains more than a few errors.

• We never recommend RAID-4, because RAID-DP at 14 data plus 2 parity is far more reliable. RAID-DP is more reliable than RAID-1, the most reliable and performant of the non-dual parity schemes. With HP you get to choose; reliability (RAID-6 for truly performance sapping protection) or RAID-1 for performance and protection at serious capacity cost; RAID-5 if you like your risks and performance about equally divided. With NetApp, you get superior performance, capacity and protection from RAID-DP.
• NetApp space reservation has been consistently and deliberately misunderstood by HP. And others.
• You choose snapshots or full clones on HP, and the choice is a tough one. HP EVA snapshots perform badly and consume little space; clones perform well, but consume lots of space. NetApp snapshots and clones don't have that problem.
• There is no cost for an NDMP licence on a NetApp system. And there never has been. Ever.

Conclusion; If HP only understood NetApp's technology...

When your HP sales rep slides this across your desk and suggests that HP's products are wonderful, think business travel. Then just grit your teeth and smile politely.

.

Please Squeeze Me (Part 2)

In this multi-part post, I'm exploring specific space-saving technologies; deduplication and compression, and EMC's recent entry into the market with the Celerra Data Deduplication product.

The first part covered some of the basic technology of deduplication and compression. This time; what the implications of using each of the technologies -- either separately or together -- means for the user of storage systems that feature it.

I think a terminology list is useful here.

Fixed block deduplication	Deduplication of matching blocks level (NetApp uses a 4K block)
File level deduplication	Deduplication of matching files
Compression	Encoding data into a form where it's smaller than the original; all storage systems use lossless compression techniques.
Inflation or Decompression	Decoding data back into its original form.
Hydration	Any technique that reduces the size of data; includes compression and deduplication
Rehydration	Any technique that restores data to its original form; the revers of hydration
Reduplication	The repetition of a sound, as in claptrap, hokey-pokey, razzle-dazzle, teenie-weenie. (See footnote)

EMC's entry into the storage deduplication market has been quite late; only this month did the Celerra get some form of deduplication.

Celerra Data Compression And A Bit Of Deduplication

EMC's solution based on two technologies; compression and file level deduplication.

This works on NAS (CIFS and NFS) only; no block level iSCSI support, and it doesn't apply to the CLARiiON backend at all, so no deduplication there. And here's the target EMC have to hit with this technology. Note the lack of VMware, backups and other data store types,

Volume type	NetApp Typical Deduplication	NetApp Typical Range
Home Directories	35%	20-50%
CIFS Shares	35%	20-50%

First, the file is compressed. There's a lower limit on compression, which is reasonable; no point in having overhead where the savngs are minor. There's an upper limit too, of 200MB, which caught me by surprise. Compression is claimed to return 40-50% space savings, which is fairly impressive given that the Celerra uses no form of hardware compression; it's run as a background task, and uses spare cycles on the CPU.

Once compressed (or if too small/too big and left in its fully inflated state) the file is deduplicated. A key is generated (shades of EMC's Centera here, although it's a more robust SHA-1)  and compared to existing file keys. If it's a duplicate, the space is released. If it's not, the file is moved to a hidden area (which makes it secure from accidental reading or writing I presume).

At first blush, it looks like the majority of the space savings are though compression (40-50%), followed by a 10% reduction due to duplicate files being eliminated. Take 3 duplicate 1MB files, compress and deduplicate; voila, 500KB of storage required.

The Downsides

Firstly, there are the downsides of compression and decompression on hot data. remember, data must either be inflated in memory if you need to read it (think backup) or inflated on disk (if you need to write to it.

If we want to read one of our 1MB files, it needs to be inflated in memory. There's a small but definite overhead here, although it may be offset by reading less raw data.

If we need to write to it, the 1MB file needs inflated on disk. That means a 500K read, in memory inflation and a 1MB write. The upper limit of 200MB is there for a reason; CIFS times out if IO isn't satisfied pretty quickly. Form the EMC whitepaper;

• Performance testing has shown that Celerra can reduplicate a 200MB file in less than [the CIFS timeout value of] 25 seconds even when under heavy load.

That first write could take a very long time indeed.

The Mitigating Factors

It's not all doom and gloom. EMC again.

• Most applications do not modify files. They typically make a local copy, modify it and [...] write the entire new file [...] discarding the old copy. Therefore, the file is never reduplicated on the file server; it is just replaced.

Actually, this one is just based on old data, or perhaps it's just a best guess. NetApp's public research into CIFS data access patterns shows that this is not true for corporate type data; read/writes make up over 45% of access over CIFS.

• Celerra avoids processing active files (accessed or modified recently) [...] Deduplicated files are less likely to be modified and, if they are, performance is less likely to be a critical factor.

So EMC recommend that you don't address hot data. Given that most file systems have lots of cold data, and that cold data doesn't get read much or written much (that's why it's cold!), problem solved. That one has some substance; the same research shows around 10% of the data was accessed in a nearly 3 month period..

Target met! Or is it?

Large Files Dominate

EMC again.

• By default, Celerra avoids deduplicating files larger than 200MB.

200 1MB files takes up as much space as one 200MB file. Obvious. But with the Celerra compression limits you can only compress one-half the total of 400MB. Suddenly, size is important. It's only 1 file, but it's a significant 50% of the data. How much data is like that, I wonder.

I can't tell for certain. But here's a clue; my laptop has only 18 files bigger than 200MB, including things like CAB files for Windows Office. But that's a staggering 6GB out of 22GB data; more than 25%. (No large VMware images in this set either, because it's a VM I'm running on).

And the deduplication doesn't work on files smaller than 24KB; too much overhead. How many of those I wonder... 35580 of them less than 24k. I need a big assumption here'; let's say the average is 12K. That's approximately 400MB, so +/- a few 100s won't make a big difference; the assumption is a safe one.

The Sums

Let's add the effects together, using my laptop data as an example.

• 22GB - 6.4GB of files that won't deduplicate = 15.6GB
• 15.6GB where 90% are unused for an extended period = 14.1GB

I can address 14.1GB with this product. The other 7.9GB is untouchable.

• 14.1GB compressed by 50% = 7.1GB
• 7.1GB deduplicated by 10% = 6.4GB

That's 7.9GB+6.4GB = 14.3GB, or a total reduction of 35%.

Whew. Just made it! 35% is achievable in theory.

If you can get 50% compression. Remember, Microsoft's NTFS already does data compression. And on the directories where I have it turned on, I get something like 15% compression, not 50%.

And if 10% of your compressible files are duplicates. And at some cost.

Worth It?

No, because of the kind of impacts this compression and deduplication brings that NetApp's block deduplication just doesn't suffer from. Here's a list of things to watch out for;

• EMC dedupe is for Celerra only
• EMC dedupe does not support VMware (VMDKs)
• EMC dedupe does not support LUNs (iSCSI or FC SAN)
• EMC dedupe is only done at a file level.
• EMC dedupe only runs on one datamover at a time
• EMC dedupe is targeted at only infrequently accessed (inactive) files
• EMC dedupe only works on file sizes less than 200MB
• EMC compression only works on file sizes greater than 24K
• EMC provides no indication of savings to be made.
• EMC gives no estimate on the overall compression and deduplication you should expect
• EMC recognizes that there's a performance impact in reading deduped and compressed files.
• EMC dedupe requires EMC dedupe compatible backup tools (of which there is NetWorker and Veritas NetBackuo, for a total of 2) unless you want to inflate all the data
• EMC dedupe restricts restores to full volumes only from backups
• EMC dedupe means file level restores have to use SavVols and CoFR (copy-on-first-write) snapshots that impact performance

Looks all the world to me like EMC are squeezing you, not your data.

Now, compare that with NetApp's solution and the savings you can make. No contest.

 

---

Reduplication. An aside; what's with this word? It may be EMC's attempt to avoid some truly horrible alternatives. Undeduplicate sounds like two pairs of underwear, inflation is bad economics, decompression is the noise air makes coming out of a diving suit, and choosing those words is understandably Not A Good Thing. Although perhaps a more settled word like rehydration might have been better and more widely understood.

Reduplication as is normally understood means the repetition of a sound, as in claptrap, hokey-pokey, razzle-dazzle, teenie-weenie. Just like EMC's deduplication, methinks.

.

Please Squeeze Me (Part 1)

In this multi-part post, I'm exploring specific space-saving technologies; deduplication and compression, and EMC's recent entry into the market with the Celerra Data Deduplication product.

There are 3 commonly used deduplication technologies, and compression, and now the the storage industry is beginning to take advantage of them.

The end goal; storing more for less.

The three varieties of deduplication are fixed block, variable and file level. They're all remarkably similar in terms of what they do; they single-instance some level of data. Variable block I see as a variety of deduplication akin to file level, so I'm going to hand-wave it away right now and focus on the file level and fixed block varieties.

First, let's clear up one point. Are compression and deduplication synonymous? Here's what one company thinks; it's from an EMC whitepaper (more of which in part 2, from Achieving Storage Efficiency through EMC Celerra Data Deduplication, Jan 2009).

• Compression is often considered to be different from deduplication. However, compression can be described as infinitely variable, bit-level, intra-object deduplication. 

To be honest, I think that description is so wrong it's not even wrong. It might even have Shannon spinning in his grave.

• Deduplication is a form of delta encoding. In other words, only the differences between objects is stored; in the NetApp case, only different and unique blocks in the same volume.
• Compression is the process of encoding information using fewer bits than the original data would use. There's no delta and no deduplication.

I know why EMC are saying this. There are major differences between compression and deduplication in cost and outcome, particularly with respect to performance and capacity, but the industry buzz around the deduplication word is so attractive, it makes good marketing sense.

Yes, I know that EMC have a product called called Celerra Data Deduplication, and that it deduplicates data. But it should really be called Celerra Data Compression And A Bit Of Deduplication, because compression is where the savings come from.

And the downsides, of which more in part 2.

File Level Deduplication

This works only on objects like files; the major difference between it and fixed block deduplication is the size of the object. First, it's variable length, and second, it's an all or nothing test for duplicates.It's often describes as SIS -- Single Instance Storage.

If two files are identical, it's possible to store only one and have two directory entries point at the same file. UNIX has been supporting this for years with symbolic links; the links are aliases of the data that makes up the file. But that's an expensive active management process for the savings involved, and file level deduplication brings the advantages of automation and accuracy.

At a minimum, there's a metadata (data describing data) overhead of counting the number of times a file is pointed to, and some hash of its contents so that files can be matched.

And file level deduplication depends on lots of duplicate files; files have to match for length and content exactly. One further limitation is that files must normally reside in the same file system. That can limit the benefit, and ratios will depend on how file systems are arranged and their contents.

Writing to files stored like this also brings a downside; the file must be copied. That's twice the space, and there's the overhead of the IO to read the original and write the copy too.

How much file-level deduplication can be achieved? That depends, and I've not seen much research around this issue. Anecdotally, 10 to 20% seems reasonable. A reminder; if it's higher than that, perhaps you need to do some housekeeping and just delete all those duplicates...

Fixed Block Deduplication

NetApp continues to be the only vendor with deduplication on primary storage. It's a simple technology in principle; for every block written to a volume, check if there's been an identical block written before. If the answer is yes, then release the block to the free block pool, and point at the duplicate block.

This is based on deduplicating 4K blocks; and it works on SAN block based and NAS file-based data too. And because NetApp systems use pointers to blocks, reading and writing data in deduplicated volumes is simple. Read a block; get the block pointed to. Update a block or write a new block; write it, point to it, and the system will deduplicate at leisure later.

There's metadata associated with this technique; we need to save a unique  fingerprint for each block.

It's all covered in this NetApp Technical Report, so I shan't repeat much more. But it's worth drawing out one of the more interesting tables that indicate the levels of deduplication achievable;

Data Types	Typical Space Savings	Range
Backup Data	90%	85-95%
VMware VMs	70%	50-90%
Geoseismic	55%	40-70%
Database Backups	55%	40-70%
Home Directories	35%	20-50%
CIFS Shares	35%	20-50%
E-mail PSTs	30%	20-40%
Mixed Enterprise Data	30%	20-40%
Document Archives	25%	20-30%
Engineering Archives	25%	20-30%

For many classes of data, this is a considerable saving.

Compression

One thing that NetApp doesn't do is compression on its primary storage system (FAS and V Series). It's a feature of our VTL backup device, for reasons that will become apparent.

Compression is a well understood technology with a wide range of applications. Good compression depends on the type of data being compressed; the more random the data, the less likely it is to compress. For the kind of application data you might find on the average NAS box, as much as 50% should be achievable. That is a saving worth making.

There are downsides to compression. There's the not inconsiderable CPU load of compressing and inflating (decompressing) the data, although some systems (like NetApp's VTL) use specialised hardware to offload the task and avoid the processing impact.

More importantly; there are two things you can't do easily once the data is compressed. Reading and writing.

Reading means metadata; you can't read compressed data it without inflating it, and that normally means reading the whole object that was compressed, decompressing it and saving it in its original inflated form.

To get round this problem, systems such as NetApp's VTL retain a certain amount of metadata, so that the system can find blocks of compressed data, and just inflate these blocks.

An example; if I want to read bytes 10000 through 20000 of a compressed file, I look up the meta-data to get the blocks that contain this range, read just them, and inflate in memory before passing back to the requestor. There's no need to reinflate the data on disk at all; it can stay in its compressed format.

Writing means Inflation. Compressing data produces variable rates of savings, so it's not possible to simply update a section of a file by compressing it and getting it to exactly fit in the space available. Writing means inflation; again, the compression hardware can do this and avoid the CPU impact, but there's no avoiding the fact that the data is now occupying more space.

And there's additional IO to account for; the compressed file needs to be read, and the inflated file written back to disk, which can cause considerable delays on doing that first write.

(Worth noting that, as NetApp VTLs are archive and backup devices, they don't need to do updates like this; so the compression plus metadata approach works well.)

Compression levels of up to 50% are achievable for some, but not all, types of data. The data we store tends to be non-random in nature. Text files compress very well, as do certain Microsoft file formats such as PowerPoint. The notable exceptions are already compressed file formats like MPEGs and ZIPs for example.

And if you're wondering why you can't get any benefit by compressing compressed data, the pigeon-hole principle explains why..

---

More in part 2.

.

EMC and the Fake Rolex

Much of the mail that I receive on my computer at home consists of spam. Badly written, most of it is offering me enough Viagra to give me a serious balance problem, chances to play roulette, easy money from the sons of deposed African cabinet ministers who have amassed stolen fortunes that they want to launder through my bank account....

It's a serious irritation, but occasionally it's amusing. An example; this heavily capitalized email from the fabulously named Lagina Cowper.

From: Lagina Cowper
Subject: FAKE ROLEX!!!!

Look a MILLION DOLLARS with our ROLEX WATCHES!!! LEADING CHEAPER than the real thing, we have DAYTONA CSOMOGRAPH, ROYAL BLACK, OYSTER,...

NetApp has some fairly sophisticated email filters that weed out most spam. So you can imagine my surprise when EMC's Chuck Hollis blogged today, and my RSS feed popped this into my work inbox. It's about Frankenstorage. Chuck says;

...NetApp did their interpretation of frankenstorage when they announced a lash-up with TMS's RamSan-500 to tick the box that, yes, they had a flash story to tell. 

Chuck is calling NetApp systems Frankenstorage?

That's funny! Having tried to hijack the unified storage message, EMC's very own Lagina Cowper is suggesting that other competitiors' storage are just bolted together mishmashes of parts.

Where, I wonder, did he get the idea of Frankenstorage from? There's no stopping a good idea, so perhaps it was from my blog. I noted in November last year that the Celerra NS range was due out shortly.

It Lives! EMC’s FrankenNAS.

Unsurprising will be the throwing of the big switch to zap into life a new EMC NS range based on the Celerra and CX4. I’ve lost count of the exact number of moving parts in the average EMC NAS solution, but any press release will probably skim over that and claim “market leading unified storage” with a cuddly personality. I’d suggest counting the bolts and keeping a safe distance.

I also wonder if Avamar based NAS deduplication will get announced too? I’ll be fascinated to see how they wire that one up in FrankenNAS’s two brains. And how much it costs.

I suppose it's a common enough analogy that Chuck could have thought of it independently, so let's put the copying of the Frankenstein imagery aside. Here's the serious point.

By lagging behind the competition, EMC suffers from the marketers' dilemma; how do we make yesterday's products appeal to today's customer? The answer; copy. Copy everything. Ape your competitors' stance, and employ serious marketing muscle to make it look like you invented the idea in the first place. 

If you can't make it, fake it.

But here's an example of EMC copying that really gets to the heart of the issue. Unified storage. Chuck keeps saying that the Celerra is unified storage, even though it's a bolted together set of hardware, and so fails miserably by his own criteria. But Chuck keeps missing the point, and will miss it forever because he can't seem to grasp one of the fundamentals of this industry.

It's not about the hardware. It's all in the software.

It's the software that makes NetApp's solution unified storage, and exposes the Celerra for the bolted together FrankenNAS is really is. Let me repeat; software.

Question. How many operating systems, types of snapshots and clones, pieces of incompatible replication, varieties of deduplication software does the average EMC customer have to buy and manage? Lots, and that's a problem. EMC's answer is to fake a solution. They call it "unified storage", but it's plainly and painfully obvious it isn't.

So you can understand why I laughed out loud when I read this from Chuck's blog;

How do we stop these monstrosities?  That's up to customers, isn't it? 

Good question, Chuck. I for one will be making sure they understand the real issue around Frankenstorage.

Your marketing spin on this is just the same as the cheap watch spam from Lagina Cowper. Except Lagina is more honest; that Rolex really is a fake.

.

Because We Have To...

I spend too much time worrying. Far too much. I get concerned that our competitors are going to find out our Big Secret. A colleague has asked me to respond to a competitive analysis of NetApp's deduplication, and I can't get past this paragraph. The stress is killing me.

"So, we ask again, why is NetApp the only vendor to provide dedupe on the primary volume. Answer: because they have to."

NetApp provide leading edge technology solutions, a list so long I won't post it here; for that, our corporate website does a great job. But I still worry.

NetApp has tens of thousands of customers using these technologies to their considerable advantage, including deduplication. That's from measured data we collect from customer systems (using AutoSupport). And yet I still worry.

NetApp has a Virtualization Space Guarantee program, now extended to include other virtualization products including Microsoft's Hyper-V and Citrix' XENServer.

Ulcers. The fear is gnawing away at me. The Big Secret, our dirty laundry. I can't take the stress of this, so it's time to confess.

...it [NetApp] tries desperately to hide but at the same time seems driven to confess as if subconsciously trying to purge some unresolved guilt. 

This isn't my usual shrink speaking. This is an HP competitive storage analyst I'm quoting, but he's good, no? He's so right, that's where the stress is coming from; hiding the Big Secret.

It's time to come clean. I confess. We only provide deduplication on primary volumes because we have to.

Confession made, the guilt is gone. What a relief! I feel better already.

.

MAD Blog: The E in EMC Stands For Emulated

Yes, it does, I tell you. EMC is not what you think it is; it's emulated! The real EMC wants to hide this from you!

Only kidding. With EMC, what you see is what you get. Unfortunately.

Point in case; here's EMC's uber blogger, Chuck Hollis, on the subject of EMC's FC solutions over NetApp's, specifically about the NX4;

Like other Celerras, it does the full unified storage thing: iSCSI, NAS and "real deal" FC that isn't emulated.

The FC is native, and is not mediated by the NAS head.

In his comments, Tony Asaro can't quite fathom this out. He wasn't alone. Chuck bounces back, and over a series of comments (my bold italics); 

But you lose the thread there -- most FC care about predictable performance behavior, and emulation can often get in the way of that, as you lose the underlying geometry and device characteristics.

Regarding disk geometry, NetApp (as do all emulated environments) attempts to emulate physical disks using a file system.

The real debate is --- when you need FC and what it does well, are you better off with an emulated version, or a native implementation?

My point is that -- at the application level -- emulated FC drives behave differently than real ones. And, in many cases, it can matter.

There's a mishmash of ideas here. FC stands for everything; drives, protocols, storage heads, even disk geometry is relevant. But the L for Logical in LUN seems to have somehow passed by without a mention. Clever, clever NX4!

Sigh. Still, we did get one admission.

I have no basis to comment on NetApp's specific implementation of the FC protocol. Unless I talk to one of our FC engineers, I just wouldn't know. Or know if any deficiencies or advantages really mattered.

That would have been a good move before blogging on the subject, no?.

So go and ask an engineer to explain what "emulated FC" is. Then come back and let me know. Personally, I'm betting on "Uhmmm...", and a puzzled look.

---

I'm grateful to Kostadis for the MAD blog idea. if you're tired of reading posts where vendors try and poke each other in the eye, just skip the ones marked MAD.

.