NetApp - The Missing Shade of Blue

I'm Sending Chuck a Bluey

My youngest daughter is going out with a professional soldier. That's tough for her; soldiers in the UK Armed Forces are often on active duty, and he's served several long tours of duty in Afghanistan already, something that she finds difficult to handle emotionally when he's way.

During his last tour, she wanted to send him a bluey. It's the Services equivalent of a telegram; you write your bluey online, it gets sent and printed locally, and most if not all blueys are delivered in 24 hours, even to servicemen right on the front line. It's good for morale, and the service is excellent.

Her first look at the website resulted in tears. When she looked at the information to get the letter to him, she discovered that all she knew was his name, his rank, and that he was in Afghanistan. No other information, and although she thought she knew his regiment, she wasn't sure.

Dad to the rescue. The boyfriend's name is unusual. Very unusual indeed, with a first name from Greek antiquity, and a surname that appears four times in the local telephone directory -- and they're all family members. This guy has a name you'd never forget; a name, like Roman Rock (not his real name, because he's still on active service). If there's another person in the UK with the same name, far less in the Army, I'll eat my shorts. So I calmed her down, and helped here address her bluey, to Roman Rock. That was all the information we could fill in.

It got to him in less than 12 hours. We didn't know where he was located, but the Army did, because he had a unique name.

Chuck Hollis had an interesting blog on filling out blueys. Well, sort of; on what he sees as the death of the filesystem. The topic struck me as interesting; The Future Doesn't Have A File System.  As usual, but with good reason, I disagree.

Object-based information stores are different than filesystems in several important ways. 

First, you use a token or other uniform identifier to get your information.  File systems imply location, tokens don't -- no such thing as a broken link or a moved file system.  Not to mention, tokens can uniquely identify gazillions of information objects.

Disagree; filenames don't imply location. They're just names; metadata that allows you to uniquely identify the data you wish to access. This is a problem that the internet created, and that the internet solved; see later.  

Second, they have the ability to associate all sorts of metadata with the object itself.  As the information object goes, its metadata travels with it.  A very useful property indeed.

And that differs how? You can associate metadata with a filename -- if you can do it for an "object", you can do it with a file. (Even old-fashioned filesystems keep things like last access time, size, and the name itself is often used to give further, human readable clarification, such as .html or .doc; although it may lie and we may care not to use it for such, that doesn't change to point of associating metadata with the name-as-a-handle.)

Third, the ability to hang metadata off the object gives us the ability to create all sorts of useful policies and services around the information without having to put everything in some sort of database or repository.

Double eh? If this (it's a GUID, or globally unique identifier);

c2f41010-65b3-11d1-a29f-00aa00c14882

doesn't require a repository or a database for its metadata, I'll eat my shorts. Again.

The internet solved this problem a good while ago. See RFC1737, RFC2141 and RFC3986. The essence of the RFCs; part is a name (like Roman Rock) and part is a location (like Afghanistan). The part that is the name doesn't tell you where the file is located; but the part that describes the location can be completely absent.

An example is in order. To demonstrate how far things have changed since Chuck banged away with Unix pipes and vi in the1900s, this returns a file;

http://blogs.netapp.com/shadeofblue/2009/08/poetry-corner.html

Interestingly, the file doesn't exist until you ask for it, because it's dynamically generated from parts. There's no directory shadeofblue, 2009, or 08, and no file poetry-corner.html either. And (here's a clue how far this goes) it doesn't live on a server at netapp.com either. It's all name and no location, and it works across the entire internet, not just inside a single object store.

Filesystems aren't the problem here; it's an attempt to make Atmos relevant. I think I'll send a bluey to Chuck and let him know; The Future Doesn't Have an Atmos.

 

---

I'm off to VMworld next week in San Fransisico. It's my first time at a VMworld conference, and I'm not quite sure what to expect.  But I do hope to meet lots of interesting people I've never met before while I'm out there. If you recognize me (yes, I look like the photograph), please introduce yourself. And no, I don't bite!

Free Cheese from EMC

EMC's Barry Burke seems mighty pleased to announce that Virtual Provisioning (what everyone else calls thin provisioning) is now "free" (at least on the V-Max and DMX4 storage arrays but not on the CLARiiON).

I'm having a remarkably similar argument with HP folks on their assertion that the software on their LeftHand storage arrays is "free", even though there's a big difference in the price of similarly specified servers and the P4000 based on the same tin. The same with Sun; their "free" software on the the Sun 7210 array costs a lot. The system is based on a Sun X4540, which is much cheaper.

And Barry's challenge "We'll see if others follow suit" line is just so, well, yesterday's news. There are any number of storage vendors that have thin provisioning at no additional cost, including NetApp FAS systems; since 2004 to be precise.

What Barry means by free is at no additional cost, and it's an important difference. And I don't  think buyers are fooled by this use of the word. It's the total cost that you need to look at, including the free. He eventually gets there; right in the last-but-one paragraph where it's "all at no extra charge".

Free? There's always free cheese in mouse traps, but the mice there aren't best pleased.

.

Venezuelan Beaver Cheese

The EMC blogosphere is full of "Why I Work For EMC!" posts at the moment, and to be honest, it's a concerted strategy to present EMC as the coolest and bestest employer in the tech industry. Ever.

I'm not in any doubt as to why, and if you've been following the press recently, I suspect you aren't in any doubt either. EMC bloggers are supporting the central thesis of Uncle Joe's open letter to Data Domain employees, that EMC is one of those rare things, a company that's coolest and best and biggest, the undisputed leader of companies, a world-class, #1 company.

With one voice, they're crying "I love it here! And so will you."  Which leads the inquisitive (actually, in my case, paranoid) amongst us to wonder if this is an orchestrated attempt to provide a bit of a public shove and push to the idea that corporate culture at EMC is just fine and dandy, and that there nothing to fear from being acquired. As an echo and reinforcement of Joe Tucci's message, the blogs I’ve seen definitely toe the line.

Whatever the agenda, what are they saying about why they love EMC so? There's a lot of high-falutin' sentiments on display; some personal observations about what it's like to work there, how to get promoted, where to wander round corridors talking to engineers about cool stuff, advice on what it takes to prosper, enjoy, fit in, etc etc.

The EMC they describe is like Venezuelan Beaver Cheese.

I love cheese. I've eaten a huge amount and variety in my time, from the humble Cheddar to the exotic ash covered Saint Maure. But if it was possible, the one I would really want to taste is the very rare Venezuelan Beaver Cheese. Some would have you believe that it's more than just a cheese; that it's the coolest and best and biggest cheese, the undisputed leader of cheeses, a world-class, #1 cheese.

Unfortunately, Venezuelan Beaver Cheese isn't just rare, it doesn't exist at all outside of a Monty Python sketch. Just like the picture of working at EMC we're getting from these blogs. I don't believe they're being honest with themselves.

EMC is probably more like a ripe Vieux Boulogne, a cheese that’s an acquired taste best washed down with strong drink. Co-incidentally, it has the unenviable reputation for being the world's smelliest.

---

Update 15June2009: In the "I wish I'd said that" category; Jay Kidd's blog post on EMC's bid for Data Domain.

Pay More Tax and Retire at 70

The last time I felt overwhelmed by big numbers was when UK's Chancellor of the Exchequer Alastair Darling made his last budget announcement, and he declared that there was a borrowing requirement of some 175 billion for the public sector economy. That's an eye-wateringly large number, much, much larger than any previous government's borrowing requirement.

As Senator Everett Dirkson once noted, "A billion here, a billion there, pretty soon it adds up to real money." We're in real money territory now.

And we (at least us poor unfortunates resident in the UK) will be paying for this debt for years; some estimate until 2020 or beyond. Others reckon the debt is so large, it needs higher taxes now and we'll not be retiring until we're much older.

If you live in the UK, Darling's budget means you've no choice. Pay more tax and plan on retiring at 70. Either that, or it's emigrate somewhere a little less financially punishing.

EMC have exactly the same attitude to budgeting as the UK government.

In his latest blog on the Symmetrix V-Max, Chuck Hollis is in big numbers mood, Dirkson- and Darling-sized big numbers. Go take a look and report back here on how you feel. I was surprised -- nay, shocked, I tell you -- at the bazillions of zeroes required to tell us what EMC did in developing the V-Max.

Having your eyeballs seared by all these digits makes it difficult to get at the detail, but one number did catch my eye, because Chuck beats on it. The amount of money spent on developing the V-Max.

A cool $200 million in staff costs alone, never mind the 1.5 acres of air conditioned machine room, or the 135,000 disks, or the total electrical output of a small country  required to bring this project to fruition. I suspect the real costs were nearer $400 or even $500 million if that's the staffing figure alone.

Gadzooks, but that's a lot of money. So who pays? 

The sales of the Symmetrix DMX have been on a steep downward trajectory for some time. That's because consumers aren't buying it in the same volumes because they don't have the need (their business has shrunk) or the money isn't there (they used to be a bank). The lack of growth is generally true of the entire IT industry, but the "big iron" has taken more than it's fair share of the beating. And the V-Max is big iron -- just look at all those numbers!

So my bet is on anyone that buys anything from EMC -- CLARiiON, Celerra, Centera, DMX, or even V-Max, Documentum, RSA, maintenance, support etc -- will pay the premium of getting EMC some of their money back, given EMC's less than stellar financial performance recently. Because the C in EMC doesn't stand for Charity, that's for sure. And the E doesn't look like it stands for Economical.

If you bought into EMC, the V-Max budget means you've no choice. Pay more tax and plan on retiring at 70. Either that, or you might want to consider migrating to something a little less financially punishing.

.

Please Squeeze Me (Part 3)

In two previous blog entries I explored EMC's late entry into the deduplication market with the EMC Celerra Data Deduplication product.

I used the contents of my laptop to demonstrate that getting to 35% savings was a bit of a squeeze using EMC's technique for deduplicating.Perhaps it wasn't a good representative sample of the kind of data you might typically see on home directories, and that it might be better with real life user data.

Nothing like real life data for making a point.

So here are some real numbers from a real user. I've used anonymous customer data before, in a usable space analysis. Using some advanced technology (in this case, NetApp's Information Server 1200) I can make a much more accurate assessment of the capabilities of EMC's latest but not so greatest. In fact, for customers looking to see how much EMC deduplication can't save them, I can strongly recommend its analytic capabilities.

The conclusion? 30% deduplication on the Celerra, with some generous assumptions. See below the line for the gory details.

The real figures will be much worse, and look out for all these other caveats.

EMC dedupe:

• is for Celerra only
• disables MPFS 
• does not support VMware (VMDKs)
• does not support LUNs (iSCSI or FC SAN)
• is only done at a file level.
• is targeted at only infrequently accessed (inactive) files
• only works on file sizes less than 200MB
• compression only works on file sizes greater than 24K
• has a performance impact in reading deduped and compressed files.
• restricts backups and restores to full volumes 
• means file level restores have to use SavVols and CoFR (copy-on-first-write) snapshots that impact performance

What will it look like on your data? 20% space savings? Less? The EMC calculator doesn't work on actual data, just %age estimates by file type. Give it try.

NetApp provides two calculators. A deduplication calculator, like the EMC tool (except that NetApp's deduplication works across all primary data, not just NAS files), and SSET, a space saving estimation tool that uses actual data on your system. Actual results are within +/-5% of the space savings that SSET predicts.

I still think it looks like all the world to me like EMC are squeezing its customers, not your data. And, missing in action; V-Max deduplication?

---

The Details

EMC's "Deduplication"

Let's recap. For home directories, EMC's deduplication uses the following rules for "cold" or not frequently accessed files that pass filtering;

• If the file is less than 24K, don't compress, only deduplicate
• If the file is >200MB, don't compress or deduplicate; leave it alone
• Otherwise, compress and deduplicate

For "cold", I'm using a figure of 90% of the data; and I'm also assuming that all files between 0K and 250MB are compressible by 50%. That will make the EMC numbers much better than are achievable in practice. In reality, YMMV. You might see a smaller or larger %age of files as cold, and for certain, not all files will compress by 50%; many will not compress at all.

But as you'll see, my original 35% was generous.

Large Files Still Dominate

As I noted before, large files may be few in number, but the represent the bulk of the data. Here's a sample from a user system. This file system contains 20 million files in just under 13TB; it's CIFS home directory data. (Click to see full sized).

Small files make up the bulk of the raw file count, and the large files in excess of 250MB are very few in number (less than 0.03% of the total file count).

But those large files over 250MB make up over 41% of the used space.

That leaves 59% of the data for EMC to deduplicate (well, compress first and deduplicate second).

Duplicates

Duplicate files are interesting. There are very few duplicates for files in excess of 50MB, and the savings from deduplicating those show a 1% decrease. So not deduplicating them has very little effect.

Files smaller than 50MB show an average of 13% duplicate space savings, although over 30% of files are duplicate by count. Here, it's medium sized files; most of them make up the duplicates. Small sized files are rarely duplicates. I'll err on the side of safety and use 10%.

At least EMC are deduplicating the right files!

Active Files

In terms of active files, 35% of the space was made up of active PSTs. EMC's deduplication doesn't do active files, and again, that's a big chunk of data we can't address. But most of them are in excess of 250MB and are in the "large files" category, so to remove them would be to double count, so I haven't.

The Results

Now we have some raw figures, let's go compress and deduplicate.

Taking our 13TB file system, we have

1. 13TB total, but only 60% or 7.8TB is addressable files <250MB
2. 90% of the 7.8TB is cold, giving 7.0TB to compress and deduplicate
3. With 50% compression, 7.0TB compresses to 3.5TB
4. 3.5TB file-level deduplicates by 13% to 3.1TB
5. There's a total saving of 7.0TB - 3.1TB, or 3.9TB
6. 13TB - 3.9TB is a 30% saving

A 30% Saving Based On Real Data

Here were my working figures and assumptions, and I was generous.

• For files <250MB
• 90% of your data is cold
• 13% of your data is duplicated
• All files compress by 50%

If we use the numbers EMC provide from their whitepaper (Achieving Storage Efficiency through EMC Celerra Data Deduplication, Jan 2009);

• For files <200MB
• 10% of your data is duplicated
• Files that compress do so between 40-50%

.

A Hornet Watching EMC

It's been launched, the party's over, the champagne bottles popped and drunk. The much-hyped V-Max is here.

As a professional EMC watcher, I have to congratulate EMC on keeping the launch under raps. It made a significant difference from Joe's usual chatty style on discussing futures (think Atmos, where the eventual official launch was a bit of a damp squib as the details were well known before lift off).

And StorageZilla was obviously in a straitjacket in a backroom of the EMC manufacturing plant in Cork, labeled up as due for release only after the event started.

The reaction to all this has been fairly muted, though. Val B had a blog up 24 hours before the bar opened, HDS have a couple of blogs on the subject -- one a whole list of questions which I thought pretty reasonable -- and 3Par did a rather odd guerilla piece. Some of the press had pieces; and no-longer-the-independent Steve Foskett summarised the whole thing for us. Lots of EMC!

All in all, given the scale of the hype, I thought the response fairly low key. Perhaps a day later we're all still a bit hungover from the party to work out -- is it any good?

That's why I haven't managed to digest the entirety of the announcement. There is rather a lot to take in, with a good deal of it marketing hooplah and broohah that needs to be sifted out before I can see clearly the direction EMC are headed. I can't speak for others, especially users, but that's where I am right now. Reading and trying to make connections.

And although Barry Burke breathlessly hyped blogged his little heart out on the subject of the technology, so far I'm having a hard time of connecting it back to reality and what you actually do with this stuff.

So I'm still digging through what EMC veep Dave Donatelli described at the launch as as the VMA and V-Max "twofer". (That's a word that's not well understood outside of the US, by the way. He may also not be aware that it also refers to attaching two light bulbs to a dimmer. Someone ought to tell him.)

Eventually, I suspect that although some to most of the V-Max and VMA details will turn out to be less impressive than at first sight, there will no doubt be some novel technologies that I can get my teeth into.

One thing I'm positive on; the reaction to the huge launch party wasn't big enough for EMC. Stirrer-in-Chief for EMC has his big stick out and is furiously trying to generate some reaction, mainly by poking a bunch of hornets who really weren't interested in stinging EMC for executing a fabby marketing launch.

Just give us a bit of time to clear our heads and dig through the mountains of discarded streamers and ribbons. When we've something to say, don't worry; we'll say it.

Here's one for starters; there are issues of when you can get a lot of this technology. There seem to be a number of futures, like FAST, that when subtracted out from the announcement, make for a much smaller ripple than EMC might claim in the pond that is the storage industry.

And am I surprised at this kind of pre-announcement by EMC? Not really, although EMC just hate it when we appear to do it to them.

Hypocrites! Bzzzzz... we're coming after you....

.

Who Wants To Be A Millionaire?

When NetApp came out with Virtualization Guarantee* Program (the 50% version) I was unsurprised at the ferocious chorus of disapproval from EMC, particularly from their bloggers and various EMC pundits on IT news sites. Given that the program is optional -- no one is forcing customers to take it up -- and that Gartner, amongst others, wrote approvingly of the idea, it really didn't concern me that much, and it was to be expected as EMC didn't have one (or even want to offer one). But what has got me and others really worked up was the mischaracterisation of the offer, with some pretty whacky observations about the details.

Next, we announced an updated version of the program based on 35% savings, using other people's storage (including EMC’s) fronted by a NetApp V Series. With this offer, you'd have thought we were dumping toxic waste on your doorstep if your only source of information was an EMC blog.

But now VMware has just announced their 50% savings program. At this point, I'd thought we'd get radio silence from EMC on the subject of our offer. After all, when EMC’s subsidiary (they own 90% of VMware) announces a program that's remarkably similar in nature to a competitor's program, and EMC has just spent the last 3 months dinging a program very like it, you'd probably expect at least a red face or two.

Nope. There's no shame in Hopkinton.  

Peter O'Connor (he's a Big Cheese Down Under with NetApp) is as sick and tired of hearing EMC’s usual line on this, as is everyone else. He’s decided it's time for EMC to put up or shut up.  

And as VP of Sales in ANZ, Peter's got the moolah (a cool $1million of it) to make it happen.  

The offer is to an existing EMC customer in Australia. For up to 50TB of raw storage, including controllers, software, and services for free if they’re willing to migrate their existing VMware environment to NetApp and go public with the results. There are no strings attached to this NetApp offer. Regardless of what the result is, they get to keep the kit.

It’s our response to the EMC Guarantee, listed below.

• It's not fair to compare RAID-DP to RAID-10.
• Nobody uses RAID-10 for VMware. 
• The performance of deduped volumes is bad, bad, bad!
• It's all too hard to get working.
• EMC can do dedup for VMware too! Honest.
• It's only a marketing stunt!
• The VMware guarantee is nothing to do with us.
• blah drone blah-blah drone

We’ve challenged EMC to make the same offer, and go head to head on demonstrating their value in public in a real customer environment and compare their results with ours. Let's see what happens.

.

Please Squeeze Me (Part 2)

In this multi-part post, I'm exploring specific space-saving technologies; deduplication and compression, and EMC's recent entry into the market with the Celerra Data Deduplication product.

The first part covered some of the basic technology of deduplication and compression. This time; what the implications of using each of the technologies -- either separately or together -- means for the user of storage systems that feature it.

I think a terminology list is useful here.

Fixed block deduplication	Deduplication of matching blocks level (NetApp uses a 4K block)
File level deduplication	Deduplication of matching files
Compression	Encoding data into a form where it's smaller than the original; all storage systems use lossless compression techniques.
Inflation or Decompression	Decoding data back into its original form.
Hydration	Any technique that reduces the size of data; includes compression and deduplication
Rehydration	Any technique that restores data to its original form; the revers of hydration
Reduplication	The repetition of a sound, as in claptrap, hokey-pokey, razzle-dazzle, teenie-weenie. (See footnote)

EMC's entry into the storage deduplication market has been quite late; only this month did the Celerra get some form of deduplication.

Celerra Data Compression And A Bit Of Deduplication

EMC's solution based on two technologies; compression and file level deduplication.

This works on NAS (CIFS and NFS) only; no block level iSCSI support, and it doesn't apply to the CLARiiON backend at all, so no deduplication there. And here's the target EMC have to hit with this technology. Note the lack of VMware, backups and other data store types,

Volume type	NetApp Typical Deduplication	NetApp Typical Range
Home Directories	35%	20-50%
CIFS Shares	35%	20-50%

First, the file is compressed. There's a lower limit on compression, which is reasonable; no point in having overhead where the savngs are minor. There's an upper limit too, of 200MB, which caught me by surprise. Compression is claimed to return 40-50% space savings, which is fairly impressive given that the Celerra uses no form of hardware compression; it's run as a background task, and uses spare cycles on the CPU.

Once compressed (or if too small/too big and left in its fully inflated state) the file is deduplicated. A key is generated (shades of EMC's Centera here, although it's a more robust SHA-1)  and compared to existing file keys. If it's a duplicate, the space is released. If it's not, the file is moved to a hidden area (which makes it secure from accidental reading or writing I presume).

At first blush, it looks like the majority of the space savings are though compression (40-50%), followed by a 10% reduction due to duplicate files being eliminated. Take 3 duplicate 1MB files, compress and deduplicate; voila, 500KB of storage required.

The Downsides

Firstly, there are the downsides of compression and decompression on hot data. remember, data must either be inflated in memory if you need to read it (think backup) or inflated on disk (if you need to write to it.

If we want to read one of our 1MB files, it needs to be inflated in memory. There's a small but definite overhead here, although it may be offset by reading less raw data.

If we need to write to it, the 1MB file needs inflated on disk. That means a 500K read, in memory inflation and a 1MB write. The upper limit of 200MB is there for a reason; CIFS times out if IO isn't satisfied pretty quickly. Form the EMC whitepaper;

• Performance testing has shown that Celerra can reduplicate a 200MB file in less than [the CIFS timeout value of] 25 seconds even when under heavy load.

That first write could take a very long time indeed.

The Mitigating Factors

It's not all doom and gloom. EMC again.

• Most applications do not modify files. They typically make a local copy, modify it and [...] write the entire new file [...] discarding the old copy. Therefore, the file is never reduplicated on the file server; it is just replaced.

Actually, this one is just based on old data, or perhaps it's just a best guess. NetApp's public research into CIFS data access patterns shows that this is not true for corporate type data; read/writes make up over 45% of access over CIFS.

• Celerra avoids processing active files (accessed or modified recently) [...] Deduplicated files are less likely to be modified and, if they are, performance is less likely to be a critical factor.

So EMC recommend that you don't address hot data. Given that most file systems have lots of cold data, and that cold data doesn't get read much or written much (that's why it's cold!), problem solved. That one has some substance; the same research shows around 10% of the data was accessed in a nearly 3 month period..

Target met! Or is it?

Large Files Dominate

EMC again.

• By default, Celerra avoids deduplicating files larger than 200MB.

200 1MB files takes up as much space as one 200MB file. Obvious. But with the Celerra compression limits you can only compress one-half the total of 400MB. Suddenly, size is important. It's only 1 file, but it's a significant 50% of the data. How much data is like that, I wonder.

I can't tell for certain. But here's a clue; my laptop has only 18 files bigger than 200MB, including things like CAB files for Windows Office. But that's a staggering 6GB out of 22GB data; more than 25%. (No large VMware images in this set either, because it's a VM I'm running on).

And the deduplication doesn't work on files smaller than 24KB; too much overhead. How many of those I wonder... 35580 of them less than 24k. I need a big assumption here'; let's say the average is 12K. That's approximately 400MB, so +/- a few 100s won't make a big difference; the assumption is a safe one.

The Sums

Let's add the effects together, using my laptop data as an example.

• 22GB - 6.4GB of files that won't deduplicate = 15.6GB
• 15.6GB where 90% are unused for an extended period = 14.1GB

I can address 14.1GB with this product. The other 7.9GB is untouchable.

• 14.1GB compressed by 50% = 7.1GB
• 7.1GB deduplicated by 10% = 6.4GB

That's 7.9GB+6.4GB = 14.3GB, or a total reduction of 35%.

Whew. Just made it! 35% is achievable in theory.

If you can get 50% compression. Remember, Microsoft's NTFS already does data compression. And on the directories where I have it turned on, I get something like 15% compression, not 50%.

And if 10% of your compressible files are duplicates. And at some cost.

Worth It?

No, because of the kind of impacts this compression and deduplication brings that NetApp's block deduplication just doesn't suffer from. Here's a list of things to watch out for;

• EMC dedupe is for Celerra only
• EMC dedupe does not support VMware (VMDKs)
• EMC dedupe does not support LUNs (iSCSI or FC SAN)
• EMC dedupe is only done at a file level.
• EMC dedupe only runs on one datamover at a time
• EMC dedupe is targeted at only infrequently accessed (inactive) files
• EMC dedupe only works on file sizes less than 200MB
• EMC compression only works on file sizes greater than 24K
• EMC provides no indication of savings to be made.
• EMC gives no estimate on the overall compression and deduplication you should expect
• EMC recognizes that there's a performance impact in reading deduped and compressed files.
• EMC dedupe requires EMC dedupe compatible backup tools (of which there is NetWorker and Veritas NetBackuo, for a total of 2) unless you want to inflate all the data
• EMC dedupe restricts restores to full volumes only from backups
• EMC dedupe means file level restores have to use SavVols and CoFR (copy-on-first-write) snapshots that impact performance

Looks all the world to me like EMC are squeezing you, not your data.

Now, compare that with NetApp's solution and the savings you can make. No contest.

 

---

Reduplication. An aside; what's with this word? It may be EMC's attempt to avoid some truly horrible alternatives. Undeduplicate sounds like two pairs of underwear, inflation is bad economics, decompression is the noise air makes coming out of a diving suit, and choosing those words is understandably Not A Good Thing. Although perhaps a more settled word like rehydration might have been better and more widely understood.

Reduplication as is normally understood means the repetition of a sound, as in claptrap, hokey-pokey, razzle-dazzle, teenie-weenie. Just like EMC's deduplication, methinks.

.

Please Squeeze Me (Part 1)

In this multi-part post, I'm exploring specific space-saving technologies; deduplication and compression, and EMC's recent entry into the market with the Celerra Data Deduplication product.

There are 3 commonly used deduplication technologies, and compression, and now the the storage industry is beginning to take advantage of them.

The end goal; storing more for less.

The three varieties of deduplication are fixed block, variable and file level. They're all remarkably similar in terms of what they do; they single-instance some level of data. Variable block I see as a variety of deduplication akin to file level, so I'm going to hand-wave it away right now and focus on the file level and fixed block varieties.

First, let's clear up one point. Are compression and deduplication synonymous? Here's what one company thinks; it's from an EMC whitepaper (more of which in part 2, from Achieving Storage Efficiency through EMC Celerra Data Deduplication, Jan 2009).

• Compression is often considered to be different from deduplication. However, compression can be described as infinitely variable, bit-level, intra-object deduplication. 

To be honest, I think that description is so wrong it's not even wrong. It might even have Shannon spinning in his grave.

• Deduplication is a form of delta encoding. In other words, only the differences between objects is stored; in the NetApp case, only different and unique blocks in the same volume.
• Compression is the process of encoding information using fewer bits than the original data would use. There's no delta and no deduplication.

I know why EMC are saying this. There are major differences between compression and deduplication in cost and outcome, particularly with respect to performance and capacity, but the industry buzz around the deduplication word is so attractive, it makes good marketing sense.

Yes, I know that EMC have a product called called Celerra Data Deduplication, and that it deduplicates data. But it should really be called Celerra Data Compression And A Bit Of Deduplication, because compression is where the savings come from.

And the downsides, of which more in part 2.

File Level Deduplication

This works only on objects like files; the major difference between it and fixed block deduplication is the size of the object. First, it's variable length, and second, it's an all or nothing test for duplicates.It's often describes as SIS -- Single Instance Storage.

If two files are identical, it's possible to store only one and have two directory entries point at the same file. UNIX has been supporting this for years with symbolic links; the links are aliases of the data that makes up the file. But that's an expensive active management process for the savings involved, and file level deduplication brings the advantages of automation and accuracy.

At a minimum, there's a metadata (data describing data) overhead of counting the number of times a file is pointed to, and some hash of its contents so that files can be matched.

And file level deduplication depends on lots of duplicate files; files have to match for length and content exactly. One further limitation is that files must normally reside in the same file system. That can limit the benefit, and ratios will depend on how file systems are arranged and their contents.

Writing to files stored like this also brings a downside; the file must be copied. That's twice the space, and there's the overhead of the IO to read the original and write the copy too.

How much file-level deduplication can be achieved? That depends, and I've not seen much research around this issue. Anecdotally, 10 to 20% seems reasonable. A reminder; if it's higher than that, perhaps you need to do some housekeeping and just delete all those duplicates...

Fixed Block Deduplication

NetApp continues to be the only vendor with deduplication on primary storage. It's a simple technology in principle; for every block written to a volume, check if there's been an identical block written before. If the answer is yes, then release the block to the free block pool, and point at the duplicate block.

This is based on deduplicating 4K blocks; and it works on SAN block based and NAS file-based data too. And because NetApp systems use pointers to blocks, reading and writing data in deduplicated volumes is simple. Read a block; get the block pointed to. Update a block or write a new block; write it, point to it, and the system will deduplicate at leisure later.

There's metadata associated with this technique; we need to save a unique  fingerprint for each block.

It's all covered in this NetApp Technical Report, so I shan't repeat much more. But it's worth drawing out one of the more interesting tables that indicate the levels of deduplication achievable;

Data Types	Typical Space Savings	Range
Backup Data	90%	85-95%
VMware VMs	70%	50-90%
Geoseismic	55%	40-70%
Database Backups	55%	40-70%
Home Directories	35%	20-50%
CIFS Shares	35%	20-50%
E-mail PSTs	30%	20-40%
Mixed Enterprise Data	30%	20-40%
Document Archives	25%	20-30%
Engineering Archives	25%	20-30%

For many classes of data, this is a considerable saving.

Compression

One thing that NetApp doesn't do is compression on its primary storage system (FAS and V Series). It's a feature of our VTL backup device, for reasons that will become apparent.

Compression is a well understood technology with a wide range of applications. Good compression depends on the type of data being compressed; the more random the data, the less likely it is to compress. For the kind of application data you might find on the average NAS box, as much as 50% should be achievable. That is a saving worth making.

There are downsides to compression. There's the not inconsiderable CPU load of compressing and inflating (decompressing) the data, although some systems (like NetApp's VTL) use specialised hardware to offload the task and avoid the processing impact.

More importantly; there are two things you can't do easily once the data is compressed. Reading and writing.

Reading means metadata; you can't read compressed data it without inflating it, and that normally means reading the whole object that was compressed, decompressing it and saving it in its original inflated form.

To get round this problem, systems such as NetApp's VTL retain a certain amount of metadata, so that the system can find blocks of compressed data, and just inflate these blocks.

An example; if I want to read bytes 10000 through 20000 of a compressed file, I look up the meta-data to get the blocks that contain this range, read just them, and inflate in memory before passing back to the requestor. There's no need to reinflate the data on disk at all; it can stay in its compressed format.

Writing means Inflation. Compressing data produces variable rates of savings, so it's not possible to simply update a section of a file by compressing it and getting it to exactly fit in the space available. Writing means inflation; again, the compression hardware can do this and avoid the CPU impact, but there's no avoiding the fact that the data is now occupying more space.

And there's additional IO to account for; the compressed file needs to be read, and the inflated file written back to disk, which can cause considerable delays on doing that first write.

(Worth noting that, as NetApp VTLs are archive and backup devices, they don't need to do updates like this; so the compression plus metadata approach works well.)

Compression levels of up to 50% are achievable for some, but not all, types of data. The data we store tends to be non-random in nature. Text files compress very well, as do certain Microsoft file formats such as PowerPoint. The notable exceptions are already compressed file formats like MPEGs and ZIPs for example.

And if you're wondering why you can't get any benefit by compressing compressed data, the pigeon-hole principle explains why..

---

More in part 2.

.

EMC Says SSD Is EOL with TLA now EFD

I have very little against coining words, phrases or acronyms that clarify the subject at hand, or simplify communication.

• DNA. Deoxyribonucleic acid; tongue twister
• BBC. British Broadcasting Corporation; doesn't exactly roll of the tongue
• LOL. Laffs Out Loud, for the keyboard and time challenged.

The IT industry has been doing it for years. But there are limits to TLAs, and when obfuscation and distraction are the goal, I'm not a fan.

EFD. Enterprise Flash Drive.

EMC are now employing this new acronym to distract the world from the pretty obvious fact that it really doesn't have a clue how to use this flash technology at all. In the bizarre and Byzantine world that is EMC, the invention of new acronyms is normally an admission that you're about to see the goalposts move.

And here we go...

EMC bloggers are now using EFD with reckless abandon, no doubt dictated by the EMM (EMC Marketing Machine) to for the purposes of FUD. I noticed it on several EMC blogs recently; Chuck Hollis' blog first, it leaked over to StorageZilla, and now Barry Burke has chimed in with what can be only described as the best justification I've ever seen for LOLing at what EMC are doing.

I'd suggest EFD needs to RIP. May it die a swift and unlamented death.

If you really want to see how to use new technology, rather than how to coin useless acronyms, point your browser here.

---

Acronym Translator

• TLA: Three Letter Acronym
• EMC: Embrocation Marketing Company. Purveyor of smelly lotions.
• SSD: Solid State Drive. A disk made of flash instead of rotating rust.
• EFD: Enterprise Flash Drive. SSD after an EMC lotion lather and rub down.
• FUD: Fear, Uncertainty & Doubt. The emotions brought on by smelling an EFD
• EOL: End of Life. Killed. Terminated. Unloved.
• RIP: Rest In Peace. Now you're EOL, please stay dead.

.