NetApp - The Missing Shade of Blue

I'm Sending Chuck a Bluey

My youngest daughter is going out with a professional soldier. That's tough for her; soldiers in the UK Armed Forces are often on active duty, and he's served several long tours of duty in Afghanistan already, something that she finds difficult to handle emotionally when he's way.

During his last tour, she wanted to send him a bluey. It's the Services equivalent of a telegram; you write your bluey online, it gets sent and printed locally, and most if not all blueys are delivered in 24 hours, even to servicemen right on the front line. It's good for morale, and the service is excellent.

Her first look at the website resulted in tears. When she looked at the information to get the letter to him, she discovered that all she knew was his name, his rank, and that he was in Afghanistan. No other information, and although she thought she knew his regiment, she wasn't sure.

Dad to the rescue. The boyfriend's name is unusual. Very unusual indeed, with a first name from Greek antiquity, and a surname that appears four times in the local telephone directory -- and they're all family members. This guy has a name you'd never forget; a name, like Roman Rock (not his real name, because he's still on active service). If there's another person in the UK with the same name, far less in the Army, I'll eat my shorts. So I calmed her down, and helped here address her bluey, to Roman Rock. That was all the information we could fill in.

It got to him in less than 12 hours. We didn't know where he was located, but the Army did, because he had a unique name.

Chuck Hollis had an interesting blog on filling out blueys. Well, sort of; on what he sees as the death of the filesystem. The topic struck me as interesting; The Future Doesn't Have A File System.  As usual, but with good reason, I disagree.

Object-based information stores are different than filesystems in several important ways. 

First, you use a token or other uniform identifier to get your information.  File systems imply location, tokens don't -- no such thing as a broken link or a moved file system.  Not to mention, tokens can uniquely identify gazillions of information objects.

Disagree; filenames don't imply location. They're just names; metadata that allows you to uniquely identify the data you wish to access. This is a problem that the internet created, and that the internet solved; see later.  

Second, they have the ability to associate all sorts of metadata with the object itself.  As the information object goes, its metadata travels with it.  A very useful property indeed.

And that differs how? You can associate metadata with a filename -- if you can do it for an "object", you can do it with a file. (Even old-fashioned filesystems keep things like last access time, size, and the name itself is often used to give further, human readable clarification, such as .html or .doc; although it may lie and we may care not to use it for such, that doesn't change to point of associating metadata with the name-as-a-handle.)

Third, the ability to hang metadata off the object gives us the ability to create all sorts of useful policies and services around the information without having to put everything in some sort of database or repository.

Double eh? If this (it's a GUID, or globally unique identifier);

c2f41010-65b3-11d1-a29f-00aa00c14882

doesn't require a repository or a database for its metadata, I'll eat my shorts. Again.

The internet solved this problem a good while ago. See RFC1737, RFC2141 and RFC3986. The essence of the RFCs; part is a name (like Roman Rock) and part is a location (like Afghanistan). The part that is the name doesn't tell you where the file is located; but the part that describes the location can be completely absent.

An example is in order. To demonstrate how far things have changed since Chuck banged away with Unix pipes and vi in the1900s, this returns a file;

http://blogs.netapp.com/shadeofblue/2009/08/poetry-corner.html

Interestingly, the file doesn't exist until you ask for it, because it's dynamically generated from parts. There's no directory shadeofblue, 2009, or 08, and no file poetry-corner.html either. And (here's a clue how far this goes) it doesn't live on a server at netapp.com either. It's all name and no location, and it works across the entire internet, not just inside a single object store.

Filesystems aren't the problem here; it's an attempt to make Atmos relevant. I think I'll send a bluey to Chuck and let him know; The Future Doesn't Have an Atmos.

 

---

I'm off to VMworld next week in San Fransisico. It's my first time at a VMworld conference, and I'm not quite sure what to expect.  But I do hope to meet lots of interesting people I've never met before while I'm out there. If you recognize me (yes, I look like the photograph), please introduce yourself. And no, I don't bite!

Free Cheese from EMC

EMC's Barry Burke seems mighty pleased to announce that Virtual Provisioning (what everyone else calls thin provisioning) is now "free" (at least on the V-Max and DMX4 storage arrays but not on the CLARiiON).

I'm having a remarkably similar argument with HP folks on their assertion that the software on their LeftHand storage arrays is "free", even though there's a big difference in the price of similarly specified servers and the P4000 based on the same tin. The same with Sun; their "free" software on the the Sun 7210 array costs a lot. The system is based on a Sun X4540, which is much cheaper.

And Barry's challenge "We'll see if others follow suit" line is just so, well, yesterday's news. There are any number of storage vendors that have thin provisioning at no additional cost, including NetApp FAS systems; since 2004 to be precise.

What Barry means by free is at no additional cost, and it's an important difference. And I don't  think buyers are fooled by this use of the word. It's the total cost that you need to look at, including the free. He eventually gets there; right in the last-but-one paragraph where it's "all at no extra charge".

Free? There's always free cheese in mouse traps, but the mice there aren't best pleased.

.

Please Squeeze Me (Part 3)

In two previous blog entries I explored EMC's late entry into the deduplication market with the EMC Celerra Data Deduplication product.

I used the contents of my laptop to demonstrate that getting to 35% savings was a bit of a squeeze using EMC's technique for deduplicating.Perhaps it wasn't a good representative sample of the kind of data you might typically see on home directories, and that it might be better with real life user data.

Nothing like real life data for making a point.

So here are some real numbers from a real user. I've used anonymous customer data before, in a usable space analysis. Using some advanced technology (in this case, NetApp's Information Server 1200) I can make a much more accurate assessment of the capabilities of EMC's latest but not so greatest. In fact, for customers looking to see how much EMC deduplication can't save them, I can strongly recommend its analytic capabilities.

The conclusion? 30% deduplication on the Celerra, with some generous assumptions. See below the line for the gory details.

The real figures will be much worse, and look out for all these other caveats.

EMC dedupe:

• is for Celerra only
• disables MPFS 
• does not support VMware (VMDKs)
• does not support LUNs (iSCSI or FC SAN)
• is only done at a file level.
• is targeted at only infrequently accessed (inactive) files
• only works on file sizes less than 200MB
• compression only works on file sizes greater than 24K
• has a performance impact in reading deduped and compressed files.
• restricts backups and restores to full volumes 
• means file level restores have to use SavVols and CoFR (copy-on-first-write) snapshots that impact performance

What will it look like on your data? 20% space savings? Less? The EMC calculator doesn't work on actual data, just %age estimates by file type. Give it try.

NetApp provides two calculators. A deduplication calculator, like the EMC tool (except that NetApp's deduplication works across all primary data, not just NAS files), and SSET, a space saving estimation tool that uses actual data on your system. Actual results are within +/-5% of the space savings that SSET predicts.

I still think it looks like all the world to me like EMC are squeezing its customers, not your data. And, missing in action; V-Max deduplication?

---

The Details

EMC's "Deduplication"

Let's recap. For home directories, EMC's deduplication uses the following rules for "cold" or not frequently accessed files that pass filtering;

• If the file is less than 24K, don't compress, only deduplicate
• If the file is >200MB, don't compress or deduplicate; leave it alone
• Otherwise, compress and deduplicate

For "cold", I'm using a figure of 90% of the data; and I'm also assuming that all files between 0K and 250MB are compressible by 50%. That will make the EMC numbers much better than are achievable in practice. In reality, YMMV. You might see a smaller or larger %age of files as cold, and for certain, not all files will compress by 50%; many will not compress at all.

But as you'll see, my original 35% was generous.

Large Files Still Dominate

As I noted before, large files may be few in number, but the represent the bulk of the data. Here's a sample from a user system. This file system contains 20 million files in just under 13TB; it's CIFS home directory data. (Click to see full sized).

Small files make up the bulk of the raw file count, and the large files in excess of 250MB are very few in number (less than 0.03% of the total file count).

But those large files over 250MB make up over 41% of the used space.

That leaves 59% of the data for EMC to deduplicate (well, compress first and deduplicate second).

Duplicates

Duplicate files are interesting. There are very few duplicates for files in excess of 50MB, and the savings from deduplicating those show a 1% decrease. So not deduplicating them has very little effect.

Files smaller than 50MB show an average of 13% duplicate space savings, although over 30% of files are duplicate by count. Here, it's medium sized files; most of them make up the duplicates. Small sized files are rarely duplicates. I'll err on the side of safety and use 10%.

At least EMC are deduplicating the right files!

Active Files

In terms of active files, 35% of the space was made up of active PSTs. EMC's deduplication doesn't do active files, and again, that's a big chunk of data we can't address. But most of them are in excess of 250MB and are in the "large files" category, so to remove them would be to double count, so I haven't.

The Results

Now we have some raw figures, let's go compress and deduplicate.

Taking our 13TB file system, we have

1. 13TB total, but only 60% or 7.8TB is addressable files <250MB
2. 90% of the 7.8TB is cold, giving 7.0TB to compress and deduplicate
3. With 50% compression, 7.0TB compresses to 3.5TB
4. 3.5TB file-level deduplicates by 13% to 3.1TB
5. There's a total saving of 7.0TB - 3.1TB, or 3.9TB
6. 13TB - 3.9TB is a 30% saving

A 30% Saving Based On Real Data

Here were my working figures and assumptions, and I was generous.

• For files <250MB
• 90% of your data is cold
• 13% of your data is duplicated
• All files compress by 50%

If we use the numbers EMC provide from their whitepaper (Achieving Storage Efficiency through EMC Celerra Data Deduplication, Jan 2009);

• For files <200MB
• 10% of your data is duplicated
• Files that compress do so between 40-50%

.

EMC and the Fake Rolex

Much of the mail that I receive on my computer at home consists of spam. Badly written, most of it is offering me enough Viagra to give me a serious balance problem, chances to play roulette, easy money from the sons of deposed African cabinet ministers who have amassed stolen fortunes that they want to launder through my bank account....

It's a serious irritation, but occasionally it's amusing. An example; this heavily capitalized email from the fabulously named Lagina Cowper.

From: Lagina Cowper
Subject: FAKE ROLEX!!!!

Look a MILLION DOLLARS with our ROLEX WATCHES!!! LEADING CHEAPER than the real thing, we have DAYTONA CSOMOGRAPH, ROYAL BLACK, OYSTER,...

NetApp has some fairly sophisticated email filters that weed out most spam. So you can imagine my surprise when EMC's Chuck Hollis blogged today, and my RSS feed popped this into my work inbox. It's about Frankenstorage. Chuck says;

...NetApp did their interpretation of frankenstorage when they announced a lash-up with TMS's RamSan-500 to tick the box that, yes, they had a flash story to tell. 

Chuck is calling NetApp systems Frankenstorage?

That's funny! Having tried to hijack the unified storage message, EMC's very own Lagina Cowper is suggesting that other competitiors' storage are just bolted together mishmashes of parts.

Where, I wonder, did he get the idea of Frankenstorage from? There's no stopping a good idea, so perhaps it was from my blog. I noted in November last year that the Celerra NS range was due out shortly.

It Lives! EMC’s FrankenNAS.

Unsurprising will be the throwing of the big switch to zap into life a new EMC NS range based on the Celerra and CX4. I’ve lost count of the exact number of moving parts in the average EMC NAS solution, but any press release will probably skim over that and claim “market leading unified storage” with a cuddly personality. I’d suggest counting the bolts and keeping a safe distance.

I also wonder if Avamar based NAS deduplication will get announced too? I’ll be fascinated to see how they wire that one up in FrankenNAS’s two brains. And how much it costs.

I suppose it's a common enough analogy that Chuck could have thought of it independently, so let's put the copying of the Frankenstein imagery aside. Here's the serious point.

By lagging behind the competition, EMC suffers from the marketers' dilemma; how do we make yesterday's products appeal to today's customer? The answer; copy. Copy everything. Ape your competitors' stance, and employ serious marketing muscle to make it look like you invented the idea in the first place. 

If you can't make it, fake it.

But here's an example of EMC copying that really gets to the heart of the issue. Unified storage. Chuck keeps saying that the Celerra is unified storage, even though it's a bolted together set of hardware, and so fails miserably by his own criteria. But Chuck keeps missing the point, and will miss it forever because he can't seem to grasp one of the fundamentals of this industry.

It's not about the hardware. It's all in the software.

It's the software that makes NetApp's solution unified storage, and exposes the Celerra for the bolted together FrankenNAS is really is. Let me repeat; software.

Question. How many operating systems, types of snapshots and clones, pieces of incompatible replication, varieties of deduplication software does the average EMC customer have to buy and manage? Lots, and that's a problem. EMC's answer is to fake a solution. They call it "unified storage", but it's plainly and painfully obvious it isn't.

So you can understand why I laughed out loud when I read this from Chuck's blog;

How do we stop these monstrosities?  That's up to customers, isn't it? 

Good question, Chuck. I for one will be making sure they understand the real issue around Frankenstorage.

Your marketing spin on this is just the same as the cheap watch spam from Lagina Cowper. Except Lagina is more honest; that Rolex really is a fake.

.

Finding a Pair of Socks

One of the common pieces of FUD (Fear Uncertainty and Doubt) that gets sent by colleagues to my inbox at ShadeOfBlue Towers is that everyone is saying NetApp FAS systems get slower as they fill up.

Some vendors have even taken to pointing our customers to this NetApp document as proof. Help me! Is it true?

Here’s a graph from the paper. (A point here; read the full paper for the details, but the lines to look at are the blue to 50%, followed by the green from 50% to 100%. And take a look at Kostadis’ blog for more detail on our WAFL technology.)

Look at the equivalently spec’ed EMC system below the NetApp line. Go ask other vendors for their graphs of performance at specific capacity points too; I suspect the silence will be deafening, or you might get lucky and get back a remarkably similar chart.

Is the FUD point made? No, because every storage system gets slower as it fills up. And some just start bad, and then get worse.

So why does increasing capacity utilization affect performance? Here’s an analogy; it’s like me trying to find a decent pair of matching socks (preferably without holes) in my sock drawer. The more socks, the longer it takes me to find just the right pair.

But with something that understood my sock needs, it needn’t take quite as long. So how do we make sure that performance stays high as the system fills? We’ve worked hard on this problem, and we’ve got some cool technology that helps.

Here’s one; the NetApp Sock Handler.

Readsets

Prefetching sounds pretty easy. For instance, in a file based NAS system (if it’s smart enough), when you start reading a file, the likelihood is that you want to read all of it.

For block based though, the read patterns can’t be ascertained the same way, as there’s no semantic information as in the file case that can help with the prefetch pattern.

For all read operations, large amounts of cache is employed in a NetApp system for staging pre-fetched data. We uses an adaptive read history logging system based on “readsets,” which provide much better performance than traditional and fixed read-ahead schemes.

In fact, multiple readsets, can support caching for individual files (for NAS) or LUNs (for SAN), which means that multiple read streams can be pre-fetched simultaneously. The number of readsets per file or LUN object is related to the frequency of access and the size of the object.

The system can adaptively select an optimized read-ahead size for each read stream based on historical factors, such as

• the number of read requests processed in the read stream
• the amount of host-requested data in the read stream
• a read access style associated with the read stream
• forward and backward reading
• identifying coalesced and fuzzy sequences of arbitrary read access patterns

Cache management is helped by algorithms that determine when to perform read-ahead operations, and how long each read stream's data should be retained in cache.

Readsets are just one of the technologies that ensures NetApp systems stay as fast as possible as the system fills.

It doesn’t help with my sock drawer, unfortunately, but it might beat the FUD off.

.

ELF, Wealth and Happiness

Usable Capacity

How to describe capacity? I’ve been exploring the whole usable capacity argument in my last few blog entries. There’s one thing for sure; we're in desperate need of new words and phrases to describe new technologies. Of specific interest to me are the inadequate words

usable capacity or capacity utilization

I consider those to be describing (albeit unclearly) the same thing. Two recent examples come to mind.

How Long is a Space Walk?

Part 1 Three Men Make a Tiger
Part 2 Space is Mind Bogglingly Big
Part 3 Think Tesseract; More Space Than A Cube
Part 4 Micrometer, Crayon, Ax

Most Sunday mornings, Carol and I take a walk down to a local coffee shop, buy a newspaper and spend a little time catching up on the week’s news. This Sunday’s big headline (apart from the collapse of Western civilization as we know it) was Zhai Zhigang’s space walk and the successful return to Earth of the three Chinese taikonauts this weekend.

In the sidebar of the article I spot some other space walk facts. The longest space walk was by US astronaut Susan Helms, at 8 hours and 56 minutes, which beats the Chinese effort (impressive though it is) into a cocked hat. Zhai was out of the capsule for only 15 minutes..

“Amazing” says I to Carol, having caught her attention. “How long do you think the longest space walk was?”.

“Let me guess”, she says. “About a mile?”