NetApp - The Missing Shade of Blue

Random Rocks And Benchmarks

“People understand contests. You take a bunch of kids throwing rocks at random and people look askance, but if you go and hold a rock-throwing contest -- people understand that.” (Don Murray)

And that, in a nutshell, is the origin of most competitive sports. What starts as "Hey, I can chuck this boulder through that window over there -- betcha you can't!" develops through time a set of rules, referees, measuring devices, a governing body (sometimes two or three) and the sport of rock-chucking may even get recognised by many, played by many and make it into the Olympics. Like spear throwing (the javelin) or blasting birds out of the sky with a shotgun (clay pigeon shooting), rock-chucking has matured into shot-putting.

Just like benchmarks. Originally used to demonstrate some feature of your system in all its one-trick glory (HP IOPS from cache is a classic example of how to do this), modern industry benchmarks make the attempt to allow some real-world comparison between competitors. The relationship to real life is debatable, but the rules by which they operate are carefully designed to inject some aspects of realism to comparative claims.

Benchmarks are intended to be a repeatable test of a set of skills. But, as with all competitive sports, sometimes there's the kid who starts his benchmarking career hanging round on street corners stoning passers-by for entertainment, or hoisting up the nearest large rock at hand and heaving it through the closest plate-glass window.

And the new kid is over at HP, using a non-benchmark as a benchmark, and generally lobbing rocks around in all directions. I'm not going to dissect the post in detail, because others have and will continue to do that. There's one little paragraph I want to focus on, because it demonstrates one of my pet peeves; benchmark intuition.

Now things were starting to make sense.   We were seeing the same sort of decay curve as shown in the IOMeter results posted in Making Sense of WAFL - Part 4.    Every time the test is run, the random component of the Jetstress database accesses fragment the LUN further and the throughput numbers get worse.  An array like EMC CX or HP EVA wont undergo this sort of decay curve since these arrays do not have internal WAFL-fragmentation problems like the FAS does.

The non-benchmark is Microsoft's ESRP, and the tester's intuitive assumption is that WAFL fragments; hence the tester's intuitive assumption that this is the source of the diminishing throughput numbers.

Bzzzt. Big fail.

Let's allow the first intuition; let's allow, for the sake of this demonstration, that "WAFL fragments your data". Here's a simple example to demonstrate why his intuition is wrong on fragmentation being the source of the problem. Exchange 2007 generates small random IOs (and that's the JetStress that HP are using in their test). The table below has 5 columns to demonstrate why small random IO works just as well (or badly, depending on your take) on randomly laid out data as sequentially laid out data.

• Random Placement: I've place 100 blocks randomly. The numbers have been generated from www.random.org. Slot 1 is block 67, slot 2 is block 19 and so on.
• Random Requested Block: this is meant to simulate IO requests from Exchange; again, drawn from a different run from www.random.org.
• Matching Block (Random Placement): this is where the requested block actually lives, So asking for block 80 requires a visit to slot 22, and so on.
• Seek Distance (Random Placement): this is the effective seek distance between requested random blocks. After we visit slot 22 (for block 80), we need to visit slot 58 (for block 37), requiring a seek of 36 slots.
• Seek Distance (Sequential Placement): this is the effective seek distance between requested sequential blocks. After we visit slot 80 (for block 80), we need to visit slot 37 (for block 37), requiring a seek of 43 slots.

(I'm having difficulty uploading the spreadsheet to TypePad, so when I get it fixed you'll be able to "Click to download the whole spreadsheet". Not yet though.)

Random Placement	Random Requested Block	Matching Block (Random Placement)	Seek Distance (Random Placement)	Seek Distance (Sequential Placement)
67	80	22	0	0
19	37	58	36	43
75	18	61	3	19
23	26	53	8	8
85	57	63	10	31
59	100	14	49	43
14	59	6	8	41
...	...	...	...	...
SUM			3269	3322

Hey, look at that! The sequentially laid out data takes more slot seeks than the randomly laid out data! Try it yourself, replace the 100 numbers in the first two columns with random numbers, and check the seek distance sum.

On average, they will be equal. In fact, if the requested blocks are random, it doesn't matter how the data is laid out. Intuition fail.

Here's the professional sport of benchmarking, which HP don't take part in (still being, as it were, at the rock-throwing stage);

• NetApp SPC Benchmarks (no HP here)
• SFS2008 NFS benchmarks (HP missing again)

And, of course, the official ESRP results (and, just as a reminder, these aren't benchmarks)

• Microsoft ESRP results

Having failed at shot-putting, perhaps HP might want to pick another sport for their talented testers. Like nude football.

[updated to correct some borked links and a typo].

Finding a Pair of Socks

One of the common pieces of FUD (Fear Uncertainty and Doubt) that gets sent by colleagues to my inbox at ShadeOfBlue Towers is that everyone is saying NetApp FAS systems get slower as they fill up.

Some vendors have even taken to pointing our customers to this NetApp document as proof. Help me! Is it true?

Here’s a graph from the paper. (A point here; read the full paper for the details, but the lines to look at are the blue to 50%, followed by the green from 50% to 100%. And take a look at Kostadis’ blog for more detail on our WAFL technology.)

Look at the equivalently spec’ed EMC system below the NetApp line. Go ask other vendors for their graphs of performance at specific capacity points too; I suspect the silence will be deafening, or you might get lucky and get back a remarkably similar chart.

Is the FUD point made? No, because every storage system gets slower as it fills up. And some just start bad, and then get worse.

So why does increasing capacity utilization affect performance? Here’s an analogy; it’s like me trying to find a decent pair of matching socks (preferably without holes) in my sock drawer. The more socks, the longer it takes me to find just the right pair.

But with something that understood my sock needs, it needn’t take quite as long. So how do we make sure that performance stays high as the system fills? We’ve worked hard on this problem, and we’ve got some cool technology that helps.

Here’s one; the NetApp Sock Handler.

Readsets

Prefetching sounds pretty easy. For instance, in a file based NAS system (if it’s smart enough), when you start reading a file, the likelihood is that you want to read all of it.

For block based though, the read patterns can’t be ascertained the same way, as there’s no semantic information as in the file case that can help with the prefetch pattern.

For all read operations, large amounts of cache is employed in a NetApp system for staging pre-fetched data. We uses an adaptive read history logging system based on “readsets,” which provide much better performance than traditional and fixed read-ahead schemes.

In fact, multiple readsets, can support caching for individual files (for NAS) or LUNs (for SAN), which means that multiple read streams can be pre-fetched simultaneously. The number of readsets per file or LUN object is related to the frequency of access and the size of the object.

The system can adaptively select an optimized read-ahead size for each read stream based on historical factors, such as

• the number of read requests processed in the read stream
• the amount of host-requested data in the read stream
• a read access style associated with the read stream
• forward and backward reading
• identifying coalesced and fuzzy sequences of arbitrary read access patterns

Cache management is helped by algorithms that determine when to perform read-ahead operations, and how long each read stream's data should be retained in cache.

Readsets are just one of the technologies that ensures NetApp systems stay as fast as possible as the system fills.

It doesn’t help with my sock drawer, unfortunately, but it might beat the FUD off.

.

Benchmarking Thin Provisioned Systems

Marc Farley of 3PAR is providing some really excellent blog fodder for me over here at ShadeOfBlue Towers, and again – true to form – he’s done so on my second favourite subject.

Benchmarking. This time, lamenting the fact that there are no benchmarks with thin provisioning turned on.

Wait a moment. That’s exactly how NetApp does its benchmarks. FlexVols are the key here. The difference between a thin provisioned FlexVol volume and a full fat thick version?

• Thin: unused space comes from a common pool of free blocks, but the pool can be over-committed.
• Thick: unused space comes from a common pool of free blocks, but the reservation in the pool is guaranteed.

There’s no difference in the way the used space is handled. Thin or thick, a FlexVol is a FlexVol.

As these SPC-1 benchmarks show. With the space guarantees turned off (see page 67 for the vol create command we used), we were using the full WAFL thin provisioning infrastructure. Oh, and we did it with snapshots turned on too.

.

Lumpy Dell and Benchmark FUD

Edward "Lumpy" Stevens was a cricketer from the time when bowling was performed underarm or underhand, rather than the modern technique of roundarm bowling. Lumpy was famous for his ability to find lumps and bumps in the pitch, and consequently his bowling confused as the ball careered erratically towards the batsman.

For Honest Lumpy did allow
He ne'er would pitch but o'er a brow

Nowadays, cricket is a much more sophisticated game. Pitches are flatter, bowlers are faster, and umpires are on the watch for ball tampering and other activities that give the bowler an advantage. And underarm bowling is not part of the modern game of cricket.

So it’s time for the umpire to send Lumpy Dell back to the pavilion.

Dell Bowls Underarm

Every day, I get concerned sales and systems engineers emailing me about some document that our customers have been given by the competition.

“I attach a new document that Dell/EqualLogic is handing out to customers. Can you put together a response?”

The document is entitled

“Guide to comparing storage performance efficiency for business applications by leveraging Microsoft sizing tests”

In case anyone from Dell is reading this and needs a visual reminder, here’s the front page.

The contents are as long as the title, at 17 pages. But what is it about, and why would a sales colleague be looking for help? Dell’s words;

Microsoft wants to ensure systems are properly configured for running Exchange. The Exchange Solution Reviewed Program (ESRP) allows vendors to validate to Microsoft’s satisfaction sizing of storage performance and capacity for Exchange.

This, to be clear, is not one of Dell’s ESRP submissions; there’s nothing wrong them. This is a separate document that Dell give their customers.

The conclusions are going to be blindingly obvious, but I’ll repeat them anyway;

Microsoft-approved results of each vendor’s self-testing illustrate several advantages for EqualLogic:…

Comparisons Galore

It’s about everyone else’s ESRP submissions. This massive piece of marketing FUD directly compares and contrasts ESRP results from just about every vendor that’s ever published one.

EMC, HP, IBM, LeftHand, NetApp, Hitachi. In dense tables with helpful explanatory footnotes, the document details the lot. Dell again;

• IOPS per disk for most vendors
• Backup and log replay performance for most vendors
• Example results of each major vendor’s primary native SAN testing
• Examples of tier-2 SAN offerings

Then a point by point analysis of each submission and its shortcomings when compared to Dell/EqualLogic.

What a tragic waste of effort.

ESRP Is Not a Benchmark

Cheekily, Dell/EqualLogic’s FUD department have included a handy section called

Locating the key data inside ESRP reports

It is, of course, missing the one vital piece of information. Pick up any ESRP submission from any vendor; here’s one of Dell/EqualLogic’s. They all have the same boilerplate text. So pick up the link pay close attention to page 18. Here’s what it says;

The ESRP Storage program is not designed to be a benchmarking program. Its tests are not designed for achieving the maximum throughput for a given solution. Rather, they are focused on producing recommendations from vendors for the Exchange application. Therefore, the data presented in this document should not be used for direct comparisons among the solutions.

It’s Not Cricket

It might be a good idea for Lumpy Dell to re-read the rule book. No-one wants to be publically outed as underhand.

[updated 10 Oct; Skip (see comments) pointed out that this blog wasn’t very clear on which document I was criticizing. I’ve made clarifications.]

After the Virtual Fireworks

When I blogged recently on NetApp 50% Virtualization Guarantee Announcement I suggested that

It’s going to be interesting to see the responses to this announcement.

It was. There was considerable reaction. One blogger (not a vendor, but a user) likened it to lighting the blue touchpaper and watching the fireworks.

And one big point in many people’s minds is why we chose to make our comparison to RAID-10, rather than RAID-5, or why we chose not to make a comparison with RAID-6.

Those are good questions, and they deserve an answer.

RAID-10 vs RAID-DP

NetApp understands that the bulk of disk deployments has been RAID-5 for a while. The driving force behind the adoption has been cost reductions – RAID-10 is often just too expensive and most applications simply do not need or use the performance increases it gives over RAID-5.

However, performance is not reason we insist upon RAID-10 as the comparison disk configuration – it’s data protection.

RAID-5 simply does not measure up to today’s realities. We are looking to compare based upon what is actually required today, and not yesterday’s best practices. We believe that vendors promoting single disk parity schemes are not looking after their customer’s best interests, and this is at the heart of matter.

Today, many people are driving to lower costs further – and the turmoil in the economy is going to greatly accelerate this activity. Companies are looking at SATA drives and large format FCP and SAS as part of the answer to lowering the costs of their storage infrastructure.

But as drive formats get larger, the likelihood of suffering data loss and an outage due to double disk failure or block read errors during a rebuild are much more real.

If you have a look at the math in this IBM Best Practices for Exchange technical paper, it gets pretty obvious that RAID-5 is a new recipe for disaster and storage vendors need to provide much better guidance for their customers.

• Over a period of 5 years, RAID-5 is 4,000 times more likely to suffer from data loss than RAID-DP
• Over a period of 5 years, RAID-10 is 163 times more likely to suffer from data loss than RAID-DP.

To be honest, the question is this – why is your storage vendor not insisting on double disk protection?

RAID-6 vs RAID-DP

So why not compare to RAID-6?  The truth is, we could and would, but we don’t find our competition selling it. It’s there on the spec sheet as a solution from some vendors – not all have the capability -- but it’s rarely put forward as a viable solution.

[Update 10 Oct: StorageZilla now reckons RAID-6 is ready for the prime time, and StorageBod (a user who comments below) also reckons EMC promote RAID-6 actively. I asked several colleagues whether they ever see EMC propose RAID-6 when they’re competing, and the answer was a resounding no.. Unscientific I know, but I’ll stick by my contention that there’s not much EMC RAID-6 out there in comparison with RAID-5 or 10.]

Most, if not all, competitive RAID-6 implementations significantly reduce capacity and performance for increased data protection. That reduced capacity and performance is something that NetApp systems with RAID-DP don’t suffer from.

[Update 10 Oct: And I’m sticking by this one too. Again, see the comments.]

Our space guarantee for VMware makes that commitment for capacity. And our performance benchmarks, both SpecSFS for NAS and SPC for SAN, are all done with RAID-DP.

No Silver Bullet

But the RAID implementation is just part of the solution. There is no one silver bullet, and no matter how good any one piece of the solution is, for best results, our customers need to implement them all as explained in the program details.

This program is not simply about RAID, but storage efficiencies and the multiple technologies we bring to bear on the problem of providing superior capacity utilization while protecting data.

To quote my colleague Nick Triantos;

Once customers see the other advantages we have, in terms of performance efficient snapshots, rapid and efficient cloning and provisioning techniques, rapid data backup and recovery, and un-compromised data protection, they will realize a whole new way to manage their storage in a VMware environment.

---

Photo: Creative Commons Attribution 2.5 Steven H. Keys and KeysPhotography.com

My thanks to Mike Shea, NetApp, for the text and analysis.

ELF, Wealth and Happiness

Usable Capacity

How to describe capacity? I’ve been exploring the whole usable capacity argument in my last few blog entries. There’s one thing for sure; we're in desperate need of new words and phrases to describe new technologies. Of specific interest to me are the inadequate words

usable capacity or capacity utilization

I consider those to be describing (albeit unclearly) the same thing. Two recent examples come to mind.

How Long is a Space Walk?

Part 1 Three Men Make a Tiger
Part 2 Space is Mind Bogglingly Big
Part 3 Think Tesseract; More Space Than A Cube
Part 4 Micrometer, Crayon, Ax

Most Sunday mornings, Carol and I take a walk down to a local coffee shop, buy a newspaper and spend a little time catching up on the week’s news. This Sunday’s big headline (apart from the collapse of Western civilization as we know it) was Zhai Zhigang’s space walk and the successful return to Earth of the three Chinese taikonauts this weekend.

In the sidebar of the article I spot some other space walk facts. The longest space walk was by US astronaut Susan Helms, at 8 hours and 56 minutes, which beats the Chinese effort (impressive though it is) into a cocked hat. Zhai was out of the capsule for only 15 minutes..

“Amazing” says I to Carol, having caught her attention. “How long do you think the longest space walk was?”.

“Let me guess”, she says. “About a mile?”

Think Tesseract; More Space Than A Cube

---

If you've come here from Isilon's "guarantee" website, please be aware that they've got the figures wrong for NetApp's space efficiency.

Very wrong. The whole series on NetApp's space efficiencies is here, and it clearly shows that NetApp's space efficiency is much higher than Isilon's miscalculated claim.

If you really want to save money and be storage efficient, you need a storage system that gives you more space than you bought, by using advanced data deduplication, double parity RAID, and a number of other space-saving technologies.  Here, try a real guarantee. Accept no substitutes!

Thanks for visiting, and enjoy the blog.

---

Part 1 Three Men Make a Tiger
Part 2 Space is Mind Bogglingly Big

Scientists have puzzled for years over the question; why does space appear to only have 3 dimensions? Some speculate that space has more, with tiny curled up dimensions that can't be seen; up to eleven at last count.

Back here in IT land, we don't even have three. Poor old disks and tapes have to do with one dimension; lines of bits laid out on a rusty glass platter or a strip of mylar.

But users of this one dimensional space have come up with a solution; map another dimension over the top of the single stream of bits. Structures like filenames in directories and volumes and LUNs give us a second dimension to our data.

The third dimension? (OK, this is all for effect, but bear with me.) Differentiate the second dimension by adding a third dimension; qualify and assign a type to the second dimension, and identify that file as a spreadsheet or a document or a database.

Here at NetApp, we've discovered the fourth dimension.

Non Duplication

There's a lot of buzz in the industry about deduplication, but what many fail to understand is that it's possible to not duplicate data in the first place. It's what a colleague of mine, Mike Riley, called our non-duplication technologies. They add a fourth dimension to the data.

An example. NetApp snapshots are famous for two reasons.

• They're fast. Not only are the fast to take, they don't get in the way of I/O to the base data they snapshot. They're snapshots which don't sap performance.
• They're capacity-free when taken. Only changed blocks to the underlying data require additional space.

So How Much Snapshot Space?

Here's the latest set of figures from the NetApp AutoSupport data discussed in my previous blog entries. (Sorry, it's a picture. Inserting tables is next to impossible.)

UPDATE: corrected, 83TB is now 83PB

The used snapshot space is less than 3% of the total usable space. Which demonstrates how conservative NetApp are with recommendations of how much space should be reserved for snapshots; these systems reserved 13.4PB, much more than the committed space of 2.4PB. For a fuller description, Val Bercovici details the flexibility of NetApp's snapshot reservations.

How much data does that represent? Multiplying the snapshots by the size of the volumes that have snapshots, it's a whopping 1,511PB of non duplicated data.

Effective Load Factor

How should we describe this non-duplicated data that has the appearance of being a much bigger set of data? I like to think of this in terms of load factor. Worst case is to compare with the raw disk space. The data we are storing is well in excess of 1,500PB on systems with an total capacity of 217PB.

That gives an effective load factor of over 7 to 1 for these systems. Considering just the snapshot data, it's a load factor of 700 to 1.

Bottom line

It really doesn't make sense measuring usable storage with a micrometer. With 4th dimension non-duplicating technologies like our SnapVault, SnapMirror, Thin Provisioning, and Snapshots / FlexClones, the effective size of your data cube has just -- dramatically -- increased. And all without hitting performance.

It takes a few %age points of the total disk space to enable this truly smart technology.

Worth it? You bet. NetApp customers would find it really hard to go back to the dark ages of storage. Snapshots are so natural for NetApp users that many don't think about this extra dimension, and those unfamiliar with virtualized storage often fail to grasp the difference between NetApp systems and a traditional SAN.

What's Not Shown; Deduplication

All the savings above are effected by non duplication technologies. Not shown are the further savings to be made with deduplication. That increases the effective load factor further; by how much I can't tell, as deduplication depends on the data, and I don't have deduplication statistics in the set I'm working with.

But it's substantial; with VMware for example, deduplication can show 80 to 95% space savings.

Dr Dedupe

If deduplication on primary storage is of interest, go visit my colleague Dr Dedupe's blog for more insight into this technology. Interestingly, he was rated in the top 10 of most valuable vendor storage blogs over at Storage Monkeys.

I (of course!) was at #11, and not in the list. Next year's Oscars, perhaps.

Sunny at ShadeOfBlue Towers

Apologies for the lateness of the this part of this series. The weather was so good this weekend, I decided that a few days in the sun at the weekend took top priority. We see the sun so infrequently here.

Space is Mind Bogglingly Big

How Big Exactly?

Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemists, but that's just peanuts to space.

 

Douglas Adams, The Hitchhiker's Guide to the Galaxy

For transatlantic readers; a chemists is a drugstore.

If that were only true for disk space. Unfortunately, it's a lot more finite than the universe, and there's only so much data you can pack on disk drives.

But how much, exactly?

The tail end of EMC's recent NetApp FUD attack deserves an answer, and I'm with W Curtis Preston on this one; it's unedifying when vendors resort to misrepresenting a competitor's technology.

In this second part of my analysis on NetApp's usable capacity I'm going to use data from NetApp systems in the field, not anecdotes and speculation.

Part one is here and it covers the source of the data I'll be using. Here, I'm covering the nature of the data.

The Bottom Line

See below for the gory details of the data. Let's cut to the chase. The average %age of space that a storage admin can see, across all these systems, all 7597 of them, was...

Usable Capacity of 66.27%

Just to emphasize; it's an average across a large number of systems, 7597 in total.

There are systems with low utilisation; particularly newer systems that are at the start of their lives. Older systems have higher percentages.

More pertinent; not shown are space saving technologies like thin provisioning and deduplication. These are technologies that can save huge amounts of space by virtualising the storage. But that's for a separate post.

Comments and questions welcome.

The Fine Print

Full disclosure verbiage. Here's the detail on the data that makes up the chart.

Data Anonymity

Here's what I can't disclose.

• The names of the customers. That would breach confidentiality.
• The number of customers from which the sample is drawn. That's commercially sensitive. I can say that they are a mix of customers across the commercial spectrum, and representative of the customers we have here at NetApp.
• Specific and individual system detail. It's unlikely, but someone may be able to identify a system from the data presented, so no line items. 

In effect, I'm anonymising the data. Actually, I'm treating the data the way I'd want my personal details to be used; for instance, how much I owe on my credit cards. I don't want my data disclosed, as in this instance of hugely damaging data loss. On the other hand, I suspect I'm somewhere in this set of summarised numbers. (My, they're frightening...)

Base and Rightsizing

We need clarity on the base used and the addressable space.

What's a 1TB to you?

To disk drive manufacturers, it's a decimal number; base 10. That 1TB disk drive you bought has 1000*1000*1000 bytes.

Not 1TB as in 1024*1024*1024 bytes. In base 2 that's a number that's 7% bigger (you do the math!). Or, put another way, 1TB base 10 is around 7% smaller than a 1TB base 2.

And not all those bytes on that monster disk are addressable. In fact, your 1TB (decimal) drive may only have a maximum of 977GB (decimal) that can be read or written to. That's another 3% discrepancy.

The discussion on base and addressability, to be absolutely clear, is not what is referred to as rightsizing. Every disk manufacturer has a real capacity that's different from manufacturer to manufacturer, and sometimes even drive model to drive model, for the same "marketing" number. NetApp, like many companies, reduces the size of the drive to account for this by using the smallest figure across types that we use, to maintain compatibility when drive models are mixed in an array.

An example of the misunderstanding that bases can bring is in this EMC blogger's post on the subject. Failing to account for base 2, the (very primitive) calculations used are 10% out. That's a big discrepancy.

I won't be counting disks (unlike EMC), because it leads to significant inaccuracies when discussing percentages that are made up from different based absolute numbers.  

The data I have is base 10. 1TB is 10^9 bytes, and 1PB is 10^12 bytes.

Data in Summary

The data was collected on 9th September 2008, from 7597 systems with a total of 216.95PB of connected storage. 

Eliminated data: two classes of system have been eliminated from the data, and aren't counted twoards the 7597 system total.

• A couple of hundred NetCache systems; they're not primary or secondary SAN or NAS devices (although the stats don't change much if they are included)
• Any V Series systems. They use the underlying SAN RAID level, and present data as RAID-0. (Incidentally, that pushes the averages up, not down.)

Systems and Data ONTAP versions: all the way from a lowly F87 (remember it?) up to the top of the range FAS6080. The data also includes ONTAP versions 5, 6 and 7, FlexVols and TradVols, and a mix of RAID-4 and RAID-DP.

Things That Don't Matter

Because of the unique architecture of NetApp systems, some things really don't matter. SAN or NAS; who cares? The data used contains a mixture of both. NetApp's Dave Hitz summarised it best;

At the protocol level, block-reads over NFS and block-reads over iSCSI are almost identical. The main difference is that NFS asks for a certain number of bytes, starting at a given byte offset, and iSCSI asks for a certain number of blocks, starting at a given block offset. With NFS you must divide by 512 to convert bytes to blocks. So what! The other difference is that iSCSI and FC-SAN use a LUN to identify the container holding the blocks, and NFS uses a file handle. I’m over-simplifying, but you get the point. At the protocol level, for block traffic, there is almost no difference between NFS and iSCSI or SAN...

...A LUN and a file are both just containers that hold blocks of data, so get over it. It was this similarity that originally convinced me NetApp could “unify” SAN, NAS and iSCSI into a single storage appliance. (This is the key idea behind unified storage.)

Information Not In The Data

Disk types (FC, SAS and SATA) and sizes. Again, this is a broad mix of systems; splitting them out by size or technology might show a difference, but this isn't really the point.

Again, comments and questions welcome.

.

3PAR and Bistromathics

A proud 3PAR introduced their new T series at the beginning of September. If you're into drag racing, they showed a really nice set of numbers in their latest SPC-1 speed test. 

3PAR quote quite a few of them in their press release, and in a breathless summary, declared that they had been achieved... 

"...with 83% capacity utilization and without complex configuration or performance tuning"

Great utilization too! And here's the SPC-1 disclosure (the devil is in the detail, believe me, so I always check) that explains; Data Protection Level: Mirroring

OK, so it's not a complex configuration (EMC supports it too, so it really can't be that difficult), and although mirroring might be considered a performance tuning advantage -- hey, use RAID-5 if the box is that good! -- I'll let it pass.

Anyone see the problem here? 83% capacity utilisation? And with all the disks mirrored?

200% Surprised

How do you get 83% utilization when everything is mirrored? Heck, how do you get anywhere near 50%? 

Eventually, I got the answer.

It's Bistromathics

The late Douglas Adams summarised his new found principle best.

"Bistromathics itself is simply a revolutionary new way of understanding the behaviour of numbers.

"Just as Albert Einstein's general relativity theory observed that space was not an absolute but depended on the observer's movement in time, and that time was not an absolute, but depended on the observer's movement in space, so it is now realized that numbers are not absolute, but depend on the observer's movement in restaurants."

Now we can add to the list. Bistromathics applies not just in restaurants, but 3PAR press releases too. 

Bistro Manager Explains

So, 3PAR's newly hired Marc Farley came and helped out a puzzled NetApp diner. He was kind enough to explain the complexities of the 3PAR bill of fare. 

• [Puzzled Diner]
  Help me understand something here though. The SPC-1 disclosure report states you used "mirroring" as the data protection mechanism. That implies RAID 1 or 10. If so, how can you possibly claim more than 50% utilization?
  
• [Bistro Manager] 
  Yes we used mirroring. The utilization is based on used capacity as a percentage of the volume's exported capacity. The ratio of used capacity to raw available capacity is 44%.

Voila! 83% is actually 44%!

Good on you, Marc. Your answer is the one that I was expecting; less than 50%. Straight answer.

But as for the press release, I don't understand how claiming 83% capacity utilization for this benchmark -- with no explanation -- is anywhere near honest.

Most people will assume that they got this with much less performant RAID-5 given the >50% capacity utilization. 

Readers of the press release are going to be  

• impressed and wrong in their impression (83% of readers)
• confused like me (the remaining 44%)

My advice? When you next get a bill from 3PAR for anything, whether it be storage or a 3 course meal, query it. Every line item. Then get Marc Farley to check it over one last time for you. 

He'll make sure you don't pay twice.

.