NetApp - The Missing Shade of Blue

The Devil is in the Detail

I am fascinated by mathematics in a “wow, that’s a cool result, how’d they work that out?” sort of way. One of my favourite blogs is Good Math, Bad Math, to which I’m drawn like a moth. The guy is brilliant, but sometimes I’m just more than a little sorry for the dismembered victims of his razor sharp mind. Great spectator sport, but it could be me…

So when I spotted EMC blogger Scott Waterhouse using statistics from an IBM paper on RAID-5 vs RAID-6 reliability, first thought was; hey, that’s wrong! But I’m no whizz with the numbers, so I thought I’d employ the biggest brain we have in NetApp on the subject; one of the authors of the original NetApp paper, Jon Elerath. One of the sharpest knives in the drawer.

Over to Jon:

Keep in mind that the model’s 6% number depends on the scrub rate (how long it takes to correct data corrupted from media issues). I assumed 168 hours (1 week). If we or EMC uses half  that (say, 84 hours), the probability of a RAID group failing in 5 years drops to about half that, or 3%.

The original data was from 2006, but currently we are seeing much lower failure rates for operational failures. It’s a figure of about half of what I saw back then.

These two reductions give a number in the order of a quarter of 6%, or 1.5%.

Also remember that as data is read, data corrupted by media defects can be corrected by the reader (say, the OS), thereby eliminating the latent defect. So, on an absolute basis, the real number of failures depends on inherent HDD operational failure rates, the scrub rate and how smart the OS is in dealing with latent defects, something that varies amongst competitive systems.

And as I pointed out before when this all started, RAID-VTL isn’t like other RAID-4 or RAID-5. The RAID group gets shut down until the rebuild  -- just on written data -- is done, reducing the risk even further.

Jon again:

But that wasn’t really the point of the model I developed. The real importance was to compare the relative values of single parity RAID to dual parity. Using the same distributions for both models shows that RAID-DP has a lower probability of failure, by about 4000-5000 times.

If Scott is even half way to the truth with his claim that NetApp’s VTL (now with deduplication) is 100% likely to fail in a five year period (although he’s corrected the more obvious errors now), then here’s the kicker.

Every EMC CLARiiON or Symmetrix system implemented with RAID5 is more at risk.

Serious stuff, Scott. I can only suggest getting StorageZilla to bless RAID-6 for Tier 1 Oracle applications if I were you. Perhaps not. Well, at least RAID-10, and then we can discuss the NetApp Space Guarantee and really make a meal of it.

.

Finding a Pair of Socks

One of the common pieces of FUD (Fear Uncertainty and Doubt) that gets sent by colleagues to my inbox at ShadeOfBlue Towers is that everyone is saying NetApp FAS systems get slower as they fill up.

Some vendors have even taken to pointing our customers to this NetApp document as proof. Help me! Is it true?

Here’s a graph from the paper. (A point here; read the full paper for the details, but the lines to look at are the blue to 50%, followed by the green from 50% to 100%. And take a look at Kostadis’ blog for more detail on our WAFL technology.)

Look at the equivalently spec’ed EMC system below the NetApp line. Go ask other vendors for their graphs of performance at specific capacity points too; I suspect the silence will be deafening, or you might get lucky and get back a remarkably similar chart.

Is the FUD point made? No, because every storage system gets slower as it fills up. And some just start bad, and then get worse.

So why does increasing capacity utilization affect performance? Here’s an analogy; it’s like me trying to find a decent pair of matching socks (preferably without holes) in my sock drawer. The more socks, the longer it takes me to find just the right pair.

But with something that understood my sock needs, it needn’t take quite as long. So how do we make sure that performance stays high as the system fills? We’ve worked hard on this problem, and we’ve got some cool technology that helps.

Here’s one; the NetApp Sock Handler.

Readsets

Prefetching sounds pretty easy. For instance, in a file based NAS system (if it’s smart enough), when you start reading a file, the likelihood is that you want to read all of it.

For block based though, the read patterns can’t be ascertained the same way, as there’s no semantic information as in the file case that can help with the prefetch pattern.

For all read operations, large amounts of cache is employed in a NetApp system for staging pre-fetched data. We uses an adaptive read history logging system based on “readsets,” which provide much better performance than traditional and fixed read-ahead schemes.

In fact, multiple readsets, can support caching for individual files (for NAS) or LUNs (for SAN), which means that multiple read streams can be pre-fetched simultaneously. The number of readsets per file or LUN object is related to the frequency of access and the size of the object.

The system can adaptively select an optimized read-ahead size for each read stream based on historical factors, such as

• the number of read requests processed in the read stream
• the amount of host-requested data in the read stream
• a read access style associated with the read stream
• forward and backward reading
• identifying coalesced and fuzzy sequences of arbitrary read access patterns

Cache management is helped by algorithms that determine when to perform read-ahead operations, and how long each read stream's data should be retained in cache.

Readsets are just one of the technologies that ensures NetApp systems stay as fast as possible as the system fills.

It doesn’t help with my sock drawer, unfortunately, but it might beat the FUD off.

.

Pillar of Weakness

Over at the Register last week, Chris Mellor was predicting doom and gloom for the storage industry, with the observation that, just as global warming is affecting our environment, the interconnectedness of everything in business will affect sales of disk storage.

It appears that Pillar Data Systems is the latest victim, with an October 29th report  (yesterday) that Pillar laid off 150 employees, or some 30% of the workforce.

That’s not good, and I feel for all those being dismissed in this downturn. I wish them all the best.

These are hard and unforgiving times for people and the companies they work for. And if a storage vendor wasn’t making a profit before the current credit crunch, they’ll struggle to make one now. The survivors in the storage industry will be those that can keep their cashflow positive. Whatever the figures for petabytes shipped show, up or down, the secret to survival for any company is making everything sold contribute positively to the bottom line.

I am interested in knowing, vendor or user, what you are doing to survive, and what you’re seeing out there. Dave Hitz recently asked the same question. With the layoffs at Pillar, the world’s changing fast, that’s for sure.

Comments here or on Dave’s blog are welcome.

---

Photo: Creative Commons Attribution 3.0 Unported licence, by Tomas Castelazo

.

When Slow Means Unavailable

Many years ago, in a land far away (actually, it was near Watford in the 1980s) I was what is sometimes called by the purveyor of the mainframe systems we used, a customer.

• I take your money in return for X; you’re my valued customer.
• I give you money in return for X; you’re a thieving bandit.

I just changed sides, I suppose.

Actually, I had customers too; internal customers, to whom my colleagues and I provided a service, for money, so I suppose your perspective of customer or  user depends on which side of the transaction you’re on.

Thin Provisioning Headaches

StorageBod is obviously suffering from the same problems of being a consumer and provider of services. He notes, a little wistfully, that thin provisioning has upsides for him (more space for less money) and downsides for his customers (no service for too much money).

“Our colleagues in the Server and Network disciplines have it easier than us; if they run out of capacity, pretty much most of the time, things just run a little bit slowly...things stop if we run out of capacity.”

Ah, if that were only true.

One of the first studies I ever did (and I’ve lost the paper – we didn’t do electronic copies in the Dark Ages of computing) was to try and correlate user perceptions of service with some meaningful metrics, in particular response time.

I’ll spare you the details (actually, I’m so old I’ve forgotten them), so here’s the gist of it.

• If the service got slower by more than 100%, then users changed their opinion from “it’s slow” to “it’s unavailable”.

It didn’t matter what the actual response time was; it was the change that was important. 

We had some systems that delivered sub-second OLTP transactions; if it doubled, it was useless. We had other systems that had average response times in the 2-3 second range. That was OK. But if it doubled, it was useless.

The thing that gets you every time is inconsistency. So when the server and network start running slowly, users will declare the service unavailable.

What I learned from this all these years ago was a valuable lesson.

• Don’t overdeliver. Make the service run slowly, and improve if you can. Degrading to your SLAs is worse than improving towards them.
• An outage is preferable to no outage and continued slow running. If you can fix your performance problems, but the service needs to come down to do it, then take the outage.

Your published SLA is meaningless. It counts for squat diddley, because the only SLA you’re being measured on is the historical service level you used to deliver. To your users, if your service starts grinding exceeding slow, you’re just another masked bandit.

I think StorageBod has got the better side of the deal, to be honest.

.

The Fallacy of Evolutionary Ascent

Picture from Wikipedia

Perhaps one of the commonest mistakes made when studying evolutionary biology for the first time is to assume that evolution has some directed goal; that the end-point, present day version is somehow “better” than what came before it, because it was designed to be superior than what came before. This mistaken reading is encapsulated in slithering beast to modern man, parodied here in true Victorian style.

That’s a complete misreading of evolutionary biology, which has no directed goal, and no designer. Variation and the pressures of natural selection don’t make for a “more superior” present day species (however you might define superior), but for a better fit to the environment.

It’s about surviving long enough to pass on your genes.

Enter the Platypus

I was very amused by Stroragebod’s analogy of WAFL (the software that underlies the layout of data on disks in a NetApp system) to the structure of a platypus. It was echoed elsewhere, and I think the humour is great, but perhaps the thinking behind it is flawed.

There’s an assumption that because monotremes (family Protheria) appear undifferentiated in their developmental history, retaining features of both egg-laying and placental live-birth mammals (the Eutheria), that they are somehow a cross between one thing and another, a sort of living fossil at the point where the two lines of mammals diverged.

Is the implication is that that these features make us in some way unsuited because we appear unspecialized? Perhaps. Or perhaps because our Victorian forebears had difficulty placing the platypus, that WAFL is in some way a fake?

Adapt or Die

The truth is a little more prosaic. Individuals survive to spread their genes because they are better fitted, or suited, to their environment.  Those individuals that die before they’re able to reproduce sometimes leave fossils, but have no genetic contribution to the present. Only those that survive populate the future.

The same goes for storage systems, or UNIX, or any other number of technological changes. Although the means of reproduction are different (PCs don’t gather in large mating groups to generate more PC offspring, or so I understand), the transmission of successful “genetic material” is by the same pressure that works in the natural world.

Variation and selection.

You only get to pass on your genes if you fit your environment and survive, That’s why WAFL is still here, and its mutated and fitter offspring from its first appearance in the computing fossil record back in 1993 are alive and well, passing on their genes in increasing numbers in the present day.

So I’m OK with WAFL-as-platypus. It’s thriving. Unlike the huge number of extinct storage dodos this industry has seen.

.

Benchmarking Thin Provisioned Systems

Marc Farley of 3PAR is providing some really excellent blog fodder for me over here at ShadeOfBlue Towers, and again – true to form – he’s done so on my second favourite subject.

Benchmarking. This time, lamenting the fact that there are no benchmarks with thin provisioning turned on.

Wait a moment. That’s exactly how NetApp does its benchmarks. FlexVols are the key here. The difference between a thin provisioned FlexVol volume and a full fat thick version?

• Thin: unused space comes from a common pool of free blocks, but the pool can be over-committed.
• Thick: unused space comes from a common pool of free blocks, but the reservation in the pool is guaranteed.

There’s no difference in the way the used space is handled. Thin or thick, a FlexVol is a FlexVol.

As these SPC-1 benchmarks show. With the space guarantees turned off (see page 67 for the vol create command we used), we were using the full WAFL thin provisioning infrastructure. Oh, and we did it with snapshots turned on too.

.

Hole Punching File Systems

In my last blog entry, I talked of the problem with file systems built over block attached storage, and the undesirable side effects that has on thin provisioning. A (much simplified) diagram is in order to demonstrate.

Here you can see the groups of blocks where hole punching would be beneficial in releasing up space that the file system isn’t using, but the block based storage system thinks is still in use. 

The solution to the problem can be tackled two ways.

“Crack” the Filesystem

By cracking I mean reverse engineering the file system. that way, we could do the free block reclamation inside the storage system.

For something like NTFS, where the data structures are proprietary and not officially documented, that’s hard; it took the Linux NTFS-3G people years to get to a stable system that could read NTFS, and a great deal more effort to make it writeable. To be frank, if Microsoft change versions with their next incarnation of Windows, it’s a wasted effort.

Plus, there are plenty of file systems that have the same problem of high water marks with holes of free space below it. Doing this job for every file system out there would be an exercise in futility.

Use the Host

NetApp’s approach was to put the functionality in SnapDrive for Windows, which runs on the host OS. Storage managed by SnapDrive logically appears to come from a locally attached storage subsystem. In reality, the capacity comes from the NetApp system.

One advantage of this is that it allows us to use interfaces to the Windows API (specifically by becoming part of the device driver layer and using the IOCTL functions) to watch for file system changes on the host, and inform the NetApp system of these changes via new and additional SCSI commands.

Because NetApp systems use WAFL (which Kostadis is covering in some detail over on his blog [update 22Oct2008: new link to the whole series]), the blocks of storage identified by SnapDrive as no longer in use can be returned to the free pool. The OS is completely unaware of this interaction.

Future Enhancements

Right now, NetApp only supports Windows for hole punching. Future enhancements to hole punching depend on some form of industry co-operation, and NetApp have submitted a recommendation to the SCSI T10 Standards Committee much along these lines; extend the file system and OS to add new SCSI commands that can identify when holes appear in the file system. That way, everyone gets to benefit from thin provisioning and space reclamation.

If…

Caveat Emptor

Let the buyer beware. There are a few requirements that your SAN storage will need to make this happen.

• Storage virtualization. Systems that only thin provision above the high water mark, and are forced to maintain a one-to-one block relationship between the LUN and what they present to the OS, won’t be able to do this.
• Granularity of blocks. NetApp uses a 4K unit of management, small enough to make hole punching worthwhile. If your SAN of choice uses a larger unit of management – some are enormous, like Dell/EqualLogic’s 15MB page – then you’re only going to see the benefit for some pretty big holes. Foamy, frothy file systems will just stay as big as they are today.
• Unless it’s NetApp, NAS may not be the answer. If your NAS server is backed off by a SAN, that may not be enough to solve the problem; the key lies in the SAN and the way it thin provisions LUNs. NetApp NAS systems don’t have this problem because they use CIFS over WAFL; deleted files are reclaimed automatically into the spare block pool.
• Innovation. It takes good quality software to implement space saving schemes like this. It’s been made much easier for NetApp because of the innovative nature of WAFL.

NetApp’s hole punching. Ask for it by name.

.

Really Thin Provisioning

We’re all guilty of over-estimating the amount of storage we require; sometimes it’s as a result of decisions made by using a micrometer, crayon and ax approach, and sometimes it’s just wanting that safety net. No one likes running short, but running fat is just too expensive.

Marc Farley over at 3Par makes the excellent point that storage resource utilization is high on everybody's list as a major problem to overcome.Just about everyone is in agreement, vendors and users alike.

Thin provisioning helps fix that by making available space look much bigger than it really is. With the right provisioning tools and management in place, you can relax and be fat and happy while really being thin.

So I applaud the recent announcement from Symantec to join NetApp in the efforts on thin provisioning. NetApp has been a leader in this area for a long time; since 2004 when it offered the industries first thinly provisioned storage device. Over the years we’ve refined it to the point where we’re able to recommend thin provisioning for just about every class of storage out there.

In January of this year, NetApp proposed to the SCSI T10 Standards Committee the creation of industry standards for thin provisioning and space reclamation. The standardization of this technology will enable much broader use as well common tools and capabilities across multiple environments.

So what problem does this proposal fix?

Filesystems and Swiss Cheese

Not all thin works the same way, because of the underlying file system. Look for the caveats on thin provisioning in your storage supplier’s literature that goes along these lines;

“Take care when thin provisioning NTFS LUNs. It is best suited to data with a low rate of change.” 

It’s because NTFS doesn’t reuse the space it frees up when a file is deleted. It keeps on writing new data right up to the advertised size of the LUN before going back to use the reclaimed and unused space. It’s like Swiss cheese; lots of holes with fresh air surrounded by some cheese.

Most thin provisioning schemes can’t handle this, because they don’t know that the holes are really free space; a block LUN has no knowledge of files, and more specifically, files that become holes when they’re deleted. Thin provision an active NTFS volume, and it’ll get fat pretty quickly.

So how do you take advantage of thin provisioning with a high rate of change?

NetApp Solution 1: Avoid the Problem, Use NAS

Instead of using a SAN to provide an NTFS volume, you can avoid the problem with a NetApp system by using it as network attached storage. Here you can get rid of a server, thin provision, and give clients direct access through shares to a self managing file system. Using CIFS, the NetApp system is in charge of the files, and knows when they’re deleted or modified. Thin provisioning file share space over CIFS on a NetApp system means you don’t have to worry about holes; because they aren’t there.

Not all systems can do this, even though they may claim to be unified storage systems. Other NAS solutions with no space reclamation communication between the NAS head and the SAN-provided LUN can’t avoid the file sized holes.

NetApp Solution 2: Hole Punching

If you want to have an NTFS LUN presented over iSCSI or Fiber Channel, then we have the solution that lets you thin provision NTFS properly; what we call hole punching. This technology already exists in NetApp systems and implements in part the proposals we’ve made to the SCSI T10 Standards Committee.

Part 2 follows shortly; the proposal, and how NetApp’s hole punching works.

.

3PAR’s Oscar Wilde

Oscar Wilde in pensive mood after hearing of
NetApp’s VMware 50% Space Guarantee 

Over at 3PAR, Marc Farley is briving (that’s blogging while driving) on the subject of funny guarantees.

Because 3PAR doesn’t have any serious guarantees of its own. Ditto Dell and EMC.

Marc reminded me of Oscar Wilde.

On hearing a remark that he thought profound, Oscar turned to his friend Whistler and said; “I wish I had said that”. To which Whistler replied;

“You will, Oscar, you will.”

You will, Marc, you will.

.

Lumpy Dell and Benchmark FUD

Edward "Lumpy" Stevens was a cricketer from the time when bowling was performed underarm or underhand, rather than the modern technique of roundarm bowling. Lumpy was famous for his ability to find lumps and bumps in the pitch, and consequently his bowling confused as the ball careered erratically towards the batsman.

For Honest Lumpy did allow
He ne'er would pitch but o'er a brow

Nowadays, cricket is a much more sophisticated game. Pitches are flatter, bowlers are faster, and umpires are on the watch for ball tampering and other activities that give the bowler an advantage. And underarm bowling is not part of the modern game of cricket.

So it’s time for the umpire to send Lumpy Dell back to the pavilion.

Dell Bowls Underarm

Every day, I get concerned sales and systems engineers emailing me about some document that our customers have been given by the competition.

“I attach a new document that Dell/EqualLogic is handing out to customers. Can you put together a response?”

The document is entitled

“Guide to comparing storage performance efficiency for business applications by leveraging Microsoft sizing tests”

In case anyone from Dell is reading this and needs a visual reminder, here’s the front page.

The contents are as long as the title, at 17 pages. But what is it about, and why would a sales colleague be looking for help? Dell’s words;

Microsoft wants to ensure systems are properly configured for running Exchange. The Exchange Solution Reviewed Program (ESRP) allows vendors to validate to Microsoft’s satisfaction sizing of storage performance and capacity for Exchange.

This, to be clear, is not one of Dell’s ESRP submissions; there’s nothing wrong them. This is a separate document that Dell give their customers.

The conclusions are going to be blindingly obvious, but I’ll repeat them anyway;

Microsoft-approved results of each vendor’s self-testing illustrate several advantages for EqualLogic:…

Comparisons Galore

It’s about everyone else’s ESRP submissions. This massive piece of marketing FUD directly compares and contrasts ESRP results from just about every vendor that’s ever published one.

EMC, HP, IBM, LeftHand, NetApp, Hitachi. In dense tables with helpful explanatory footnotes, the document details the lot. Dell again;

• IOPS per disk for most vendors
• Backup and log replay performance for most vendors
• Example results of each major vendor’s primary native SAN testing
• Examples of tier-2 SAN offerings

Then a point by point analysis of each submission and its shortcomings when compared to Dell/EqualLogic.

What a tragic waste of effort.

ESRP Is Not a Benchmark

Cheekily, Dell/EqualLogic’s FUD department have included a handy section called

Locating the key data inside ESRP reports

It is, of course, missing the one vital piece of information. Pick up any ESRP submission from any vendor; here’s one of Dell/EqualLogic’s. They all have the same boilerplate text. So pick up the link pay close attention to page 18. Here’s what it says;

The ESRP Storage program is not designed to be a benchmarking program. Its tests are not designed for achieving the maximum throughput for a given solution. Rather, they are focused on producing recommendations from vendors for the Exchange application. Therefore, the data presented in this document should not be used for direct comparisons among the solutions.

It’s Not Cricket

It might be a good idea for Lumpy Dell to re-read the rule book. No-one wants to be publically outed as underhand.

[updated 10 Oct; Skip (see comments) pointed out that this blog wasn’t very clear on which document I was criticizing. I’ve made clarifications.]