NetApp - The Missing Shade of Blue

NetApp's $1Million Essay

I don't get paid by the word for what I write, and you probably don't either. Here's a rewarding way to change all that; pick up your pen and in 500 words or so you could be getting the equivalent of $2000 a word.

Yes, it's back; NetApp's $1 million dollar virtualization challenge. (You might remember the original challenge, and what we saw as a need to call out the exceptional advantages of NetApp's guarantee.)

And we're not looking for a Shakespeare.

In no fewer than 500 words, describe your current storage infrastructure and how it supports your VMware environment. Please include as many details as possible to differentiate your submission, such as: number of servers and VMs deployed, type of current storage connected to your virtual servers, your desired business metrics, ROI and timeframe for a new storage deployment in your virtualized environment.

My colleague Vaughn Stewart has made a couple of observations on his latest blog about the program, which is based on our virtualization guarantee. Vaughn makes a point about RAID-10 vs RAID-DP in the guarantee;

As many of you may know, this program has never been without its share of criticism by other storage vendors. In my opinion, I believe this criticism has been fair as a component of the guarantee program compares RAID-10 and RAID-DP on the merits of storage efficiency. While both technologies are near equals in terms of performance and data protection it is unfair to compare the 50% utilization of RAID-10 to the 87.5% provided by RAID-DP

I beg to differ (and did so in my original blog). I believe it's valid to compare the two, because the alternative -- RAID-5 -- just isn't up to the job of protecting your data.

I said back then; so why not compare to RAID-6?  The truth is, we could and would, but we still don’t find our competition selling it. It’s there on the spec sheet as a solution from some vendors – not all have the capability -- but it’s rarely put forward as a viable solution.

And that's the extent of our disagreement. The rest of his blog is spot on; and with public disclosure of the results, I too think we'll see high rates of savings without including RAID-DP.

Get scribbling!

.

Add Lightness

My regular readers (5 or so) will now know that I'm the proud possessor of a Caterham 7, a direct descendant of Colin Chapman's iconic flying machine, the 50 year old Lotus 7.

Constructed from a tubular space frame chassis, clad in as little bodywork as is possible, and with small and light engines, the fundamentals of a car that was said to be “too fast to race” hasn’t changed in half a century. Chapman held that:

"Adding power makes you faster on the straights. Subtracting weight makes you faster everywhere."

He was one of the great engineering geniuses of the race and motor industry.

While doing a bit of research on cloud architectures, I stumbled across an email archived on an internal engineering wiki, sent by Dave Hitz way back in 1997. The discussion was about adding a specific feature to a NetApp product, and the upsides and the downsides of doing so. (I've edited Dave's missive for brevity.)

Hey -- I love philosophical controversies, so I'm going to dive right in!

... One of the most important questions you can ask about an appliance is: Should we add this feature?

... The potential drawbacks include:

• The feature decreases one or more of "fast, simple, reliable."
• Testing the feature (in every new release FOREVER) slows down our release cycle and hurts time-to-market.
• Documenting the new feature makes the manual thicker and harder to use.
• The feature has disadvantages that are hard to explain, and customers who try to use the feature may get burned.
• Very few customers actually need the feature, reducing benefit.
• The feature is expensive to implement...

So how are my little car's design and Hitz's email connected?

I'm not a engineer of either motor or software variety, but I see the similarities. What Hitz is identifying are the same engineering principles as espoused by Chapman, applied here to NetApp's software. As Einstein said (yes, it's rent-a-quote week here at ShadeOfBlue Towers);

"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction."

When I look around at the creeping featuritis that rival storage systems employ, the scale and size of the moving parts, and the horrendous complexity of the solutions they sell, I'm sure that the competition are employing intelligent fools. They certainly show little appetite to move in the opposite direction. I'll not bore you with examples, as I'm sure you know who they are and what systems and solutions they sell.

That's what struck me about NetApp when I joined NetApp some 4 or so years ago, and strikes me now. Our solutions are no more complex than they need to be; fast, simple, reliable are still the hallmarks.

But there is one item I'd like to add to Dave's email; I believe there's an extra quality that shines thorough in all our solutions and proposals. Colin Chapman, in possibly his most famous quote, said; 

"Simplify. Then add lightness."

That's the feature I think we have in abundance; added lightness. So when you're buying storage solutions -- heck, if you're buying anything -- keep Dave's list in mind for your next RFP/RFQ/RFI, And then add in this; 

In the proposed solution, please identify where you added lightness.

---

Update 8May: Steve Klinkner has an excellent series of posts on adding lightness. I've just read them (sorry Steve, I sometimes don't have enough time to keep up!), and this is one of the proofs that we take this "fast, simple, reliable" goal very seriously indeed. Well worth a read to see the kind of thinking that goes into the development of our software.

.

Hairy Canary over SSD with 64 Bits of NFS

Chuck’s latest hairy canary against NetApp deserves correction. I thought about commenting on his blog, but there are so many points to make, I decided to address his “complaint” in a little more depth over here.

64 Bit – Over A Decade Ago

What’s the deal with x64 multicore technology? Just what might qualify NetApp to "lead the 64 bit discussion"? Here’s what.

Data ONTAP is 64 bit and endian clean. In fact, we were running on 64 bit systems with the NetApp F540 in 1996.

With the CX4 just recently getting a 64 bit OS – October 2008, just this year – EMC is playing catch up; they’re over a decade behind.

That’s why NetApp, and not EMC, is leading.

Solid State Storage

NetApp are getting criticized by Chuck (again!), this time for making public our SSD roadmap. A roadmap that’s freely available to every customer of NetApp could be considered more of a public commitment than those given under NDA (Non-Disclosure Agreement). Where’s the downside? Do customers want to know or not? I think they do.

That leads on to another Chuck complaint. Just what might qualify NetApp to "lead the SSD discussion"? Here’s what.

Dr. Jon Elerath, reliability engineer at NetApp, will represent the International Disk Drive Equipment and Materials Association (IDEMA) and present at the JEDEC Board of Directors Meeting on November 3, 2008. His presentation is titled “How IDEMA Can Help the Development of Reliability Standards for Solid-State Storage.”

Elerath is the chairman of the IDEMA Reliability Standards Committee, which is comprised of members from all the top storage producers (including Seagate, Hitachi GST, Western Digital, SanDisk, and STEC) and users (including NetApp, HP, Sun, Intel, SanDisk, IBM, and SNIA).

We did much the same in May 2008 too.

And NetApp is a Founding Member of the SNIA Solid State Storage Initiative. Val Bercovici, Office of the CTO, NetApp, was also elected as the inaugural vice-chair of the SSSI‘s Governing Board.

That’s why NetApp, and not EMC, is leading.

NFS Industry Leadership

Capping it off, Chuck complains about our leadership of pNFS. Just what might qualify NetApp to "lead the NFS discussion"? Here’s what.

NetApp has extensive participation in the IETF, particularly with regard to NFSv4. More specifically NetApp representatives participate as:

• Co-chair the NFSv4 working group
• Co-author of the NFSv4 specification
• Document editors for the NFSv4.1 specification
• Spec writers for NFSv4.1 (including pNFS)

Brian Pawlowski, NetApp’s CTO, co-authored RFC1813 (the NFSv3 specification) in 1994 and is co-chair of the NFSv4 working group.

Michael Eisler. Co-authored RFC3530 (the NFSv4 specification) along with Dave Noveck, another NetApp employee, and Michael is currently an editor for the NFSv4.1 specification. Take a look at his blog and his biography; it’s wall to wall pNFS.

Trond Myklebust of NetApp is the Linux NFS maintainer (including pNFS), with a hugely significant contribution over the years,

I’ve probably missed out several other major NetApp contributions and contributors to NFS. My apologies to everyone; there’s so much that we do.

NFS has been a pretty pivotal standard for the industry for decades. NetApp is deeply committed to it, and has been for over 14 years.

That’s why NetApp, and not EMC, is leading.

.

Finding a Pair of Socks

One of the common pieces of FUD (Fear Uncertainty and Doubt) that gets sent by colleagues to my inbox at ShadeOfBlue Towers is that everyone is saying NetApp FAS systems get slower as they fill up.

Some vendors have even taken to pointing our customers to this NetApp document as proof. Help me! Is it true?

Here’s a graph from the paper. (A point here; read the full paper for the details, but the lines to look at are the blue to 50%, followed by the green from 50% to 100%. And take a look at Kostadis’ blog for more detail on our WAFL technology.)

Look at the equivalently spec’ed EMC system below the NetApp line. Go ask other vendors for their graphs of performance at specific capacity points too; I suspect the silence will be deafening, or you might get lucky and get back a remarkably similar chart.

Is the FUD point made? No, because every storage system gets slower as it fills up. And some just start bad, and then get worse.

So why does increasing capacity utilization affect performance? Here’s an analogy; it’s like me trying to find a decent pair of matching socks (preferably without holes) in my sock drawer. The more socks, the longer it takes me to find just the right pair.

But with something that understood my sock needs, it needn’t take quite as long. So how do we make sure that performance stays high as the system fills? We’ve worked hard on this problem, and we’ve got some cool technology that helps.

Here’s one; the NetApp Sock Handler.

Readsets

Prefetching sounds pretty easy. For instance, in a file based NAS system (if it’s smart enough), when you start reading a file, the likelihood is that you want to read all of it.

For block based though, the read patterns can’t be ascertained the same way, as there’s no semantic information as in the file case that can help with the prefetch pattern.

For all read operations, large amounts of cache is employed in a NetApp system for staging pre-fetched data. We uses an adaptive read history logging system based on “readsets,” which provide much better performance than traditional and fixed read-ahead schemes.

In fact, multiple readsets, can support caching for individual files (for NAS) or LUNs (for SAN), which means that multiple read streams can be pre-fetched simultaneously. The number of readsets per file or LUN object is related to the frequency of access and the size of the object.

The system can adaptively select an optimized read-ahead size for each read stream based on historical factors, such as

• the number of read requests processed in the read stream
• the amount of host-requested data in the read stream
• a read access style associated with the read stream
• forward and backward reading
• identifying coalesced and fuzzy sequences of arbitrary read access patterns

Cache management is helped by algorithms that determine when to perform read-ahead operations, and how long each read stream's data should be retained in cache.

Readsets are just one of the technologies that ensures NetApp systems stay as fast as possible as the system fills.

It doesn’t help with my sock drawer, unfortunately, but it might beat the FUD off.

.

OT: MIT OCW, A Remarkable Resource

Napier University, Edinburgh; image from Wikipedia

My youngest daughter has just started Napier University, and she and I were browsing around the intertubes looking for cheap secondhand textbooks for her Animal Biology course. Some of them are in excess of £60 each (€90, $110) new. That’s a big bill for course materials.

MIT Open Course Ware

We stumbled across this site; Massachusetts Institute of Technology Open Course Ware offers over 1800 of its courses online; PDFs, audio, video across a range of subjects. Some of them are full undergraduate courses, and there are graduate courses too; even full MIT authored textbooks.

If, like me, your university days were truncated early by too much extra-curricular activity, or if you’re interested in physics, biology, engineering – it’s a big list they offer – then this is a goldmine.

Hats off to MIT.

All I need is to find time here at ShadeOfBlue Towers to work my way through the undergraduate physics course. That’s a much bigger problem than getting my old and grey brain cells to fire in the right order, I can assure you.

[Update. My blogging tool seems to want to center align all the paragraphs. Weird. Hopefully this manual edit fixed it. Anyone else have this problem with MS Windows Live Writer?]

ELF, Wealth and Happiness

Usable Capacity

How to describe capacity? I’ve been exploring the whole usable capacity argument in my last few blog entries. There’s one thing for sure; we're in desperate need of new words and phrases to describe new technologies. Of specific interest to me are the inadequate words

usable capacity or capacity utilization

I consider those to be describing (albeit unclearly) the same thing. Two recent examples come to mind.