Extensible NetApp: Thinking outside of the box

Snake Oil, Object Stores and Dinosaurs

On Monday I called File Virtualization Snake Oil, and Martin called me on that.

And before that I told Chuck that his object store pontifications were wrong, and he called me a dinosaur.

So let me defend myself...

Let me explain Snake Oil

My objection to File Virtualization is to the set of products in the market that don't work. Products like F5 Acopia and Rainfinity. They simply do not work. They are bumps in the wire that don't scale. They break data protection, they make backup harder, and they create all sorts of new failure modes. 

Furthermore, their main value prop is the ability to move data between tiers, and frankly there are simpler, more robust and more elegant ways to do that, even before NetApp introduced Data Motion

Object Stores

The proponents of object stores are the new wave of file system bigots. A traditional file system consists of an object store (a series of inodes with a sequence of blocks) and a namespace (a directory structure that points to a series of inodes). To find an object you do a look up into the namespace and then retrieve the data from the object store. That's how they all work, and have worked since Kernighan and Ritchie wrote their original file system.

Now I am told, that embedded name look up through directories is *flawed*. That it's in the past.

Let's be clear, extracting the namespace from the object store, and declaring the death of file systems is surprising.

The user-view of a file system is that they do a name lookup and then retrieve the object from the object store. The only question we are asking is whether the name lookup must be hierarchical and must be enforced in the storage system.

And guess what, it doesn't to both. But just because I removed the namespace look up from being embedded with the object store, doesn't mean I suddenly don't have a file system. I just have a file system that consists of multiple layered elements.

And I think having those layered elements is goodness. Because embedding the name space lookup in the storage device is too constricting. And guess what we've been doing that since 1994 when we invented this http protocol. Hmm... coincidentally NAS was invented then as well...

The 75$ VDI disruption

Vaughn Stewart wrote today about how NetApp is able to deliver VDI at a storage cost of 75$ per seat.

Chuck Hollis wrote, in reply to Vaughn, with what is the essence of the problem with being on the wrong end of a disruption:

I'm actually glad that you've decided to focus exclusively on "cost per desktop". That's what VDI is all about, right? Saving money without worrying about delivering other forms of business value?

You've decided to compete with $50 desktop drives from BestBuy.

Best of luck with that.

-- Chuck

The reality of a disruptive technology is that it provides enough value at the right price point. Not the whole value, but enough of the value. And yes, the price point for VDI is the damn 50$ disk drive at BestBuy (or Frys for those of us in California ...).

If you can't deliver at that price point you're making a TCO argument. About how over time the customer will see real savings. Sort of how Sun competed with Linux for years, and customers kept saying but I want to pay less..

EMC claims to have figured this out from the early part of this decade: that either you deliver the price and value or you might as well not show up ...

Part of the point on my whole series on the DAS disruption is that you have got to be able to hit the price point. TCO is funny money. Real dollars are what people care about.

The fact that NetApp is able to compete with the cheapest alternative and deliver compelling value is nothing short of amazing. This may, just may, represent the tipping point for VDI in the enterprise.

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

FAS Deduplication is here for the long haul

The current tussle between NetApp and EMC has, predictably, created a dust-up amongst the corporate blog-o-sphere. After all the storage industry believes in death-match-competitive-blogging, none of this polite, civilized discourse, here we reach for the knives and hack!

And it would be tempting to point out that Chuck and Storagezilla are like Aesop’s fox, but that would be too easy.

However, there is a point of confusion in the storage and analyst community about the relationship between FAS dedup and inline dedup technologies like the one provided by Data Domain. There is some opinions that say that this proposed acquisition is driven by flaws in our technology. There are some customers who may be wondering whether this reflects some flaws in our technology.

So let me state again, what I said earlier:

The rest of the storage industry has adopted inline because they lack NetApp's storage virtualization layer that enables primary deduplication.

But because that confusion is real, I wanted to remind folks about why FAS dedup is unique. And more importantly, why NetApp FAS dedup is different and must be different to meet the market requirements our storage plays in than other inline dedup solutions.

I went over this in a post several months ago in a post titled A little digression on Inline Dedup.

The basic points I made are the following:

1. Deduplication is a compute intensive operation
2. Single core performance is stagnating
3. Latency isn’t something it’s the only thing
4. Backup workloads are latency insensitive but throughput sensitive
5. Primary storage workloads are latency sensitive

And I observed that because of 1, 2 and 5 any inline process is problematic if not deeply flawed.

And then in another post I observed that the important question for dedup isn’t how efficient the dedup process is, but how well do you perform after the dedup process is complete.

Our deduplication process performs well because of it is integrated into our core WAFL data structures and because, unlike everyone else in the storage industry, we’ve solved the random IO performance problem introduced by on-disk layout fragmentation.

FAS dedup is the most widely deployed dedup solution, the only workable primary storage dedup solution, realizes tremendous cost savings, and continues to provide compelling value to our customers.

Additional Information and Where to Find It

As mentioned above, in connection with the proposed acquisition of Data Domain, on June 4, 2009, NetApp filed with the SEC a Registration Statement on Form S-4 (Commission File Number 333-159722) containing a Proxy Statement/Prospectus for Data Domain’s stockholders, and NetApp and Data Domain each plan to file with the SEC other documents regarding the proposed transaction. The definitive Proxy Statement/Prospectus will be mailed to the stockholders of Data Domain. BEFORE MAKING ANY INVESTMENT OR VOTING DECISION, INVESTORS AND SECURITY HOLDERS OF DATA DOMAIN ARE URGED TO READ THE PROXY STATEMENT/PROSPECTUS AND OTHER DOCUMENTS FILED WITH THE SEC CAREFULLY IN THEIR ENTIRETY BECAUSE SUCH DOCUMENTS CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION.

Investors and security holders will be able to obtain free copies of the Registration Statement, Proxy Statement/Prospectus and other documents filed with the SEC by NetApp and Data Domain through the web site maintained by the SEC at www.sec.gov and by contacting NetApp Investor Relations at (408) 822-7098 or Data Domain Investor Relations at (408) 980-4909. In addition, investors and security holders will be able to obtain free copies of the documents filed with the SEC on NetApp’s website at www.netapp.com and on Data Domain’s website at www.datadomain.com.

Participants in the Acquisition of Data Domain

NetApp, Data Domain and their respective directors, executive officers and certain other members of management and employees may be deemed to be participants in the solicitation of proxies in respect of the proposed transaction. Information regarding these persons is set forth in the Proxy Statement/Prospectus described above. Additional information regarding NetApp’s executive officers and directors is included in NetApp’s definitive proxy statement on Schedule 14A, as filed with the SEC on July 14, 2008, and additional information regarding Data Domain’s executive officers and directors is included in Data Domain’s Annual Report on Form 10-K and Form 10-K/A, as filed with the SEC on March 13, 2009 and April 30, 2009, respectively. You can obtain free copies of these documents from NetApp or Data Domain using the contact information above.

Better Than Real Fibre Channel: The Other Guys

Brian Rawlings, correctly, took me to task over my parochial focus on WAFL as the only Better Than Real Fibre Channel technology.

The reality is that the kind of things WAFL does are being done by other vendors. I'll claim not as well, or not as elegantly or not as completely, but there are other vendors who are breaking with Traditional Storage Array architectural orthodoxy that disk layout must correspond to what the client thinks the disk layout is.

So just for the record, the other guys include but are not limited to the folks at 3Par, Compellent, and Equallogic.

And for the record, the future belongs to us (just that more of the future belongs to NetApp :-))

The importance of architecture

This is a more technical/philosophical post than my usual commentary, forewarned is forearmed.

Earlier I wrote about the notion that software is like group theory. What I said was that there exist abstractions that form in a software system over time, and that the key to understanding and improving a software system is to understand those abstractions.

I also argued, that those abstractions were not accidental, but were actual manifestations in software of how the world really worked. By making the case that these were real abstractions and not artificial constructs, I argued that changing or fighting them was done at your peril.

What's important, of course, is that any mathematical, and therefore by extension, software abstraction is an approximation of the world, not the real thing. And that approximation may help you solve some problems better than others ...

When we argue about software and hardware systems in the corporate storage blog-o-sphere we are arguing about specific notions of how the world is actually constructed. I believe part of the acrimony and bitterness is that we're not just puffing our chests and cheering our team on, but we actually have different wold views.

Enough with the blah, blah, architecture, blah, group theory, blah

My recent posts about WAFL reminded me of a theory Prof. David Cheriton repeated in his Stanford course on distributed systems. What Prof. Cheriton observed was that many times innovation happens when, by exploiting a computer trend, you relax a previously hard constraint making a previously hard thing easy.

The example he used, if my increasingly faulty memory serves me right, was the use of HTML/XML and ASCII to serialize or marshall and then deserialize or unmarshall parameters.

A canonical problem in distributed system design is how to transport data over the wire. The problem is that data is represented as a stream of binary digits in main memory. The only thing that gives it context is the structure software imposes on the data.

The problem in a distributed system is that the software can be written in two different programming languages running on different hardware meaning that there has to be well understood language and hardware independent rules about what the stream of bits mean.

For example, suppose I have remote procedure call int foo(int b). I need to convert the input parameter b into some format that can be transported over the wire, and interpreted at the other side as an integer with the same value.  This transformation is what is known as marshalling and unmarshalling, I'll call it translation.

The traditional approach to this problem was to use a binary protocol that encoded the integer into a system independent format as you went over the wire and then translated the binary data into the system dependent format at the point where it needed to be interpreted. Software on either side of the wire using the translator could send and receive data.

Building such and infrastructure that is robust is a relatively hard but straightforward problem.

Building such an infrastructure that is widely used by everyone was impossible.

The problem was that there was no industry standard for the translators. For computer programs to talk to each other over the wire they needed proprietary and expensive binary translators. CORBA, for example, was a failed attempt at creating such a standard.

Enter HTML/XML

HTML/XML are horribly inefficient compared to a binary translator. HTML/XML are inefficient  with their use of bandwidth and their use of compute resources. For example a four byte integer quantity can turn into an 11 byte string, plus the meta-data required to tag the value.

But HTML/XML do not require a proprietary binary translator on either side of the wire.

So the idea was to relax your requirements for performance to get ease of use. And that was okay because CPUs  were becoming faster, memory was becoming cheaper and bandwidth was becoming more plentiful.

The net result was that HTTP+HTML/XML became the defacto mechanism for programs to communicate to each other over the internet and things like ONC RPC/CORBA/JAVA RMI/DCOM died or are dying a slow and increasingly painful death.

But if that's too abstract and confused...

Computer programming languages are another good example.

Programs written in C can be faster and more efficient than programs written in Java. But the time to develop, debug and qa those programs is vastly greater than an equivalent program written in Java. 

By trading off computational efficiency (memory and CPU resources), programmers are made more efficient, resulting in more software being developed.

I am storage architect not a computer science historian...

When Dave Hitz and James Lau architected WAFL they relaxed the fundamental constraint that file systems had preserved for a long time, namely that the client notion of disk layout actually corresponded to the on-disk-layout.

This was a profound decision. By relaxing that constraint, WAFL had taken on the job of providing the illusion to clients that the disk-layout was sequential even if it wasn't.

At the time that might have appeared to be a ridiculous thing to do. And given the reaction of the other guys (vendors of Traditional Legacy Arrays) still seams to be a ridiculous thing to do.

But...

If you looked at the computer trends, maybe it wasn't. By exploiting the increasing performance of CPU's, the increasing density of memory, it became possible to write smarter and smarter algorithms that were more and more capable of providing the illusion that the disk layout was sequential even if it wasn't (for more on this topic see an earlier post)

And by relaxing that constraint, it became possible to deliver innovation like thin provisioning, Real Snapshots and deduplication without compromising performance.

Why?

All of these features, primary dedup, Real Snapshots, high performance RAID-6, require an on-disk layout that is different from the client layout. For our, NetApp, internal abstractions, these are easy to do and a natural extension of our software abstractions. And modifying your system in natural ways is easy only if your abstractions easily map to those abstractions.

For Traditional Storage Arrays, fixed in their notions of disk layout, asking them to support a model where the on-disk layout is different from the client layout challenges the fundamental abstractions they have evolved over the last 15 years. And so they try and create approximations, like pudgy provisioning, but ultimately the extensions looks like an ill fitting suit, because they are.

Real Fibre Channel is really Traditional Legacy Array

Chuck Hollis likes to call the CLARiiON LUN a Real LUN.

I've taken to mocking Chuck and EMC by calling their storage Real Fibre Channel.

But after my recent series on RAID-6, and Thin Provisioning and the fact that Real Fibre Channel has architectural  flaws that make it less than useful for either of those critical capabilities, and the limitations around snapshots, I've been thinking that the term was just plain wrong.

Real Fibre Channel is like the PDP-11:  A legacy hardware architecture that has had it's moment in the sun.

Henceforth, I'll call the CLARiiON and the architectural dinosaurs that are like it, what they really are:

Traditional Legacy Array.

RAID-DP: Better performance, with less hardware and more availability.

A couple of weeks ago I focused on answering the following question:

And although the answer was interesting, the question was uninteresting. All I explained was that with more hardware we can perform better.

A much more interesting question is why does NetApp RAID-DP perform so well compared to other RAID-6 implementations.

Why is RAID-DP performance more interesting?

Regardless of whether you believe NetApp or EMC, RAID-6 is more resilient than RAID-5, and RAID-5  or RAID-4 is fundamentally unsound for very large disks.

So if RAID-5 or RAID-4 are unsound, why the angst about RAID-6?

Performance.

It turns out that RAID-6 performs worse than RAID-5 because every update to disk consumes more IOPS. If you recall from my earlier blogs, RAID-5 on a Real Fibre Channel array is expensive, because a single update to a single block on a single RAID stripe will turn into several read operations and two writes, one for the new data and one for the updated parity.

For RAID-6, you now have to do three writes, one for the data and two for the parity.

So, RAID-6 performance is worse, but why do I care?

The reason you care, is that if you need the performance and resiliency you need to use RAID-10.

Huh?

RAID-10 performance is excellent and offers much better resiliency than RAID-5 and on larger SATA disks is more or less the only alternative if you care about your data….

Wow…

If you need to get performance and resiliency, you need to buy 2x the disk capacity you require.

Kinda kills the entire value proposition of shared storage.

In fact if you have to deploy RAID-10 on the storage back-end, I really think that the right architecture you should be looking at is host-level replication with brick architectures.

Don’t believe me, then here’s the proof from the folks at Hopkington

Since EMC does not publish benchmarks, and I don’t have any RAID-6 arrays sitting around, all I can do is quote the EMC whitepaper (page 13):

RAID-6 write performance is less than that of RAID-5

and the reason

The extent of the performance varies depending on write sizes and RAID group sizes, but the extra parity information that has to travel on the back-end buses to disk results in lower system bandwidth for a system with all RAID-6.

This is why RAID-10 is recommended for any environment where performance is important.

So why is RAID-DP faster?

When it was first explained to me many moons ago, I was a little bit surprised. Yes RAID-DP is optimal, but a key part of the performance win is how WAFL does write allocation.

If you read my earlier blog, the reason  WAFL performance with low utilization is so good is because we end up doing sequential full stripe writes and because finding new places to do sequential writes requires very little effort.

RAID-DP is just a natural extension of that approach. Instead of doing the parity calculation one stripe at a time we do it multiple stripes at a time. The efficiency obtained by not having to write in place means we get really good performance and avoid the RAID-6 performance penalties in Real Fibre Channel arrays.

Now it turns out that although we have to do more work at high utilization and that brings our performance in-line with Real Fibre Channel, what the picture doesn’t show is that we achieve that performance while doing RAID-6 or RAID-DP.

In fact our SPC-1 benchmark show how  WAFL using RAID-DP has the same performance as a RAID-10 CLARiiON configuration.

So another way of looking at the picture from above is that our performance at high utilization still kicks ass because we require fewer spindles.

or

WAFL and RAID-DP provide better performance, with better availability with improved utilization than the alternatives.

Real Thin Provisioning vs Pudgy Provisioning

Last week I observed that thin provisioning creates fragmentation, but I never got into the details.

And I further observed that our competition wasn’t really doing thin provisioning. I didn’t have a catchy name for that kind of provisioning, and today I do: pudgy provisioning.

Today I want to go into the details.

In a Real Fibre Channel environment, the space for the LUNs are pre-allocated and assigned to individual LUN's.

The picture above tries to show the story when using Real Fibre Channel LUNs.

When a LUN gets allocated the free space in the storage array gets committed to the particular LUN. This space allocation is not just a logical accounting trick, but is an allocation of particular disk blocks. In effect, the space allocation is pre-committed to a LUN before the application using the LUN actually uses the space.

The rationale for this scheme is to ensure that the disk layout that the array uses is identical to the disk layout the application assumes to ensure good sequential throughput. For more check out this blog entry.

The problem is that to ensure good performance the storage array wastes space.

What about thin provisioning?

The design goal for thin provisioning is to reduce the amount of waste by deferring the space allocation in the array to when the space actually gets used by the application.

The picture above tries to show the theory of thin provisioning. Unlike Real Fibre Channel space only gets committed to the LUN only when it’s actually used, eliminating the waste.

The picture, however, is misleading, because it implies that the layout on disk doesn’t change.

Enter Fragmentation

The problem with thin provisioning is that applications do not necessarily do their write operations in a way that ensures that data layout on disk is preserved.

Suppose the write order for the application is

But then the problem is that the layout on disk looks like this:

NetApp, because this kind of layout issues just happens to be endemic to how WAFL works, has solved this problem.

If you can’t do thin provisioning, how about pudgy provisioning?

If you don’t have WAFL and 15 years of experience dealing with the problem, making this work as well as WAFL is hard.

But there are shortcuts.

WAFL will allocate a disk block on disk as a disk block gets written by the application.

The most obvious alternative is to instead of doing a one-to-one disk block allocation to do a one-to-many disk block allocation. Or what I like to call pudgy provisioning.

Let’s be precise:

Whenever a single block allocation is made, rather than allocate  a single block at the next free space in the array, the array allocates an entire region.

The value of this approach is that the on disk order matches the application’s notion of what the disk order is:

.

The problem of course, is that , if the application doesn’t allocate all of the storage blocks then you’ve got waste.

Huh?

Let me see if I can follow this logic:

1. Application groups ask for too much storage
2. Real Fibre Channel creates waste because space is allocated before it’s used
3. Thin provisioning allows you to allocate only space that is used
4. If you don’t have WAFL then fragmentation on disk becomes a performance problem
5. So instead of thin provisioning do pudgy allocation where you allocate more space than the application needs but not as much as before
6. Which leads to waste…

This makes perfect sense.