Extensible NetApp: Thinking outside of the box

The principle of uniform resource latency

Just before I commented on Karl Dohm's quixotic quest to prove the NetApp sales team were snake oil salesmen, I was talking about IOPS data density.

In that post, I threw out the following statement:

I'll also state that applications expect consistent performance rather than variable performance and perform worse as variability increases.

This is a very important and fundamental concept and underlies a lot of how I think about Flash and storage tiers will shake out. So I will spend a little bit of time on this point.

So let's begin at the beginning.

Within a company, which is a conglomeration of humans working to a single set of goals, consistency is very prized. Consistency makes long term planning possible. Variability makes long term planning impossible.

The software systems that exist to support human beings must, therefore, fit into a world where people expect consistency and predictability.

As a result, software solutions have been designed and architected around providing consistent behavior.

Over time, software architectures have evolved to the point where some layer of software, typically the application, expects all the resources it uses to have the same performance.

What do I mean?

Well an application that uses a database is designed around the assumption that regardless of what field in the database is accessed, there is some maximum time the database will take to respond. The application is then designed to work given that maximum. When the application gets better behavior from the database, the user of the application is pleasantly surprised, but at no time does the user of the application experience worse performance than the expected maximum time.

In practice what this means is that application authors expects the underlying software (database, operating system, file system, storage, server, network) to provide uniform resource access to the resources they manage. Vast amounts of innovation are invested in software to provide a sand-box that provides the illusion of performance uniformity.

From a narrow and simplistic storage system point-of-view, this implies that along with IOPS there is an expectation of uniform latency, which is why benchmarks like SPC and SFS measure latency. And just in case I forget to say this, providing uniformly good latency is not just a SAN thing, but is also a NAS thing too…

And why all this effort?

Because of the simplicity of programming model. To provide that simplified model, a significant amount of complexity and cost are introduced throughout the infrastructure, but as an industry we have chosen to make that tradeoff.

So how to provide uniform resource access?

At one extreme, Seymour Cray remarked You can't fake what you don't have, and designed a system to provide uniformly excellent performance. At the other extreme you have the web which promises nothing.

What's important to note is that between spending a lot of money, and doing nothing there are lot of well established in-between places, also known as caching, that make it possible to provide on average excellent latency at a fraction of the worst case cost.

In my next in this series, I'll consider, and this time I promise, Flash and the new storage tiers.

MAD Blog: HP not letting it go ...

There is a specter haunting HP's Karl Dohm, the specter of NetApp's sale force snookering customers about the real value of NetApp storage.

Patrick mentions the true test, i.e. that there are many happy NetApp customers who are running Exchange.  There is truth to this of course, but it isn't a good basis of comparison because every major array vendor has happy Exchange customers.  However, its reasonable to say that these installations can't know what they don't know. 

Karl can not accept the notion that the value NetApp brings to the market is real. After first trying to prove that NetApp can not run on Exchange:

The problems we are talking about here are the core of WAFL, and are clearly not easy to fix - or they would be already fixed.  NetApp is not unique is having problems of course, all array vendors have their strong and weak points.  But to assert that WAFL has no weaknesses around fragmentation, performance, and capacity utilization defies common sense.  The old wounds are there for a reason.

Karl had to back peddle and admit:

I'm not saying you can't run Exchange successfully with NetApp.  In fact I'm sure you can.  The question looking for an answer is whether the user gets good value in choosing NetApp to run Exchange.

So reading Pat Cimprich's latest response was particularly gratifying.

Look, the FUD that HP is currently throwing out there is age old FUD that EMC has been throwing out for a while.

After 10 years of watching an artiste like EMC spread FUD, watching this ham-fisted attempt at spreading the same FUD is, well, embarrassing. And having to respond to it, is well irritating.

IOPS data density and latency

As I started to write about flash and storage tiers, I realized there were some basic underlying concepts that were not clear in my head. So I will first try to clarify those before I get to flash.

Earlier I wrote about the IOPS density of spinning media. The point of the post was this graph:

What this graph shows, other than my inability to use SnagIt to correctly cut-n-paste an image, is that at some point in time the number of hot bytes of data on a disk drive is negligible compared to the number of cold bytes of data.

After I started writing my series on storage tiers, I realized that there is a corresponding notion of IOPS density, but this time for the data not for the media.

So I am going to define a new term:

IOPS data density: The ratio of hot vs cold data for any dataset at any given point in time.

For some datasets, for example, home directories, the IOPS data density is quite low. If you read the paper Meaurement and Analysis of Large Scale File System Workloads by Andrew Leung et al, you'll notice in section 4.2 that the amount of data that is transferred between the client and the server is a fraction of the overall amount of data that is available.

The problem with the term IOPS data density is that as defined, it does not capture the notion of latency. Users of data don't care about IOPS, they care about the latency (or time) it takes to perform an operation.

I'll state that the performance of an application, where an application may be a database or an NFS client,  is related to the average performance of the average IO. I'll also state that applications expect consistent performance rather than variable performance and perform worse as variability increases.

So to expand my definition a little bit:

IOPS data density: The ratio of hot vs cold data for any dataset at any point in time. The application using the dataset expects the hot data to have a sufficiently low latency that is reasonably uniform across all of the hot data.

Next we explore the different kinds of IOPS data density and how they map to the different storage tiers.

Re-branding tiers to Captive, Shared, Capacity

Stephen Foskett suggested I re-brand my tiers of storage. And after some thought I agreed.

So we have two IOPS tiers

• Captive (used to be Captive IOPS)
• Shared (used to be Shared IOPS)

And one capacity tier called (well you guessed it):

• Capacity (used to be Capacity Efficient)

Thanks to Stephen for pushing me to do this.

101 posts, one year later and still going strong

This is my 101st post almost a year after my first.

A good friend of mine said when I began:

Blogging is useful when you have nothing useful to say, because you can share that with world!

If the number of comments I've gotten is any indication, I've managed to interest some people in what I have had to say.

I started out with the intention to focus on things that were outside of the box, and ended up writing a whole series on WAFL.

Which goes to show you that what you want to talk about is rarely what you do talk about.

So what's changed since November of last year when i started writing?

First there was the blog debates with EMC and what happened with that?

• Chuck Hollis has one more person objecting to his commentary.
• Storagezilla has named a law after me.

Then there was the series on backup, SnapManager for Virtual Infrastructure, and WAFL

One of the things I did spend time on was coming up with useful definitions about storage systems. So for example

• I spent time on unified storage
• and a unified architecture
• snapshots
• and a new storage classification that is more useful than tiers.

But my favorite series was a series I had on the Fixed Point Theory of Data Centers something hat remains a central organizing principle about how I think about technology.

The most esoteric post I wrote is my comparison of system architecture to group theory.

Well enough looking back.

Time to go forward...

Disk and the new storage classification model

In my last post I explained the new storage classification that divided storage into Captive IOPS, Shared IOPS and Capacity Efficient.

Now I want to explain how disk based storage systems fit into this model.

What's interesting about disk is that you buy IOPS and get capacity for free. If you read my blog post about IOPS density, what I described was this phenomenon that disk capacities have grown so astronomically large in comparison to the IOPS that the disk provided, the effective IOPS for most of the capacity is 0.

If you are buying for IOPS then you are getting a lot of unneeded capacity. If you continue to add IOPS you are continuing to add even more excess capacity over time.

So what does that mean?

Well if you look at storage which is deployed for Captive IOPS tier, the disk capacity on the spindles deployed must be sufficient for the application. Why? Because most of the capacity is unusable because you can not drive enough IOPS through the disk. When you buy more IOPS you add more capacity, but most of the pre-existing capacity was sitting there empty anyway, so adding more capacity to a system overflowing with capacity is like adding water to the ocean.

The same holds true for the Shared IOPS tier. Namely there is a mismatch between capacity and IOPS. Each incremental increase of IOPS will result in huge increase in available but unused capacity.

What's interesting for both the Captive IOPS and Shared IOPS tier is that customers end up paying for a lot of unused capacity to get IOPS and there is no effective way to work around that.

Where disk is a really great fit is in the Capacity Efficient tier. With disk every time you add a new spindle, you are adding insignificant net new IOPS as compared to the jump in capacity. Well it turns out this is maps exactly to the economics of this tier, where the goal is to increase capacity much faster than you are adding IOPS.

And so?

I don't think disk is the new tape.

I think the economics of the Capacity Efficient tier are so very closely aligned to disk, that I have to believe that disk will continue to dominate in that tier. The Capacity Efficient tier still expects reasonable random IO performance, but is willing to endure slower performance for large increases in capacity.

Flash next...

MAD Blogging: A meta-blog post

Martin came up with this funny way to characterize the kind of blog flame war that periodically erupts between bloggers from different storage companies: Mutually Assured Destruction (MAD) Blogging.

Why is it MAD blogging? Because all we manage to convince our mutual customers of is that we know how to poke each other in the eye.

Although I would love to pretend I will no longer engage in MAD blogging, I know I will.

But I also know some folks, including Martin, get tired of reading those posts.

So out of respect for those readers, whenever I am about to engage in MAD blogging, my blog post title will have the words MAD Blog in the title.

Want to read a good old fashioned bare knuckled debate, read the posts. If you want to avoid the down and dirty debates, skip the posts.

Explaining the new storage classification model

Yesterday I wrote about a new storage classification model that I thought was simpler to understand than the more traditional, and useless, subjective tiering scheme.

But as I re-read my post, I really wished I had done a better job of explaining the motivation.

So let me motivate the rational for the classification of storage into a Captive IOPS  Tier, Shared IOPS Tier and Capacity Efficient Tier.

So without any ado...

Captive IOPS Tier

In this tier, there is an application let's say Exchange, that is so important that it gets it's own storage. The number of IOPS of the storage is sized by the application requirements.

The picture looks something like this:

As the application IOPS requirements increase, more IOPS are added. More specifically as you add more exchange servers, you add more IOPS to the storage array.

The picture evolves to look like this:

Shared IOPS Tier

In this tier  there are a collection of applications that each have, on their own, modest IOPS requirements but in aggregate require a large number of IOPS. For example, a virtual infrastructure may have a large number of small applications each individually requiring a modest number of IOPS but the total IOPS across all the applications is huge

This picture attempts to show what I mean:

.

This tier attempts to amortize the cost of the IOPS across a large number of applications. This tier exists because it is not practical to add a very small number of IOPS.

Capacity Efficient Tier

In this tier as more applications are added, more capacity is added faster than IOPS. For example, consider NAS home directories. Users need more capacity to store more data but the total IOPS they need stay pretty much the same. Furthermore, the incremental number of IOPS required by one incremental user is actually quite small compared to the capacity the user requires. So, in this tier, although IOPS increase over time, capacity increases far faster.

This picture attempts to show what I mean:

Next

In my next set of posts, I'll look at how disk and flash play in those three classifications.

The end of storage tiers

Flash has once again thrown into stark relief the absurd classification of storage into tiers

Talk to a storage vendor and Tier 1 is their most expensive stuff. Talk to a storage architect and Tier 1 is their most expensive stuff. If you're lucky there is some overlap.

Then we have Flash. Is it Tier 0? Does Flash make Disk Tier 5? What is the role of Flash and Disk? Is Disk the new Tape? So do we need to have Tier -1 for storage that is faster than Flash?

Then there is the whole disk storage is secondary storage. Secondary to what?

I never really did get all of those classifications of storage into tiers. I tend to think of storage in terms of how it is used.

So instead let me propose a new model for storage tiers based on the ratio of application CPU and memory, the amount of IOPS required and the capacity needs of the application or the ratio CPU:memory:IOPS:Capacity.

Based on that ratio there are three storage tiers

1. Captive IOPS, where IOPS are all dedicated to a single application. In this deployment the ratio is 1:1:1:1. Add more CPU and Memory and you add more IOPS and Capacity. Because of the nature of the application and how many IOPS it consumes, there is nothing left over for another application.
2. Shared IOPS, where IOPS are shared across a collection of applications. In this deployment the ratio is M:N:1:1. As you add more CPU and memory, the number of IOPS increase but not at the same rate. So you can share the IOPS across a number of applications rather than dedicating them to a single application.
3. Capacity Efficient where the number of IOPS is dwarfed by the capacity requirements. In this deployment the ration M:N:1:Q. where as M and N increase, Q increases but IOPS do not. A good example is a backup server. As more data gets backed you need more capacity, but you don't actually need more IOPS. Another good example is a home directory where capacity needs increase, but actual IOPS do not.

Next, I'll explore the implications of these three tiers.

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...