Extensible NetApp: Thinking outside of the box

Unified Storage: The Professional Movie

After my award winning amateur performance, NetApp's marketing team decided I need to get my story done professionally.

I like to think of my first video as my screening test, this is the real thing.

In my last post there was a sense of confusion about data protection.

I said NetApp had a single mechanism for doing data protection and touted the value of that.

Some folks wondered: isn't flexiblity a good thing? And I agree. What I meant, was that  for DR, there was a single mechanism for doing data protection regardless of data access protocol (CIFS, NFS, FC, iSCSI).

If you want to hear more about our data protection, I recommend this video

Versatility of SnapMirror -- NetApp -- Storage Efficiency

A video of unified storage

After blogging about the value of unified storage, I thought a video would be nice…

The assault on RAID

Earlier this week I wrote about how the core architecture of Traditional Legacy Arrays limits the performance of RAID-6. Just to recap, RAID-6 requires an additional read and an additional write over and above RAID-5, limiting it’s performance and applicability.

What I didn’t do is explore the consequences of such limitations.

The first, and most obvious consequence is the proliferation of RAID-10. SATA disks are too big to be used with RAID-5. The only alternative is to use RAID-10 when you need both resiliency and performance!

This singular fact turns the economics of RAID on it’s head.

Let’s be clear, the reason RAID became so popular is because it solved two generic problems: performance and availability at a lower cost than the alternatives. The problem with disk drives is that they have a limited number of IOPS per spindle. The only way to get more IOPS is to combine more of them and spread the IOPS over the spindles. The downside is that the MTBF (mean-time-between-failures) of the disk drives makes the group of disk drives increasingly less reliable.

So if you’re a customer that has business critical application that requires availability and performance, you’re more or less screwed. To get the availability and the performance you need to 2-3x the storage. As an example, consider GFS, that uses mirroring and no RAID. GFS will have at will have at least 3 full copies of the data.

Enter RAID.

RAID is so common place now-a-days that we forget how miraculous it must have appeared to folks when it first showed up. RAID allowed the computer industry to provide resiliency and performance at a lower cost! So for example, using RAID-5, you only need 1/5 (4+1) additional storage to get terrific data resiliency compared to the 3x that GFS requires.

Enter Big Drives

The problem with RAID-5 and RAID-4 is that the probability of various kinds of failures increases as the disk drive capacity increases. No surprise.

NetApp pioneered the use of RAID-6 with our FAS systems, and because of WAFL we are able to preserve the RAID value proposition of increasing capacity and performance with modest increase in physical capacity.

But we’re not the market.

As much as I would love to be the top-dog in the storage industry, the Traditional Legacy Array vendors, and in particular EMC, don’t have a very good RAID-6 implementation. Or rather, as I said earlier, their core architecture precludes a RAID-6 implementation that they can recommend for all workloads.

And because of their share of the market, they define the technology expectations for everyone who intends to use storage.

The problem is that we don’t live in a static universe. Software companies, like Microsoft, are continuously trying to reduce the total cost of their solutions. And although it’s nice to be able to bill customers for RAID-10, the reality is that for every dollar that goes into the hardware, Microsoft makes less money.

So Microsoft and the rest of the software industry, faced with the reality that the dominant Traditional Legacy Array vendors, were pushing RAID-10, or basically 2x the physical capacity for each byte of data stored, looked elsewhere for a solution.

It must have been particularly galling to software vendors that disk drive capacities were increasing, disk drive cost per GB per decreasing, but the storage infrastructure was increasing in cost reducing the total available pie.

Enter host based replication etc…

The cool thing about technology trends and curves, is that everyone is staring at them.

If I was a software application architect here’s what I would observe. The biggest cost for an application from a hardware perspective is the storage, and biggest cost factor is the disk drives.

Today with RAID-5, the picture looks like this:

The red controllers are provided by Traditional Legacy Array vendors, and there is a small increase in the total capacity to get excellent resiliency (the pink area).

But with bigger SATA drives, and the absence of a high performance RAID-6, the world looks like this:

Ouch. The total cost of the solution just jumped up! You get cheaper disk drives, but you need more of them!

Now if I am an application architect, I think to myself, you know that RAID stuff can be done in software, and I can do the mirroring at the application level, and the server vendors sell really cheap disk drives, so maybe the picture can really be made to look like this:

Each server has a pool of disk drives, and mirroring is done in the host to obtain some degree of storage resiliency. Yes the costs are greater than RAID-5, but it’s not clear they are that much higher than RAID-10 … And you might even believe they were lower than RAID-10 because you’re saving on the dollars you had to give to the storage company.

So what does this have to do with NetApp?

I really believe that RAID-10 mirroring or DAS is an inefficient way to manage disk drives.

I also believe that the popularity of DAS/Host mirroring solutions is a response to the failure of the market leaders to build an array that can preserve the RAID value proposition.

Since RAID-10 and DAS/Host mirroring are a response but the wrong fundamental approach, I believe that architectures, like NetApp’s, that make RAID-6 work will ultimately win out.

And oftentimes in the debate about DAS vs shared storage, we confuse the specific problems of the Traditional Legacy Array vendors, with the right long term technology trend.

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

How Dedup and Snapshot Change Backup for VMware

One of the more interesting conversations I've had recently was how NetApp end-to-end deduplication and our replication technology fundamentally changes how you think about backup in a virtualized environment.

The core challenge in a VMware or HyperV environment is that the restore is extraordinarily painful because of the amount of data that needs to be moved. You are no longer just restoring a file, but an entire image including OS, application image, and data.

And the single biggest bottleneck to a restore is the data transfer from the repository to wherever the primary is connected.

The typical infrastructure has data flowing from some primary system, through a backup server to some backup media.  The notion being that primary storage is expensive, and the backup target is much cheaper, making the restore time acceptable from an overall cost perspective.

If data transfer is so painful, then why transfer the data? Why not keep it around on the local system?

Well there are three reasons:

1. Cost of keeping extra copies
2. Performance cost of keeping extra copies on the primary
3. Recovery after a disaster.

It turns out that NetApp Snapshot and our Deduplication fundamentally change the cost equation because of how they work together. NetApp Snapshot technology only keeps a point-in-time copy of the blocks that have changed inside of a volume. NetApp Deduplication eliminates any duplicate data between different storage containers within a volume.

And of course, unlike other snapshot technologies, a NetApp Snapshot is performance neutral so using them is not a space-time tradeoff.

And it turns out that deduplication has a small performance impact as well.

What does this mean? Well it means that the amount of copies a storage architect can keep on the primary is significantly higher than using other storage arrays. More snapshots, means less likely to go to a remote machine to restore data and that translates into faster recovery times!

I can see someone point out that the primary storage system will still be more expensive even if snapshots and deduplication is used. But here's the rub, the secondary  system costs money and the restore costs money in terms of lost productivity during the restore. To make this worthwhile the primary system has to be cheaper than the cost of the secondary system AND the restore cost. This is a much lower bar to jump over.

But what about a disaster?

Enter SnapMirror our volume mirroring technology. Well it turns that SnapMirror allows you to replicate an entire volume and all of it's snapshots. It also turns out that SnapMirror will only replicate the deduplicated data.

What this means is that cost of the network and the secondary system is lower than using alternative technologies because the amount of data is less, and the amount of computation is less (no deduplication on the secondary).

It also means that the recovery after a disaster is faster, because the data is online and can be directly accessed rather than stored on Tape or a VTL and then restored to a primary storage system.

But that's not all ...

Because you are keeping the backup data on the primary storage system and are not moving the data other than to replicate for DR, this also simplifies the backup infrastructure. There are no complex highly available backup servers that need to be setup and managed.

And because you are keeping all of the backup data with the primary copy, there is no painful need to index and track data on remote devices.

So in conclusion

You get faster restores, faster recovery after a disaster, less complexity because there is no backup process and backup system to manage all because the cost of doing this got cheaper...

Pretty cool...

Vmware and NetApp

Steve Klinkner has put together a nice set of posts about how to use Vmware and NetApp infrastructure.

If you want a nicely written tutorial you should check it out.

Plowing through the SPC-1 Haze or How to Enjoy Your Dessert

I fully expect to see over the next month a vast and voluminous debate within the blogosphere about the validity, or not, of SPC-1, of NetApp's numbers or of EMC's numbers.

I am, anxiously, anticipating the point by point dissection of the data. There will be tits and there will be tats from both sides. And the press will get involved.

But that's not what I found intriguing about NetApp's results.

What I found intriguing was that, finally, you could eat your cake and have it too.

There is a presumption, and a reasonably valid one, that you have two choices:

Virtualization or Performance

Given any piece of hardware, if you have more instructions to execute you must, by definition be slower. And since NetApp has more virtualization it must perform worse.

But if you are a data center operator, what EMC is really saying is the following:

You should be willing to ignore the simplicity, ease-of-use, that NetApp storage brings to the table, because you know that their performance must be worse than ours because we have less virtualization.

After all, virtualization, must make things worse.

But what does this mean in practical terms?

Let's consider snapshots.

The beauty of NetApp snapshots is that their performance impact is minimal, and that means that you can keep them on your primary storage system.

But why is that important? Because if you have to do a recovery (you know restore from a backup) you don't have to copy a lot of data. And the amount of data that has to be moved is the single biggest factor in any recovery situation.

Now if you believe the EMC FUD about this benchmark result, you will be told that virtualization and, therefore, NetApp snapshots must affect performance. And let's assume that you do believe them.

What EMC is really trying to say is the following:

Hey, Mr. Overworked Storage Architect, you can trade off performance during normal operations for crappy recovery times if you buy our stuff vs crappy performance during normal operations with astounding recovery times if you buy their stuff.

And so now you have to sit in a room and figure out the following:

For which applications do I care about recovery times and for which do I care about operational performance?

And if you make a mistake, you might find yourself in a world of hurt. Either because the operational performance is worse than expected, or some recovery is taking so long that it is affecting the business...

What we're saying is that NetApp storage eliminates that debate. And that's why our systems are simpler. Because you can have the performance you need, and the recovery times you require.

And that is why this SPC-1 benchmark was so important.

Replication: You keep on using that word. I do not think it means what you think it means.

One of my favorite movies is The Princess Bride. And one of my favorite scenes in the movie is when the Sicilian Vizinni keeps saying that it's inconceivable that the Dread Pirate Roberts can do X, and the Dread Pirate Roberts does X, ultimately leading to this:

Vizinni: HE DIDN'T FALL? INCONCEIVABLE!

Inigo Montoya:You keep on using that word. I do not think it means what you think it means.

I sometimes feel that way about Replication.

EMC, because of its size and its position in the market and the technologies it sold, has managed to convince the world that replication and by extension storage based business continuance is all about synchronous replication.

The idea was that if a disaster were to befall one data center, you would be able to instantaneously bring up the other data center, because all of the state would be perfectly mirrored.

As a result, the notion of replication became synonymous with continuous replication which became synonymous with business continuance which became synonymous with DR.

But that assumed a fail-stop failure. That's a a disaster where everything in the data center blows up at the precisely same point in time, ensuring that the remote site has a consistent state and can just go ahead chugging along.

But what about other failures, like a bit of data that gets corrupted because of human or hardware failures? Or a network bit-flip? Or what if, for whatever reason, your application fails to recover because the data is crash but not application consistent?

In which case, the only recovery option is to go to your last known backup and then slowly and very painfully roll forward. Which makes a synchronous solution useful, because you have more data to recover, but not the answer to DR.

So you need a different kind of replication technology. A replication technology that replicates your backups, and does that efficiently, something like SnapShot and SnapMirror.  SnapShot are your backups, and SnapMirror moves those backups very efficiently between sites, and has some clever technologies for re-syncing after the disaster hits.

Which gets me back to my original whine, that calling everything replication obscures a whole slew of very powerful technologies that solve real problems. And that not every form of replication is the same.

And that some forms of replication actually solve the DR problem, and some are complementary, but how they heck is a customer supposed to figure all of that out?

The poorly named replication

Over the past several months, I've been trying to understand the difference between the replication market and the backup market.

For example, NetApp sells replication software. But our customers think of our solutions as backup products. Heck, if you look at Protection Manager, you'll discover that we call a Snapshot a backup.

Why?

So it turns out that, the IDC, the keeper of numbers, classifies storage system copying as replication, and classifies backup software as, well, backup. Even if the backup software is, well, making copies from one server to another.

Which got me thinking about the storage industry, in general.

No one actually wants replication. Replication is a technology. It's a means to an end.

However, the storage industry is so enraptured with the technologies that we produce, that instead of convincing the IDC that we actually solved a problem like Business Continuance, we managed to convince the IDC that we sell, well, copy. Because somehow replication is, way, way cooler. Except of course for our customers who have a backup problem.

Which brings me to why I love Protection Manager. The product is focused on the top-level goals of the customer: backup and recovery, not on the technology that implements backup and recovery. As a result Protection Manager has no references to terms such as  Snapshot, SnapMirror or SnapVault, because you, the customer, don't care about how we backup your data, just that we do it.