Extensible NetApp: Thinking outside of the box

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

Death to agents that add little value

Paul Turner wrote Death to Agents. And that got me thinking about agents in general.

There is a common and reasonable perception that everyone hates agents. They are a pain to deploy, they are pain to manage, they are a pain to upgrade. And yet we want the functionality they bring to the table. And so we install them, grouchily.

Paul's post really speaks to the following question:

If the agent functionality can be done without an agent do we need the agent?

And the answer is a resounding: No!

And I believe that for Storage Resource Management, the delta what can be done without an agent, and with an agent is so small, that the value of agents is negligible.

The problem is that for a lot of other very interesting functionality we need agents. For example, to create an application consistent image we need to interact with the application to create that image. And a secure, robust, and reliable way to interact with an application is to use agents.

Of course, as environments add network APIs that provide some if not all of the functionality of an agent, we need to continue to evaluate whether an agent makes any sense.

So for example, when we built SMVI, we specifically did not install an agent either in the Console OS or inside the guest, because VMware provides all of the necessary APIs.

On the other hand, when we integrated OSSV with VMware we chose to deploy an agent in the Console OS because we could deliver a significant performance boost over agentless or VCB approaches.

The Arbitrage of Backup Part I: The Backup Cost Model

Ever since I wrote about how deduplication is re-defining backup for VMware environments, I began to wonder if I could describe the economics of backup in a set of simple enough equations that would explain why this made sense.

So here's an attempt.

Let's assume we have a dataset D.

And the size of the dataset Size(D).

And the cost of storing a byte of data is CostPerByte.

Then the cost of storing a D is Size(D) * CostPerByte and we'll call that value CostToStore.

The cost of storing a full backup copy of D then is CostToStore.

Now the value of a backup is really hard to derive. The problem is that the value of a backup declines over time because it becomes increasingly unlikely that a restore will take place from that backup.

Furthermore, the notion of value is very subjective. So rather than introduce some magic constants let's make the simplifying assumption that CostToStore is a good proxy for the value of the data. After all if the value of the data is low, then the CostToStore should also be low.

Okay so then:

ValueOfBackup(Age) = CostToStore/Age

This equation says that for any backup image, the value of the backup is equal to the cost of storing and inversely proportional to the age.

So now that I have all of the terms, then I can create an equation that says that the number of backups stored is the following:

NumberOfBackups = TotalAmountOfMoney/CostToStore

And here is where the arbitrage comes in. To store more backup copies you need to reduce the CostToStore.

So expanding the equation a little bit:

NumberOfBackups = TotalAmountOfMoney/(Size(D)*CostPerByte)

And we'll call it The Backup Cost Model.

In my next post I'll discuss why this is a good model.

Tape, Roman Chariots and Data Management

There is this great story that describes how the railroad gauge of 4 feet and 8.5 inches derives from the Imperial Roman War Chariot. The point behind that story is that we continue to do things long after the reason  for doing them has evaporated.

It turns out that tape is playing the same role in the data center.

So it turns out that backup is the only ubiquitous data protection policy.  Regardless of how small you are, protecting data from operational errors (human frailty and software frailty) is something you do. The first time you accidentally delete an important file and  the first time an upgrade fails is the moment a backup procedure gets created. And that tends to happen within a few months of acquiring a computer system.

Because backup is so ubiquitous it's very highly automated. There is an awe inspiring amount of software developed and sold for backup.

And once you have a backup policy, the next thing you start to worry about is disaster recovery. Namely, how do you recover all of the data after a system or site goes belly up. And the obvious thing to do is leverage the existing backup infrastructure to get the data back after a disaster. And then once you've got disaster recovery covered, the next thing to worry about is long term archival. And it's logical to just leverage the same backup infrastructure, just hold onto the backups for longer...

So far so good.

And back in the Ur-days when tape was a lot cheaper than disk, the only way to economically do backup was to use tape.

Okay so back when you just did backup and restore, you probably chose to do a single full backup followed by some large number of incremental backups. You probably held onto the tapes for a modest amount of time and you had a small number of tapes. Life was good.

But then you wanted to leverage Disaster Recovery so you had to start taking more full backups more frequently. Why? Because doing full recovery from a set of incremental backups is simply not practical. And all of a sudden you had more tapes, and things started to take more time, and you needed bigger pipes.

And then you wanted to do long term archival, and all of a sudden you had  even more tapes staying around for longer.And now you had a whole set of policies that you needed to put into place to recycle tapes.

So now what?

Well now we have a set of processes and procedures that exist because that's the way tape made things exist. We have servers that have enough IO bandwidth to be able to do full backups on a regular basis. We have network infrastructures that only exist to sustain the vast amount of data that needs to be moved for backup. We've created and sized an entire infrastructure predicated on moving all of the data very frequently because that's how tape made us do it...

Okay...

But here's where it gets insidious, we know look at the mess that tape has created, and instead of asking the question:

Is a data protection infrastructure predicated on creating whole copies on a regular basis flawed?

We ask the question:

How can I make creating and storing full copies more efficient?

Rather than use new technologies that make backup and restore efficient, disaster recovery efficient and archive efficient and effective, we continue to rely on creating vast amounts of redundant copies, keeping those copies around for a while and praying that deduplication will save our skins.

And when folks argue that deduplication really should be done on the primary, and that we should look at ensuring that as little data moves we act surprised. Data management is about managing thousands of whole copies, that's the way it's always been ...

At some point, 50 years from now people will ask why do we expend so much energy eliminating redundant data when we could have avoided creating the data in the first place. And someone will write an article about the role of tape and whole images and people will laugh...

SMVI part IV: Enter Snapshots

In part I of my series on SMVI, I described  the backup challenge caused by the need to move more data in the same amount of time using fewer resources.

In part II of my series on SMVI, I described the disaster recovery challenge..

In part IV of my series on SMVI and the data protection challenges of a VMware environment, I want to transition to the value of the NetApp Snapshot.

So what are the requirements for a great backup technology:

1. Able to quickly create the backup copy regardless of the size of the datastore
2. Able to keep the backup copies around for a while
3. Minimal amount of additional equipment beyond the additional storage
4. Each backup image should consume no more storage than the incremental difference with the last backup.
5. No CPU, Memory bandwidth,IO impact on the server
6. No performance impact on the storage.
7. Works the same way regardless of the type of VMware datastore
8. Able to restore data quickly.
9. Can be extended to a DR strategy with minimal additional equipment.
10. Can be used to create VMware consistent backups.

But why are these all necessary?

1. Shrinking backup windows, data explosion.
2. The whole point of backup is to keep copies around for a while.
3. Backup is a form of insurance, so cost is important.
4. See #3
5. As I blogged about, the bigger the impact the less you can consolidate ...
6. If backups impact the storage, then we have the same set of issues as #5
7. Different VMware environments will required different kinds of datastores, having to have multiple distinct backup strategies for each is a management cost.
8. The whole point about taking a backup is to be able to do a restore. The speed of a restore affects how quickly you can go back to business as usual.
9. Every backup strategy is part of a DR strategy.
10. Because crash consistency has risks on recovery failing.

Guess what, a NetApp Snapshot satisfies all of these requirements, and a NetApp Snapshot is the only technology that satisfies all of these requirements. Some technologies satisfy some of these requirements, but only a NetApp Snapshot satisfies them all.

So how do Snapshots work? There is an astounding amount of EMC FUD on the net that describes how a NetApp Snapshot works, and not as much NetApp technical documentation. At the core, the story from Hitz's paper remains the same (although there have been about 15,000 man years of engineering done since his paper). We are able to create a snapshot in constant time because we have a map of the blocks that are allocated on disk. A snapshot is really just a copy of the block map rather than the actual disk blocks.

Because a snapshot is a point-in-time copy of a Flexible Volume, a Snapshot can be used to create a point-in-time copy of any kind of storage object within the Flexible Volume: LUNs or Files. For a good technical paper on Flexible Volumes I recommend the recently published Usenix paper.

So let's walk through each one of those data protection requirements and see how NetApp storage can satisfy them::

1. Given the fact that a datastore is mapped to a Flexible Volume, and a snapshot of a Flexible Volume takes the same amount of time regardless of size, we can do that. Have to backup 2 TB datastore or 200GB datastore, it takes the same amount of time, practically 0.
2. You can keep 255 snapshots per Flexible Volume and you can have up to 500 Flexible Volumes per high-end controller.
3. The controller is all you need to take a snapshot.
4. Snapshots from their inception create a point in time copy of a Flexible Volume that only retains the blocks that changed since the last point in time copy.
5. Since no data flows through the server, this is obvious.
6. We've gone over that one before.
7. Snapshots are of a Flexible Volume and work the same regardless of the kind of datastore.
8. Because we don't have to move the data over the network, but only from one place on disk to another place on disk, we have a ridiculous performance advantage compared to other backup solutions, but it gets better with some tricks I'll describe later.
9. SnapMirror allows one to replicate the Snapshots to a remote site.and because the remote site can also be used as a primary storage system ...
10. Yes, that's what SMVI does.

In my next post in this series I'll explain how SMVI uses NetApp Snapshot to create VMware consistent backups..

Removing a document does not change the facts

Unfortunately EMC has chosen to remove the white paper I talked about

Deleting said white paper does not change the facts.

The Clariion has fake snapshots. They are fake because they are not:

- performance neutral

- space and time efficient

Using Clariion snapshots storage architects have to choose between getting there on time and driving sober. NetApp's real snapshots, on the other hand, allow storage architects to offer to their application customers

- Lower RPO because you can take more snapshots with low space utilization

- Lower RTO because less has to be moved

without having to choose between performance and data protection. 

No  amount of document management on EMC's web site can change those facts.

Since you can no longer get an EMC document to tell you the truth about performance crippling Clariion snapshots, I suggest you go to the SPC web site where you can see the data behind this chart:

Using Clariion's fake snapshots is performance crippling and not recommended for workloads where performance is important.

Deleting documents does not hide the truth.

Redefining Backup for VMware

NetApp today shipped SnapManager for Virtual Infrastructure (SMVI), a product that when combined with real snapshots, NetApp's deduplication technology, primary storage capabilities and SnapMirror technology, fundamentally re-writes the rules of what backup infrastructures should be.

To give you a flavor of what I am talking about...

Consider a traditional VMware backup infrastructure looks like this:

The problems with this infrastructure are obvious.

1. Data has to move a lot. The movement through the various infrastructure components costs time and it also costs money as you have to scale the IO throughput to handle the data flood.
2. Lots of different moving parts. You have to manage your primary storage, and if you are not using NetApp, another secondary storage infrastructure and finally a completely separate backup software infrastructure.
3. And oh by the way, I ran out of pixels to be able to show the DR strategy.

The NetApp solution on the other hand looks like this:

Once you've deployed your ESX server on NetApp storage, all you have to do is install SMVI on the same machine or a separate machine from Virtual Center and you are done.

No separate servers, no separate software, and no separate infrastructure. The backup images are stored on the storage system itself.

There is no data copy when you create a backup, and there is a minimal data copy on a restore. That means faster and cheaper backups.

And oh-by-the way the DR solution is pretty simple too...

And finally, unlike the folks at EMC, you can use our snapshots without worrying about performance...

A Reminder of the Value of REAL Snapshots

One of the more irritating aspects about being a technologist instead of a marketing professional is that I have to deal with the marketing folks of other companies. I don't have to deal with them personally, of course, but I do have to deal with their misappropriation of technology terms.

And the most irritating misappropriation is the use of the term snapshot to describe any technology that creates a point-in-time copy of a range of disk blocks.

Ideally, I wouldn't have to write this blog entry, because every storage customer out there would understand that a snapshot was, by definition:

- space efficient

- performance neutral

And that, therefore, using storage snapshots a customer would have:

- improved RPO because they can have more copies of data

- improved RTO because they can move less data on a restore

- no need to trade off RPO/RTO against performance of the system

But as I am reminded again and again, and today by no less of an authority than EMC's own white paper, if you are using a Clariion a snapshot is

- performance crippling.

As EMC says on page 8:

For Exchange, snapshot replication on Clariion is not recommended because it can impact performance.

As a storage architect you are forced to choose between your recovery time and application performance, which is kind of like choosing between arriving on time and driving sober.

So if you happen to be an EMC customer, or unfamiliar with real snapshots you may find the rest of this post interesting.

First a history lesson

When NetApp's founders invented WAFL they created a unique mechanism for creating a point-in-time copy of a collection of blocks that was constant-time with respect to the number of blocks, and extraordinarily space efficient. They chose to call that technology a snapshot.

What was true then, remains true today, Snapshot do not impose any meaningful overhead on the storage system in terms of performance, and are extremely space efficient.

Net-net...

Using a NetApp Snapshot you can have more recovery points without having to worry about the performance impact on your storage array and because of their space efficiency, not worry about the cost.

But that's not all.

Having snapshots on the primary system can also improve your recovery time.

Why?

Well it turns out, as I've said here, the amount of data that has to be moved is the key determinant in restore performance. And if you have a business critical application the amount of time you are down waiting for a restore to be completed can be measured in real dollars.

So if the data is on the primary storage system, and the storage vendor has a product like SnapManager for Exchange restore times can be extraordinarily fast.

How Dedup and Snapshot Change Backup for VMware

One of the more interesting conversations I've had recently was how NetApp end-to-end deduplication and our replication technology fundamentally changes how you think about backup in a virtualized environment.

The core challenge in a VMware or HyperV environment is that the restore is extraordinarily painful because of the amount of data that needs to be moved. You are no longer just restoring a file, but an entire image including OS, application image, and data.

And the single biggest bottleneck to a restore is the data transfer from the repository to wherever the primary is connected.

The typical infrastructure has data flowing from some primary system, through a backup server to some backup media.  The notion being that primary storage is expensive, and the backup target is much cheaper, making the restore time acceptable from an overall cost perspective.

If data transfer is so painful, then why transfer the data? Why not keep it around on the local system?

Well there are three reasons:

1. Cost of keeping extra copies
2. Performance cost of keeping extra copies on the primary
3. Recovery after a disaster.

It turns out that NetApp Snapshot and our Deduplication fundamentally change the cost equation because of how they work together. NetApp Snapshot technology only keeps a point-in-time copy of the blocks that have changed inside of a volume. NetApp Deduplication eliminates any duplicate data between different storage containers within a volume.

And of course, unlike other snapshot technologies, a NetApp Snapshot is performance neutral so using them is not a space-time tradeoff.

And it turns out that deduplication has a small performance impact as well.

What does this mean? Well it means that the amount of copies a storage architect can keep on the primary is significantly higher than using other storage arrays. More snapshots, means less likely to go to a remote machine to restore data and that translates into faster recovery times!

I can see someone point out that the primary storage system will still be more expensive even if snapshots and deduplication is used. But here's the rub, the secondary  system costs money and the restore costs money in terms of lost productivity during the restore. To make this worthwhile the primary system has to be cheaper than the cost of the secondary system AND the restore cost. This is a much lower bar to jump over.

But what about a disaster?

Enter SnapMirror our volume mirroring technology. Well it turns that SnapMirror allows you to replicate an entire volume and all of it's snapshots. It also turns out that SnapMirror will only replicate the deduplicated data.

What this means is that cost of the network and the secondary system is lower than using alternative technologies because the amount of data is less, and the amount of computation is less (no deduplication on the secondary).

It also means that the recovery after a disaster is faster, because the data is online and can be directly accessed rather than stored on Tape or a VTL and then restored to a primary storage system.

But that's not all ...

Because you are keeping the backup data on the primary storage system and are not moving the data other than to replicate for DR, this also simplifies the backup infrastructure. There are no complex highly available backup servers that need to be setup and managed.

And because you are keeping all of the backup data with the primary copy, there is no painful need to index and track data on remote devices.

So in conclusion

You get faster restores, faster recovery after a disaster, less complexity because there is no backup process and backup system to manage all because the cost of doing this got cheaper...

Pretty cool...

Introducing Deduplication for EMC

NetApp today proudly announces the introduction of the V-Series support of Deduplication for EMC storage. Finally NetApp allows EMC customers to have a single enterprise wide deduplication layer across all of their storage tiers. A  deduplication layer that is almost as easy to use as the one NetApp's 2500 customers have already been using and  deployed on over 10,000 systems.

To get the savings of deduplication  EMC customers will just have to insert a V-Series between their existing EMC storage and their servers.

For NetApp customers adding new hardware to get new features is weird. For EMC customers, who have to deal with 10 different EMC platforms, this should be par for the course.

Having taken advantage of the obvious benefit of deduplication, my hope is that customers will notice that they have now introduced a single unified storage virtualization layer into their infrastructure.

So their world that looked like this:

A world that had separate and distinct infrastructures

• that have to be managed, and organized and planned separately
• that tend to be over-provisioned and underutilized.

That bad world, will now look something like this:

With a single enterprise wide virtualization layer that has a unified approach to how data is protected and provisioned. A layer that can be accessed via all storage protocols and can support any application.

Using that virtualization layer EMC customers will be able to eliminate redundant data before it gets created with other technologies like SnapShot and FlexClone. These EMC customers will also be able to reap the benefits of using our policy automation layer to do simplified provisioning and data protection.

So, EMC customers, although the reason you will bring us in will be to save money, we hope you'll keep us because we dramatically reduce the operational overhead of your infrastructure.