Extensible NetApp: Thinking outside of the box

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

SMVI part V: Virtual Machine Consistent Backup

I promise to get to what SMVI does but I need to explain one more thing... For those who understand the term Virtual Machine Consistent Backup, just skip this post.

Two forms of Backup

There are two forms of backup, crash consistent and consistent.

A crash consistent backup is the moral equivalent of pulling the plug on a server and then backing up the data. The state of the data that is being backed up with respect to the users of the data is indeterminate.

A consistent backup is the moral equivalent of first performing a clean shutdown of the application, then the server and then performing the backup.

Of course, in practice, a consistent backup can be performed without shutting down an application or server by putting the application and server into hot backup mode while the backup takes place.

What's the difference?

A crash consistent backup may result in:

1. Application data loss during recovery
2. A backup that can not be recovered from
3. Longer recoveries because of application and server consistency checks
4. Has no additional impact to the application beyond moving the data
5. Simple to do because you just copy the data with no concern for its state

A consistent backup will

1. Have no data loss during recovery
2. Will always allow recovery
3. Enables very quick recoveries because there is no consistency checking
4. Enables finer grained recoveries.
5. While the backup is in process, application performance may be impacted.
6. Complex to do because you need to perform some co-ordination between the server, the application and the backup process.

So given that consistency is better than crash consistency the challenge is how to make consistent backups as easy to perform as crash consistent backups and that's where SMVI fits in ...

What is a Virtual Machine Consistent Backup?

A virtual machine consistent backup is the equivalent to performing a server consistent backup. And by that we mean that any in-flight I/Os are committed to stable storage before the backup is performed.

in the case of VMware, the way that is achieved is by using VMware snapshots.

For HyperV you would use the VSS framework. 

SMVI part IV: Enter Snapshots

In part I of my series on SMVI, I described  the backup challenge caused by the need to move more data in the same amount of time using fewer resources.

In part II of my series on SMVI, I described the disaster recovery challenge..

In part IV of my series on SMVI and the data protection challenges of a VMware environment, I want to transition to the value of the NetApp Snapshot.

So what are the requirements for a great backup technology:

1. Able to quickly create the backup copy regardless of the size of the datastore
2. Able to keep the backup copies around for a while
3. Minimal amount of additional equipment beyond the additional storage
4. Each backup image should consume no more storage than the incremental difference with the last backup.
5. No CPU, Memory bandwidth,IO impact on the server
6. No performance impact on the storage.
7. Works the same way regardless of the type of VMware datastore
8. Able to restore data quickly.
9. Can be extended to a DR strategy with minimal additional equipment.
10. Can be used to create VMware consistent backups.

But why are these all necessary?

1. Shrinking backup windows, data explosion.
2. The whole point of backup is to keep copies around for a while.
3. Backup is a form of insurance, so cost is important.
4. See #3
5. As I blogged about, the bigger the impact the less you can consolidate ...
6. If backups impact the storage, then we have the same set of issues as #5
7. Different VMware environments will required different kinds of datastores, having to have multiple distinct backup strategies for each is a management cost.
8. The whole point about taking a backup is to be able to do a restore. The speed of a restore affects how quickly you can go back to business as usual.
9. Every backup strategy is part of a DR strategy.
10. Because crash consistency has risks on recovery failing.

Guess what, a NetApp Snapshot satisfies all of these requirements, and a NetApp Snapshot is the only technology that satisfies all of these requirements. Some technologies satisfy some of these requirements, but only a NetApp Snapshot satisfies them all.

So how do Snapshots work? There is an astounding amount of EMC FUD on the net that describes how a NetApp Snapshot works, and not as much NetApp technical documentation. At the core, the story from Hitz's paper remains the same (although there have been about 15,000 man years of engineering done since his paper). We are able to create a snapshot in constant time because we have a map of the blocks that are allocated on disk. A snapshot is really just a copy of the block map rather than the actual disk blocks.

Because a snapshot is a point-in-time copy of a Flexible Volume, a Snapshot can be used to create a point-in-time copy of any kind of storage object within the Flexible Volume: LUNs or Files. For a good technical paper on Flexible Volumes I recommend the recently published Usenix paper.

So let's walk through each one of those data protection requirements and see how NetApp storage can satisfy them::

1. Given the fact that a datastore is mapped to a Flexible Volume, and a snapshot of a Flexible Volume takes the same amount of time regardless of size, we can do that. Have to backup 2 TB datastore or 200GB datastore, it takes the same amount of time, practically 0.
2. You can keep 255 snapshots per Flexible Volume and you can have up to 500 Flexible Volumes per high-end controller.
3. The controller is all you need to take a snapshot.
4. Snapshots from their inception create a point in time copy of a Flexible Volume that only retains the blocks that changed since the last point in time copy.
5. Since no data flows through the server, this is obvious.
6. We've gone over that one before.
7. Snapshots are of a Flexible Volume and work the same regardless of the kind of datastore.
8. Because we don't have to move the data over the network, but only from one place on disk to another place on disk, we have a ridiculous performance advantage compared to other backup solutions, but it gets better with some tricks I'll describe later.
9. SnapMirror allows one to replicate the Snapshots to a remote site.and because the remote site can also be used as a primary storage system ...
10. Yes, that's what SMVI does.

In my next post in this series I'll explain how SMVI uses NetApp Snapshot to create VMware consistent backups..

Redefining Backup for VMware

NetApp today shipped SnapManager for Virtual Infrastructure (SMVI), a product that when combined with real snapshots, NetApp's deduplication technology, primary storage capabilities and SnapMirror technology, fundamentally re-writes the rules of what backup infrastructures should be.

To give you a flavor of what I am talking about...

Consider a traditional VMware backup infrastructure looks like this:

The problems with this infrastructure are obvious.

1. Data has to move a lot. The movement through the various infrastructure components costs time and it also costs money as you have to scale the IO throughput to handle the data flood.
2. Lots of different moving parts. You have to manage your primary storage, and if you are not using NetApp, another secondary storage infrastructure and finally a completely separate backup software infrastructure.
3. And oh by the way, I ran out of pixels to be able to show the DR strategy.

The NetApp solution on the other hand looks like this:

Once you've deployed your ESX server on NetApp storage, all you have to do is install SMVI on the same machine or a separate machine from Virtual Center and you are done.

No separate servers, no separate software, and no separate infrastructure. The backup images are stored on the storage system itself.

There is no data copy when you create a backup, and there is a minimal data copy on a restore. That means faster and cheaper backups.

And oh-by-the way the DR solution is pretty simple too...

And finally, unlike the folks at EMC, you can use our snapshots without worrying about performance...

A Reminder of the Value of REAL Snapshots

One of the more irritating aspects about being a technologist instead of a marketing professional is that I have to deal with the marketing folks of other companies. I don't have to deal with them personally, of course, but I do have to deal with their misappropriation of technology terms.

And the most irritating misappropriation is the use of the term snapshot to describe any technology that creates a point-in-time copy of a range of disk blocks.

Ideally, I wouldn't have to write this blog entry, because every storage customer out there would understand that a snapshot was, by definition:

- space efficient

- performance neutral

And that, therefore, using storage snapshots a customer would have:

- improved RPO because they can have more copies of data

- improved RTO because they can move less data on a restore

- no need to trade off RPO/RTO against performance of the system

But as I am reminded again and again, and today by no less of an authority than EMC's own white paper, if you are using a Clariion a snapshot is

- performance crippling.

As EMC says on page 8:

For Exchange, snapshot replication on Clariion is not recommended because it can impact performance.

As a storage architect you are forced to choose between your recovery time and application performance, which is kind of like choosing between arriving on time and driving sober.

So if you happen to be an EMC customer, or unfamiliar with real snapshots you may find the rest of this post interesting.

First a history lesson

When NetApp's founders invented WAFL they created a unique mechanism for creating a point-in-time copy of a collection of blocks that was constant-time with respect to the number of blocks, and extraordinarily space efficient. They chose to call that technology a snapshot.

What was true then, remains true today, Snapshot do not impose any meaningful overhead on the storage system in terms of performance, and are extremely space efficient.

Net-net...

Using a NetApp Snapshot you can have more recovery points without having to worry about the performance impact on your storage array and because of their space efficiency, not worry about the cost.

But that's not all.

Having snapshots on the primary system can also improve your recovery time.

Why?

Well it turns out, as I've said here, the amount of data that has to be moved is the key determinant in restore performance. And if you have a business critical application the amount of time you are down waiting for a restore to be completed can be measured in real dollars.

So if the data is on the primary storage system, and the storage vendor has a product like SnapManager for Exchange restore times can be extraordinarily fast.

Using Protection Manager, SnapDrive for Windows and SnapManager for Exchange

In my previous post I described at a high level the power of the recently released SnapManager, SnapDrive and Protection Manager products.

To give you a flavor for the workflow of the tools let me walk you through the process of how they are used. *

First the storage architect or disk to disk backup architect defines a set of policies that can be used using Protection Manager.

To create a new policy the storage architect first selects one of the predefined policies

and then modifies that policy to meet the specific requirements of his organization.

Now the policy is ready to be used by an Exchange administrator.

Before you can use SnapManager for Exchange you need to configure SnapDrive for Windows to talk to a DFM Server that has Protection Manager license.

To use the policy the Exchange administrator brings up SnapManager for Exchange

and where before he could only configure local Snapshot and SnapMirror schedules, now the administrator has additional data protection policies that include disk-to-disk backup.

The Exchange administrator  then simply assigns policies to Exchange storage groups. As each policy gets assigned to a storage group, SnapManager for Exchange through SnapDrive for Windows creates a new dataset  in Protection Manager that contains the LUNs that contain the Exchange storage group.

Each dataset that is created does not, however, have any secondary storage resources assigned to it. That has to be done by the storage administrator.

To assign the physical resources, the storage architect would use Protection Manager to find the new datasets. This is made easy by Protection Manager, because the datasets would appear as a dataset that was out of conformance with respect to the specified policy.

The storage architect then assigns the right resource pool to the storage group. Once that is done, Protection Manager will  then automatically create the backup relationships, run the schedules, and configure any additional secondary storage objects.

Once that is done the Exchange storage group is protected.

* For more info on datasets, policies and resource pools check this paper we published in LISA 2007.

* For more info on how SnapManager and SnapDrive work check this article from Tech ONTAP.

Why are we integrating SnapManager for Exchange, SnapDrive for Windows, and Protection Manager?

SnapDrive for Windows 6.0 and Protection Manager, part of the DFM 3.7 release, and SnapManager for Exchange 5.0  are the first tangible artifacts that demonstrate the power of NetApp's Integrated Data Management Vision.

Three years ago in a meeting at NetApp, a bunch of us had the radical idea that if you could enable server and application administrators to manage their own data you could unlock tremendous amounts of productivity by eliminating processes and procedures in the data center that existed to overcome limitations in storage management systems. 

From that observation came two significant technological breakthroughs. The first was the introduction of our policy based management infrastructure as part of the Protection Manager product and the second is the integration of our very successful SnapDrive for Windows and SnapManager for Exchange products with that policy based infrastructure.

Superficially Protection Manager is a GUI for managing NetApp replication. But that, as they say, completely misses the point. If you think of Protection Manager as just a GUI, then you'll probably wonder why we ever bothered with the integration at all. And you would be right in wondering.

Protection Manager, in reality, is a policy automation layer coupled with a very capable delegation model. These two technologies allow storage architects to define policies for how storage should be used and in parallel allow other people to create storage based on those very same policies. In effect, the storage architect does the hard work of figuring out how he wants people to use his storage once, and leaves the relatively simple task of provisioning storage to other people secure in the knowledge that they can not make a mistake.