Extensible NetApp: Thinking outside of the box

Backup, the new storage tiers and real snapshots

When I first described my new terminology for storage tiers, Tim Burlowski, asked how backup and recovery fit into that model.

It turns out that that is an interesting question.

Backup, the only working ILM/HSM solution

Step 10 feet away, and you realize that backup is HSM.

The backup process creates cold data, the backup image. The backup software them moves the backup copy to cheaper tier of storage. When a restore is performed, the backup software moves the backup copy from the cheaper tier to a more expensive tier.

The question on the table, then, is where does this form of HSM still make sense?

Captive IOPS

One obvious place where HSM as part of backup process is for the backup of data on a Captive IOPS tier. The tier is built using very expensive storage. Storing cold data that will probably never get accessed seems odd and a waste of money.

But that's not necessarily true.

If the Captive tier is built using disk drives, and the application  has a high IOPS data density, a lot of the capacity of the disks is  unused. The cheapest place to store backup copies, in that case, would be on the same disks you're using for primary data.

If the Captive tier is built using SSD's like EMC storage, then it makes a lot of sense to use a backup solution to move backup images off of the flash onto cheaper storage. If the application has a low IOPS data density, then a better approach is to use flash cache in front of disk.

If the Captive tier is built using a flash cache in front of disk, the cheapest place to store the backup copies may be on the disk drives.

More generally, the backup methodology of a captive tier made up of flash cache and disk, will be the same as the backup methodology of the capacity tier.

So what about the capacity tier and backup?

In the general case, the capacity tier will be backed up much in the same way that it always has been.

A capacity tier that has real snapshots can eliminate the HSM part of the backup process.

In fact, in an earlier series on backup I proved that for a VMware infrastructure, the cheapest place to store backup copies was on the same disks that stored the active data.

Now I want to generalize that comment for all storage..

If you consider IOPS density, disk drives must contain large amounts of cold data. A subset of the cold data in a data center is the backup data. That data can either be stored on a separate set of disks or on the same disks that are being used to serve IOPS.

To store the data on the same disks the following things need to be true

1. Creating the backup image must consume minimal IOPS
2. The existence of the backup image must impose no performance penalty
3. Be able to store a large number of images.
4. The storage must be configured using some form of RAID-6 or RAID 1-0
5. The cost of the backup images must be cost effective

It turns out that for NetApp systems all five are true.

So as disk capacities expand, storage architects who have the option to use real snapshots will stop moving data off of the capacity tier to some cheaper tier.

Said in a slightly more poetic way:

The final resting place for data on disk will be the disk where the data first got created.

The net effect will be a much simpler and more cost effective storage infrastructure to support backup.

One minor addendunm added after I wrote this post.

As Martin G says in the comments below, having the backup copies on the local disks doesn't protect you from site failures. So, of course, some form of storage replication is required. Thankfully NetApp has a cool technology called SnapMirror that replicates all of the snapshots to a remote destination in a space efficient way.

In an earlier post on VMware and backup I show how all of this fits together.

I should have included this need to replicate the data in my original blog post. So I am rectifying that error now.

SMVI part V: Virtual Machine Consistent Backup

I promise to get to what SMVI does but I need to explain one more thing... For those who understand the term Virtual Machine Consistent Backup, just skip this post.

Two forms of Backup

There are two forms of backup, crash consistent and consistent.

A crash consistent backup is the moral equivalent of pulling the plug on a server and then backing up the data. The state of the data that is being backed up with respect to the users of the data is indeterminate.

A consistent backup is the moral equivalent of first performing a clean shutdown of the application, then the server and then performing the backup.

Of course, in practice, a consistent backup can be performed without shutting down an application or server by putting the application and server into hot backup mode while the backup takes place.

What's the difference?

A crash consistent backup may result in:

1. Application data loss during recovery
2. A backup that can not be recovered from
3. Longer recoveries because of application and server consistency checks
4. Has no additional impact to the application beyond moving the data
5. Simple to do because you just copy the data with no concern for its state

A consistent backup will

1. Have no data loss during recovery
2. Will always allow recovery
3. Enables very quick recoveries because there is no consistency checking
4. Enables finer grained recoveries.
5. While the backup is in process, application performance may be impacted.
6. Complex to do because you need to perform some co-ordination between the server, the application and the backup process.

So given that consistency is better than crash consistency the challenge is how to make consistent backups as easy to perform as crash consistent backups and that's where SMVI fits in ...

What is a Virtual Machine Consistent Backup?

A virtual machine consistent backup is the equivalent to performing a server consistent backup. And by that we mean that any in-flight I/Os are committed to stable storage before the backup is performed.

in the case of VMware, the way that is achieved is by using VMware snapshots.

For HyperV you would use the VSS framework. 

SMVI part IV: Enter Snapshots

In part I of my series on SMVI, I described  the backup challenge caused by the need to move more data in the same amount of time using fewer resources.

In part II of my series on SMVI, I described the disaster recovery challenge..

In part IV of my series on SMVI and the data protection challenges of a VMware environment, I want to transition to the value of the NetApp Snapshot.

So what are the requirements for a great backup technology:

1. Able to quickly create the backup copy regardless of the size of the datastore
2. Able to keep the backup copies around for a while
3. Minimal amount of additional equipment beyond the additional storage
4. Each backup image should consume no more storage than the incremental difference with the last backup.
5. No CPU, Memory bandwidth,IO impact on the server
6. No performance impact on the storage.
7. Works the same way regardless of the type of VMware datastore
8. Able to restore data quickly.
9. Can be extended to a DR strategy with minimal additional equipment.
10. Can be used to create VMware consistent backups.

But why are these all necessary?

1. Shrinking backup windows, data explosion.
2. The whole point of backup is to keep copies around for a while.
3. Backup is a form of insurance, so cost is important.
4. See #3
5. As I blogged about, the bigger the impact the less you can consolidate ...
6. If backups impact the storage, then we have the same set of issues as #5
7. Different VMware environments will required different kinds of datastores, having to have multiple distinct backup strategies for each is a management cost.
8. The whole point about taking a backup is to be able to do a restore. The speed of a restore affects how quickly you can go back to business as usual.
9. Every backup strategy is part of a DR strategy.
10. Because crash consistency has risks on recovery failing.

Guess what, a NetApp Snapshot satisfies all of these requirements, and a NetApp Snapshot is the only technology that satisfies all of these requirements. Some technologies satisfy some of these requirements, but only a NetApp Snapshot satisfies them all.

So how do Snapshots work? There is an astounding amount of EMC FUD on the net that describes how a NetApp Snapshot works, and not as much NetApp technical documentation. At the core, the story from Hitz's paper remains the same (although there have been about 15,000 man years of engineering done since his paper). We are able to create a snapshot in constant time because we have a map of the blocks that are allocated on disk. A snapshot is really just a copy of the block map rather than the actual disk blocks.

Because a snapshot is a point-in-time copy of a Flexible Volume, a Snapshot can be used to create a point-in-time copy of any kind of storage object within the Flexible Volume: LUNs or Files. For a good technical paper on Flexible Volumes I recommend the recently published Usenix paper.

So let's walk through each one of those data protection requirements and see how NetApp storage can satisfy them::

1. Given the fact that a datastore is mapped to a Flexible Volume, and a snapshot of a Flexible Volume takes the same amount of time regardless of size, we can do that. Have to backup 2 TB datastore or 200GB datastore, it takes the same amount of time, practically 0.
2. You can keep 255 snapshots per Flexible Volume and you can have up to 500 Flexible Volumes per high-end controller.
3. The controller is all you need to take a snapshot.
4. Snapshots from their inception create a point in time copy of a Flexible Volume that only retains the blocks that changed since the last point in time copy.
5. Since no data flows through the server, this is obvious.
6. We've gone over that one before.
7. Snapshots are of a Flexible Volume and work the same regardless of the kind of datastore.
8. Because we don't have to move the data over the network, but only from one place on disk to another place on disk, we have a ridiculous performance advantage compared to other backup solutions, but it gets better with some tricks I'll describe later.
9. SnapMirror allows one to replicate the Snapshots to a remote site.and because the remote site can also be used as a primary storage system ...
10. Yes, that's what SMVI does.

In my next post in this series I'll explain how SMVI uses NetApp Snapshot to create VMware consistent backups..

SMVI post III: A bit of background

Before I can get into how SMVI actually works, I want to remind folks about how VMware storage and NetApp work together. For folks already familiar with VMware and NetApp you can wait for my next post where I describe how NetApp Snapshots eliminate the backup problem. 

VMware Infrastructure and NetApp Storage

Consider this picture of a physical server connected to NetApp storage device.

Starting from the right moving left let's build up to the final picture.

The left most layer shows a collection of virtual machines running some guest OS. Each virtual machine has a set of CPU, Memory and storage resources. To the guest OS running in the virtual machine, the storage resources appear as if they were one or more locally attached SCSI disk drives.

Moving a little bit more to the left, we observe that the disks are really a kind of file called a vmdk contained within a datastore. A datastore itself is a virtualized storage pool.

If the ESX server is connected to the array using FC or ISCSI, a filesystem must be created within the datastore. VMware currently only supports VMFS. In that case the datastore is called, unsurprisingly, a VMFS datastore.

If the ESX server is connected to the array using NFS, the datastore is just an accounting structure. The file system within the datastore is implemented by the array. The datastore in this case is called a NFS datastore.

Finally all the way to the left we have the NetApp storage objects. In the case of FC, the array is configured with a LUN within a FlexVol that is contained with an aggregate. In the case of NFS, the datastore is contained within a Flexible Volume which is contained within an aggregate.

Okay so that's nice, but why was that important?

Now remember, the virtual machine has a bunch of disks that are mapped through multiple layers of virtualization to a bunch of physical blocks. SMVI is used to perform backups and recovery of those virtual machine disks using snapshots. In a NetApp infrastructure, you can only take a snapshot of a Flexible Volume. So to take a snapshot of a virtual disk you have to understand the mapping between the virtual disk and the containing Flexible Volume.

Whew.

In my next post, I'll explain how NetApp Snapshot address the backup challenge.

Another take on virtualization

Dave, in his excellent post on server virtualization, asked the meta-question: What is really going on?

So here's my thinking.

Moore's law has produced ever increasing numbers of transistors and ever increasing clock frequencies that software developers have been able to exploit by doing nothing. In fact, the increasing number of transistors  and increasing clock frequency made it possible to add even more value in software faster.

From a consumer of software perspective, the value of software increased over time even though the cost stayed constant and in fact declined in real-dollars over time. It was the speed with which transistor counts doubled and clock frequencies  increased and the ease with which they were both exploited that created the high-tech industry.

Now it turns out that Intel et al have been unable to increase clock cycles on a per-core basis so they have used the increasing transistor count to add more cores.

Traditionally software developers would experience a speed up in software just by moving to the new hardware leaving more cycles to add more value. However, in this multi-core world, the only way to leverage the increasing transistor count to get performance increases was to write Multi-Threaded (MT) code that was Multi-Processor (MP) safe.

It turns out that writing MT and MP safe code is harder than writing code that ran on a single core processor. It also turns out that whereas in the past, performance improvements were transparent to the pre-existing software, increasing performance with multiple CPU cores requires changing pre-existing software.

The net effect was that software was unable to leverage the new transistors as easily as in the past.

So cores went idle. And it became harder to add more value to software. The free compute cycles of yesteryear evaporated.

And then, all of a sudden, data centers had these idle compute resources.

And some folks assumed that these were real idle CPU cycles. They, incorrectly, assumed that even if using the cycles was easy they would remain idle.

What really happened was that because traditional software could not transparently leverage the increased transistor count, the cost of a small increment in *usable* compute cycles, increased. In plain English, to run any new application you needed a new x86 server which for the most part was idle. So let's say you need 1% of a CPU, you had to buy 100% of a CPU, of which 99% was idle. But the fixed cost in racking, stacking, cooling and operating the processor was the same regardless of whether the CPU was 99% idle or 99% utilized. If the cost of a fully loaded server was 15,000 dollars, (server+racking+cabling+switches+routers), then customers were spending 14,850$ to get what should have cost only 150$ (1% of 15,000).

In effect we had broken the virtuous side effect of Moore's law: that increasing transistor count would result in more available compute cycles for applications at no incremental effort on the part of the software developer and as a result software would increase in value as developers added more features and functions with no increase in cost to the customer.

Worse, because of the economics of power and cooling we were staring at the uncomfortable realization that the cost of software (hardware+software) may be increasing rather than staying constant or falling.

Think about it, we had software that was not improving coupled with hardware costs that were going up, hardly a recipe for long term health.

Enter server virtualization which enabled the use of the cores without requiring MP safe code. And some folks got excited with server virtualization because they assumed that this was a sure fire way to save money. The theory was that we could now run all of those applications on those idle processors and that the total number of processors would decline permanently and net costs would go down. And the assumption was that this was a permanent cut in total computing cycles.

And the assumption underlying this was that we had dramatically over-provisioned our compute cycles, because these were truly idle cycles. In other words, even if the cycles were easy to use, we would not use them because they were unnecessary.

And I must respectfully disagree with that notion. If we really believed that the CPU cycles were really free then software innovation would have already stopped.

The real implication is that server virtualization made it possible to easily use those idle cycles. And once you made it easy, software developers and customers would start consuming them.

Because virtualization made it cheap again to get incremental usable CPU cycles, I believe the net effect is to increase the overall demand. This compression in servers is a temporary phenomenon. The demand for ever increasing compute cycles will continue.