Extensible NetApp: Thinking outside of the box

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

The fixed point theory of data centers.

Every so often I leave my low-earth architectural orbit to go into geosynchronous orbit. I stop thinking about how things are, but start thinking about how things will become.

The purpose is not to define new technologies that will become useful in some nebulous future, but to understand the future of particular current technologies. The point of the exercise is to understand whether a technology is fundamental or whether it's exploiting a particular discontinuity in the infrastructure that will eventually get smoothed out.

Obviously with any long term predictions the possibility of error is very high, but it's an interesting intellectual exercise. And when you are responsible for thinking through 36-72 month technology plans a very important exercise.

One of the mental tools I use is the fixed point theory. Now this theory is borne out from the observation that in any reasonable data center there are only four necessary hardware components:

1. CPU and Memory to run the applications
2. Disk drives to store the data for online access
3. Tape for backup
4. Switches

The theory says the following: if your  solution does not completely satisfy all of the requirements for one of the fixed points, the long term prognosis of your technology is problematic.

The rationale is simple.

If your technology is compelling it introduces a new fixed point. The fixed point lives between two pre-existing fixed points. The technologies on either end of the fixed point will work hard to incorporate the value of the new fixed point. Eventually they become good enough (tm). And good enough (tm) happens when the value in the pre-existing fixed points plus the cost of adding new hardware exceeds the value of the new fixed point. That can take several years but it does inevitably happen.

So if you do become a new fixed point in the data center, then you have to grow to the point where you either are a new fixed point across all data centers eg:disk as a target for backup OR you acquire sufficient capabilities to become one of the existing fixed points eg: you start off as a NAS work group server and become an enterprise storage provider.

How to Operationalize FlexClone

Joe Bloggs, a data center architect, was recently assigned the task of figuring out how to use FlexClones.

The value of FlexClones around speed, simplicity and cost savings was so obvious that Joe was really motivated to make this work.

Joe looked at the problem and broke it down into four pieces:

1. Create a consistent image of the database on the production system
2. Copy the consistent image to test and development system
3. Enable DBA's to use clones to do their development.
4. Move final image back into production.

Now it turns out that Joe had tools and processes that he was comfortable with for solving steps 1, 2 and 4. After all he had been supporting database administrators (DBA's) for a while. So his plan was to continue to use those tools and processes. The challenge was step 3.

The naive and impractical solution would have been to give the DBA' s the root password on the FAS system and pray that they did not make a mess of things. Since Joe was not into faith based storage administration, he decided that he needed a more regimented process.

His requirements were:

1. Multiple distinct business groups must be able to use the same FAS system.
2. Each distinct business group must be prevented from impacting other business groups.
3. The DBA' s must able to perform the following operations without involving the storage group:
   1. Create a clone
   2. Destroy a clone
   3. Mount a clone

 

The first and second requirement were reasonably easy to satisfy. Joe's plan was to give each business group an storage pool (an aggregate)  within a FAS. The DBA's would share the controller, but would get their own private storage sandbox. The amount of capacity for each sandbox would depend on how much the line of business wanted to spend for their DBA's.

But the third requirement was problematic. The challenge was that the FAS system does not provide the fine level of role-based access control that he wanted. But even if it did, he was not sure that he wanted to expose DBA's to the full richness of the ONTAP command set.

What Joe wanted was a tool that the DBA's could use to manage clones, and that was integrated with some form of role based access control so that DBAs never accidentally managed someone else's clone.

Joe then asked NetApp for a solution to his problem. Ryan Cox, an engineer that is part of NetApp's Rapid Response Engineering (RRE) Team came up with a tool that leveraged the Manage ONTAP API's.  The tool used the FAS APIs for manipulating storage objects and the APIs that are part of the Operations Manager product to manage Role Base Access Control. In a follow on post I'll talk about this tool in more detail.