Extensible NetApp: Thinking outside of the box

So why is this, Data Motion, thing so cool?

chuck hollis in a tweet asked:

@kostadis_netapp how is this different/better than, say storage vMotion, or external file virtualization, or other alternatives? Not clear.

And that was a legitimate question, and I was going to reply in a tweet, but 140 characters is not a great way to explain anything.

In many ways, storage vmotion and file virtualization and NetApp Data Motion, are all trying to solve the same fundamental problem of separating physical storage from a logical storage container.

When evaluating technologies the question is: what are the tradeoffs?

Storage Vmotion

Storage Vmotion, vmware's product, is great if you're only using vmware and you want to move your data one virtual machine at a time. The limitations of storage vmotion are

1. It doesn't work for hyperv, and it doesn't work for physical environments.
2. It moves the data by schlepping the data through the physical server. NetApp Data Motion moves the storage without bringing the data back into the server
3. It can't move your snapshots. If you're a NetApp customer using NetApp snapshots, storage vmotion will not move the vmdk's locked in the snapshots. 
4. Losing your data protection relationships. If you move a vm from one data store to another data store, any storage based data protection relationships get lost.
5. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

File Virtualization aka Snake Oil

File Virtualization is, in my not so humble opinion, snake oil. The premise of file virtualization is that you can track millions of files in a meta-database outside of the file system and move individual files. But let's assume it can be made to work, a lofty and unproven claim thus far.

What are the limitations of this technology

1. Scale. Moving millions of files one at time and tracking the information.
2. It can't move files in snapshots
3. Data protection. If a file gets moved between different storage tiers, then the data protection relationship gets gummed up.
4. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

NetApp Data Motion

NetApp Data Motion is, of course, perfect in every way (not!). NetApp Data Motion is great if you want to move a whole bunch of data and all of the storage based snapshots and preserve any mirroring relationships. However, there are limitations

1. Requires either NetApp storage or a NetApp virtualization appliance
2. The granularity of movement is large (a volume). So if you want to move a single file or vm it's not quite the right technology
3. Dedup'ed data stays deduped. This means faster data transfer and no dedup after the fact.

So why was I so excited?

The reason I was so excited was because NetApp Data Motion is the culmination of a journey to build a management infrastructure that divorced logical from physical storage. Not because this is the only way to skin the storage data motion cat, even though I think the advantages of this approach are significant.

Late breaking updates...

jpwarren reminded me about dedup. Do'h!

Data Motion: the next stop in storage management

Data Motion is a particularly proud moment for me.

In 2005, a small number of folks, myself included, in a conference room in a hotel in Palo Alto were looking into how NetApp could transform storage management.

Up until that point, NetApp had a small set of interesting products that provided reasonable, if not great, monitoring and modest active management. For long time NetApp hands, the phrase DataFabric Manager might be familiar.

In that room we asked: would it be possible to do something radically different?

Being arrogant, and ignorant, we thought: couldn't we dramatically simplify storage management through policy based automation? The intent was to separate logic storage from the underlying physical infrastructure. To turn storage management from a hardware problem to a software problem.

The intuition was the following, NetApp storage virtualization layer allowed dynamic reconfiguration of storage. There was nothing that said software couldn't do that dynamic reconfiguration. And if software could do that, then we could make storage truly dynamic.

The first product we shipped that embodied that vision was Protection Manager. The vision itself was articulated in a paper we published in LISA 2007, and in an article in ACM Queue.

Today we announced Data Motion. A solution that is built on top of many of the foundational technologies we dreamed up four years ago. This solution demonstrates the power of a truly virtualized storage infrastructure that allows true dynamic reconfiguration. Is it complete? Of course not, but it's real.

The vision we had of being able to dynamically adjust your storage through software with no disruption using end-to-end tools is finally a reality.

I could not be more proud of this moment.

Protection Manager pat on the back

Protection Manager was the first product to incorporate NetApp's policy based automation layer.

After having touted the value of the technology, it makes me very happy to read that Sprint Nextel was honored with ComputerWorld's "Best Practices in Infrastructure Management". What made this award special, for me, was that Protection Manager was specifically called out.

Here's the money quote:

With NetApp Protection Manager solutions, Sprint created a new infrastructure management layer to automate and speed the provisioning of resources from within the data center.

What makes this special is that it is validation of our technology by our customers and the world in general.

Introducing Deduplication for EMC

NetApp today proudly announces the introduction of the V-Series support of Deduplication for EMC storage. Finally NetApp allows EMC customers to have a single enterprise wide deduplication layer across all of their storage tiers. A  deduplication layer that is almost as easy to use as the one NetApp's 2500 customers have already been using and  deployed on over 10,000 systems.

To get the savings of deduplication  EMC customers will just have to insert a V-Series between their existing EMC storage and their servers.

For NetApp customers adding new hardware to get new features is weird. For EMC customers, who have to deal with 10 different EMC platforms, this should be par for the course.

Having taken advantage of the obvious benefit of deduplication, my hope is that customers will notice that they have now introduced a single unified storage virtualization layer into their infrastructure.

So their world that looked like this:

A world that had separate and distinct infrastructures

• that have to be managed, and organized and planned separately
• that tend to be over-provisioned and underutilized.

That bad world, will now look something like this:

With a single enterprise wide virtualization layer that has a unified approach to how data is protected and provisioned. A layer that can be accessed via all storage protocols and can support any application.

Using that virtualization layer EMC customers will be able to eliminate redundant data before it gets created with other technologies like SnapShot and FlexClone. These EMC customers will also be able to reap the benefits of using our policy automation layer to do simplified provisioning and data protection.

So, EMC customers, although the reason you will bring us in will be to save money, we hope you'll keep us because we dramatically reduce the operational overhead of your infrastructure.

It's not all about the hair.

At our end-2007-year-party the General Manager of my business unit, Louis Selincourt, and I got into a discussion over whether our business unit could achieve certain specific financial goals.

In reality what we were really arguing over was whether we could explain to enough customers the value of products like Protection Manager, and Provisioning Manager. Or said differently could we show to enough customers how they could transform their data center operations using our solutions?  And the goals we set were significant enough that meeting them would be a certain amount of proof of our ability to both show and deliver on the promise we made.

So I agreed to let him shave my head during our business unit all hands if we met those financial goals.

Having my head shaved was actually fun because I saw it as a celebration, in many ways, of a significant accomplishment.

But I also learned something about leadership and trust and keeping your word. For some folks the fact that I was willing to go through with the whole head shaving did more for my credibility than any previous action.

And I guess there's a moral here. When you make a public promise, and you keep a public promise, that helps people trust you when you make private promises.

But here are the pictures

Here is a picture of him preparing me for my shaving.

Here he is just getting started.

Half way to my new look

And a few minutes before he was done.

Using Protection Manager, SnapDrive for Windows and SnapManager for Exchange

In my previous post I described at a high level the power of the recently released SnapManager, SnapDrive and Protection Manager products.

To give you a flavor for the workflow of the tools let me walk you through the process of how they are used. *

First the storage architect or disk to disk backup architect defines a set of policies that can be used using Protection Manager.

To create a new policy the storage architect first selects one of the predefined policies

and then modifies that policy to meet the specific requirements of his organization.

Now the policy is ready to be used by an Exchange administrator.

Before you can use SnapManager for Exchange you need to configure SnapDrive for Windows to talk to a DFM Server that has Protection Manager license.

To use the policy the Exchange administrator brings up SnapManager for Exchange

and where before he could only configure local Snapshot and SnapMirror schedules, now the administrator has additional data protection policies that include disk-to-disk backup.

The Exchange administrator  then simply assigns policies to Exchange storage groups. As each policy gets assigned to a storage group, SnapManager for Exchange through SnapDrive for Windows creates a new dataset  in Protection Manager that contains the LUNs that contain the Exchange storage group.

Each dataset that is created does not, however, have any secondary storage resources assigned to it. That has to be done by the storage administrator.

To assign the physical resources, the storage architect would use Protection Manager to find the new datasets. This is made easy by Protection Manager, because the datasets would appear as a dataset that was out of conformance with respect to the specified policy.

The storage architect then assigns the right resource pool to the storage group. Once that is done, Protection Manager will  then automatically create the backup relationships, run the schedules, and configure any additional secondary storage objects.

Once that is done the Exchange storage group is protected.

* For more info on datasets, policies and resource pools check this paper we published in LISA 2007.

* For more info on how SnapManager and SnapDrive work check this article from Tech ONTAP.

Why are we integrating SnapManager for Exchange, SnapDrive for Windows, and Protection Manager?

SnapDrive for Windows 6.0 and Protection Manager, part of the DFM 3.7 release, and SnapManager for Exchange 5.0  are the first tangible artifacts that demonstrate the power of NetApp's Integrated Data Management Vision.

Three years ago in a meeting at NetApp, a bunch of us had the radical idea that if you could enable server and application administrators to manage their own data you could unlock tremendous amounts of productivity by eliminating processes and procedures in the data center that existed to overcome limitations in storage management systems. 

From that observation came two significant technological breakthroughs. The first was the introduction of our policy based management infrastructure as part of the Protection Manager product and the second is the integration of our very successful SnapDrive for Windows and SnapManager for Exchange products with that policy based infrastructure.

Superficially Protection Manager is a GUI for managing NetApp replication. But that, as they say, completely misses the point. If you think of Protection Manager as just a GUI, then you'll probably wonder why we ever bothered with the integration at all. And you would be right in wondering.

Protection Manager, in reality, is a policy automation layer coupled with a very capable delegation model. These two technologies allow storage architects to define policies for how storage should be used and in parallel allow other people to create storage based on those very same policies. In effect, the storage architect does the hard work of figuring out how he wants people to use his storage once, and leaves the relatively simple task of provisioning storage to other people secure in the knowledge that they can not make a mistake.

Introducing Provisioning Manager

One of the most exciting parts about being a Technical Director is that you get to see products and technologies that were gleams in your eye turn into actual things customers can buy.

Today, we announce the availability on the NOW site of Provisioning Manager as  part of the DFM 3.7 release.

In 2006 we introduced a new policy based management framework that is at the core of Protection Manager. Using Protection Manager, a storage architect can create protection polices and then associate those policies with provisioned primary storage. Protection Manager will use those policies to automatically configure the replication relationships, provision any secondary volumes and once everything is running ensure that nothing breaks.

Now the team has extended the policy based management framework to include provisioning for primary storage. A storage architect using the same user-interface and metaphors that were created for Protection Manager can define provisioning policies for his primary storage that act as templates for how storage should be provisioned.  The storage architect can combine the provisioning and protection policies into a single policy for provisioning storage.

Using this infrastructure we hope to have solved two two basic problems:

1. Guaranteeing the storage is always provisioned correctly, along with any related replication relationships
2. Managing thin provisioning

I am very excited to see this product ship.

Manage ONTAP

One of least well known aspects of NetApp  is our SDK program called Manage ONTAP.

And it's so poorly known, that I'll be devoting some of my time over the next couple of months to talking about them. Mostly because I believe using these APIs are the easiest way for administrators to operationalize the capabilities found within Data ONTAP. And since making their lives easier is part of my mission at NetApp, this seemed like a good fit.

So what is Manage ONTAP?

Manage ONTAP is an SDK that NetApp makes available to partners and to our customers. The SDK provides documentation, sample code, and a library that makes using APIs that are available on our storage system, easy.

The APIs themselves are a flavor XML/HTTP RPC's that are similar to but not exactly like SOAP.

NetApp products like Protection Manager, SnapDrive for Windows, SnapManager for Exchange were all built using the APIs made available through the SDK.

Over the next set of posts I intend to describe the core construct of the APIs, what you can do with them, and some of the ways we've used them inside our own products.

If there are specific topics you'd like to learn about, just add your comments to this blog entry, and I'll try and cover them as time and energy permit.

The poorly named replication

Over the past several months, I've been trying to understand the difference between the replication market and the backup market.

For example, NetApp sells replication software. But our customers think of our solutions as backup products. Heck, if you look at Protection Manager, you'll discover that we call a Snapshot a backup.

Why?

So it turns out that, the IDC, the keeper of numbers, classifies storage system copying as replication, and classifies backup software as, well, backup. Even if the backup software is, well, making copies from one server to another.

Which got me thinking about the storage industry, in general.

No one actually wants replication. Replication is a technology. It's a means to an end.

However, the storage industry is so enraptured with the technologies that we produce, that instead of convincing the IDC that we actually solved a problem like Business Continuance, we managed to convince the IDC that we sell, well, copy. Because somehow replication is, way, way cooler. Except of course for our customers who have a backup problem.

Which brings me to why I love Protection Manager. The product is focused on the top-level goals of the customer: backup and recovery, not on the technology that implements backup and recovery. As a result Protection Manager has no references to terms such as  Snapshot, SnapMirror or SnapVault, because you, the customer, don't care about how we backup your data, just that we do it.