Extensible NetApp: Thinking outside of the box

So why is this, Data Motion, thing so cool?

chuck hollis in a tweet asked:

@kostadis_netapp how is this different/better than, say storage vMotion, or external file virtualization, or other alternatives? Not clear.

And that was a legitimate question, and I was going to reply in a tweet, but 140 characters is not a great way to explain anything.

In many ways, storage vmotion and file virtualization and NetApp Data Motion, are all trying to solve the same fundamental problem of separating physical storage from a logical storage container.

When evaluating technologies the question is: what are the tradeoffs?

Storage Vmotion

Storage Vmotion, vmware's product, is great if you're only using vmware and you want to move your data one virtual machine at a time. The limitations of storage vmotion are

1. It doesn't work for hyperv, and it doesn't work for physical environments.
2. It moves the data by schlepping the data through the physical server. NetApp Data Motion moves the storage without bringing the data back into the server
3. It can't move your snapshots. If you're a NetApp customer using NetApp snapshots, storage vmotion will not move the vmdk's locked in the snapshots. 
4. Losing your data protection relationships. If you move a vm from one data store to another data store, any storage based data protection relationships get lost.
5. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

File Virtualization aka Snake Oil

File Virtualization is, in my not so humble opinion, snake oil. The premise of file virtualization is that you can track millions of files in a meta-database outside of the file system and move individual files. But let's assume it can be made to work, a lofty and unproven claim thus far.

What are the limitations of this technology

1. Scale. Moving millions of files one at time and tracking the information.
2. It can't move files in snapshots
3. Data protection. If a file gets moved between different storage tiers, then the data protection relationship gets gummed up.
4. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

NetApp Data Motion

NetApp Data Motion is, of course, perfect in every way (not!). NetApp Data Motion is great if you want to move a whole bunch of data and all of the storage based snapshots and preserve any mirroring relationships. However, there are limitations

1. Requires either NetApp storage or a NetApp virtualization appliance
2. The granularity of movement is large (a volume). So if you want to move a single file or vm it's not quite the right technology
3. Dedup'ed data stays deduped. This means faster data transfer and no dedup after the fact.

So why was I so excited?

The reason I was so excited was because NetApp Data Motion is the culmination of a journey to build a management infrastructure that divorced logical from physical storage. Not because this is the only way to skin the storage data motion cat, even though I think the advantages of this approach are significant.

Late breaking updates...

jpwarren reminded me about dedup. Do'h!

Data Motion: the next stop in storage management

Data Motion is a particularly proud moment for me.

In 2005, a small number of folks, myself included, in a conference room in a hotel in Palo Alto were looking into how NetApp could transform storage management.

Up until that point, NetApp had a small set of interesting products that provided reasonable, if not great, monitoring and modest active management. For long time NetApp hands, the phrase DataFabric Manager might be familiar.

In that room we asked: would it be possible to do something radically different?

Being arrogant, and ignorant, we thought: couldn't we dramatically simplify storage management through policy based automation? The intent was to separate logic storage from the underlying physical infrastructure. To turn storage management from a hardware problem to a software problem.

The intuition was the following, NetApp storage virtualization layer allowed dynamic reconfiguration of storage. There was nothing that said software couldn't do that dynamic reconfiguration. And if software could do that, then we could make storage truly dynamic.

The first product we shipped that embodied that vision was Protection Manager. The vision itself was articulated in a paper we published in LISA 2007, and in an article in ACM Queue.

Today we announced Data Motion. A solution that is built on top of many of the foundational technologies we dreamed up four years ago. This solution demonstrates the power of a truly virtualized storage infrastructure that allows true dynamic reconfiguration. Is it complete? Of course not, but it's real.

The vision we had of being able to dynamically adjust your storage through software with no disruption using end-to-end tools is finally a reality.

I could not be more proud of this moment.

Introducing Deduplication for EMC

NetApp today proudly announces the introduction of the V-Series support of Deduplication for EMC storage. Finally NetApp allows EMC customers to have a single enterprise wide deduplication layer across all of their storage tiers. A  deduplication layer that is almost as easy to use as the one NetApp's 2500 customers have already been using and  deployed on over 10,000 systems.

To get the savings of deduplication  EMC customers will just have to insert a V-Series between their existing EMC storage and their servers.

For NetApp customers adding new hardware to get new features is weird. For EMC customers, who have to deal with 10 different EMC platforms, this should be par for the course.

Having taken advantage of the obvious benefit of deduplication, my hope is that customers will notice that they have now introduced a single unified storage virtualization layer into their infrastructure.

So their world that looked like this:

A world that had separate and distinct infrastructures

• that have to be managed, and organized and planned separately
• that tend to be over-provisioned and underutilized.

That bad world, will now look something like this:

With a single enterprise wide virtualization layer that has a unified approach to how data is protected and provisioned. A layer that can be accessed via all storage protocols and can support any application.

Using that virtualization layer EMC customers will be able to eliminate redundant data before it gets created with other technologies like SnapShot and FlexClone. These EMC customers will also be able to reap the benefits of using our policy automation layer to do simplified provisioning and data protection.

So, EMC customers, although the reason you will bring us in will be to save money, we hope you'll keep us because we dramatically reduce the operational overhead of your infrastructure.

It's not all about the hair.

At our end-2007-year-party the General Manager of my business unit, Louis Selincourt, and I got into a discussion over whether our business unit could achieve certain specific financial goals.

In reality what we were really arguing over was whether we could explain to enough customers the value of products like Protection Manager, and Provisioning Manager. Or said differently could we show to enough customers how they could transform their data center operations using our solutions?  And the goals we set were significant enough that meeting them would be a certain amount of proof of our ability to both show and deliver on the promise we made.

So I agreed to let him shave my head during our business unit all hands if we met those financial goals.

Having my head shaved was actually fun because I saw it as a celebration, in many ways, of a significant accomplishment.

But I also learned something about leadership and trust and keeping your word. For some folks the fact that I was willing to go through with the whole head shaving did more for my credibility than any previous action.

And I guess there's a moral here. When you make a public promise, and you keep a public promise, that helps people trust you when you make private promises.

But here are the pictures

Here is a picture of him preparing me for my shaving.

Here he is just getting started.

Half way to my new look

And a few minutes before he was done.

Introducing Provisioning Manager

One of the most exciting parts about being a Technical Director is that you get to see products and technologies that were gleams in your eye turn into actual things customers can buy.

Today, we announce the availability on the NOW site of Provisioning Manager as  part of the DFM 3.7 release.

In 2006 we introduced a new policy based management framework that is at the core of Protection Manager. Using Protection Manager, a storage architect can create protection polices and then associate those policies with provisioned primary storage. Protection Manager will use those policies to automatically configure the replication relationships, provision any secondary volumes and once everything is running ensure that nothing breaks.

Now the team has extended the policy based management framework to include provisioning for primary storage. A storage architect using the same user-interface and metaphors that were created for Protection Manager can define provisioning policies for his primary storage that act as templates for how storage should be provisioned.  The storage architect can combine the provisioning and protection policies into a single policy for provisioning storage.

Using this infrastructure we hope to have solved two two basic problems:

1. Guaranteeing the storage is always provisioned correctly, along with any related replication relationships
2. Managing thin provisioning

I am very excited to see this product ship.