Extensible NetApp: Thinking outside of the box

FAS Deduplication is here for the long haul

The current tussle between NetApp and EMC has, predictably, created a dust-up amongst the corporate blog-o-sphere. After all the storage industry believes in death-match-competitive-blogging, none of this polite, civilized discourse, here we reach for the knives and hack!

And it would be tempting to point out that Chuck and Storagezilla are like Aesop’s fox, but that would be too easy.

However, there is a point of confusion in the storage and analyst community about the relationship between FAS dedup and inline dedup technologies like the one provided by Data Domain. There is some opinions that say that this proposed acquisition is driven by flaws in our technology. There are some customers who may be wondering whether this reflects some flaws in our technology.

So let me state again, what I said earlier:

The rest of the storage industry has adopted inline because they lack NetApp's storage virtualization layer that enables primary deduplication.

But because that confusion is real, I wanted to remind folks about why FAS dedup is unique. And more importantly, why NetApp FAS dedup is different and must be different to meet the market requirements our storage plays in than other inline dedup solutions.

I went over this in a post several months ago in a post titled A little digression on Inline Dedup.

The basic points I made are the following:

1. Deduplication is a compute intensive operation
2. Single core performance is stagnating
3. Latency isn’t something it’s the only thing
4. Backup workloads are latency insensitive but throughput sensitive
5. Primary storage workloads are latency sensitive

And I observed that because of 1, 2 and 5 any inline process is problematic if not deeply flawed.

And then in another post I observed that the important question for dedup isn’t how efficient the dedup process is, but how well do you perform after the dedup process is complete.

Our deduplication process performs well because of it is integrated into our core WAFL data structures and because, unlike everyone else in the storage industry, we’ve solved the random IO performance problem introduced by on-disk layout fragmentation.

FAS dedup is the most widely deployed dedup solution, the only workable primary storage dedup solution, realizes tremendous cost savings, and continues to provide compelling value to our customers.

Additional Information and Where to Find It

As mentioned above, in connection with the proposed acquisition of Data Domain, on June 4, 2009, NetApp filed with the SEC a Registration Statement on Form S-4 (Commission File Number 333-159722) containing a Proxy Statement/Prospectus for Data Domain’s stockholders, and NetApp and Data Domain each plan to file with the SEC other documents regarding the proposed transaction. The definitive Proxy Statement/Prospectus will be mailed to the stockholders of Data Domain. BEFORE MAKING ANY INVESTMENT OR VOTING DECISION, INVESTORS AND SECURITY HOLDERS OF DATA DOMAIN ARE URGED TO READ THE PROXY STATEMENT/PROSPECTUS AND OTHER DOCUMENTS FILED WITH THE SEC CAREFULLY IN THEIR ENTIRETY BECAUSE SUCH DOCUMENTS CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION.

Investors and security holders will be able to obtain free copies of the Registration Statement, Proxy Statement/Prospectus and other documents filed with the SEC by NetApp and Data Domain through the web site maintained by the SEC at www.sec.gov and by contacting NetApp Investor Relations at (408) 822-7098 or Data Domain Investor Relations at (408) 980-4909. In addition, investors and security holders will be able to obtain free copies of the documents filed with the SEC on NetApp’s website at www.netapp.com and on Data Domain’s website at www.datadomain.com.

Participants in the Acquisition of Data Domain

NetApp, Data Domain and their respective directors, executive officers and certain other members of management and employees may be deemed to be participants in the solicitation of proxies in respect of the proposed transaction. Information regarding these persons is set forth in the Proxy Statement/Prospectus described above. Additional information regarding NetApp’s executive officers and directors is included in NetApp’s definitive proxy statement on Schedule 14A, as filed with the SEC on July 14, 2008, and additional information regarding Data Domain’s executive officers and directors is included in Data Domain’s Annual Report on Form 10-K and Form 10-K/A, as filed with the SEC on March 13, 2009 and April 30, 2009, respectively. You can obtain free copies of these documents from NetApp or Data Domain using the contact information above.

The increasingly meaningless question of storage utilization

I've been recently engaged in a series of internal discussions about the question of what is storage efficiency.

As a company, NetApp has continued to deliver amazing amount of value by allowing our customers to store huge amounts of data on a relatively small storage foot print and yet gets routinely criticized for our inefficiency.

In point of fact, we create a marketing program to put our money where our mouth is, hell we even run a local marketing campaign, and still we get asked: well how efficient are you really?

Which makes me wonder if we're just barking up the wrong tree.

Storage Efficiency is so last century

SAN storage architectures built around Real Fibre Channel principles are what created the current insanity that frames the storage efficiency discussion.

They declared that the storage problem could be best be solved by creating a storage architecture that first required you to combine lots of storage into a single disk pool, and then statically parceled out the resources into smaller chunks that were handed out to servers. Storage utilization was a measure of how effective you were in using your storage pool. And then the question was of the storage you had allocated how efficient you were in using that.

Great.

So the entire storage industry is married to a storage management model that says that storage efficiency is a measure of how efficiently you used antediluvian architectures based around static partitioning.

Wonderful.

But the problem, and here's the rub Better Than Real Fibre Channel can do things that Real Fibre Channel can not. Because Better Than Real Fibre Channel can use block level reference counts, Better Than Real Fiber Channel can store 255 full copies of data at the cost of that actual unique data within those 255 copies.

So if I have a snapshot, a single NetApp snapshot, I have a full copy of the file system. But instead of being applauded for storing a huge amount of data on a small amount of storage, I have to explain the WAFL overhead.

I'll be blunt, the WAFL overhead allows you to store what to a client appears to be a full copy of the file system on the minimal amount of disk blocks necessary. You wanna do that using Real Fibre Channel, I really hope you like buying disk drives.

And if it was a couple of EMC bloggers I wouldn't mind. But this discussion has permeated the entire computer industry.

And I wouldn't care too much except the question of how we should measure storage system efficiency is not even being discussed because of the fusillade of FUD. And because at the end of the day no one really wants to talk about how inefficient their architectures are.

If they did, storage customers would learn exactly why it's better to use RAID10 to protect your storage instead of replicating your data at the application level.

It's almost 2010 ...

NetApp and Data Domain, and soon others, have demonstrated that a storage system that uses indirection between the disk block and the logical disk block can store more data on less storage than we ever imagined.

And instead of talking about how efficiently we do that, in terms of CPU/Memory, or effectively in terms of the impact to applications we continue to discuss what is the most pointless of conversations:

If I treat my very sophisticated storage array like a set of big dumb disk drives I bought at the local electronics discount store, what's the usable capacity?

Blech.

High Noon in Australia

NetApp’s Australia team has offered to give an EMC customer up to 50 TB of raw storage if they’re willing to migrate their existing vmware environment to NetApp storage. The customer has to go public with the results and regardless of the results gets to keep the storage.

This feels a little bit like a scene in a bad cowboy movie, where one of the gun-slingers frustrated with the trash-talking of some other gunslinger in the saloon says: Let’s settle this outside at high-noon.

What I found interesting about this marketing activity is that it goes to the heart of what I love about NetApp’s marketing. We’re not about spreading FUD and lies or name calling. We’re about building our success on top of our customer’s success. Which may be why we choose to participate in standard benchmarks rather than proclaim how useless they are.

At the end of the day the activity will prove that we were not stacking the deck unfairly when we compared RAID-DP to RAID-10, that the performance is good enough, that it’s not too hard to make work, and guess what you do save storage.

Understanding WAFL Performance: The F Word

An astute reader of my series would have observed that WAFL is able to achieve its surreal write performance because WAFL does not guarantee that two disk blocks are sequential even if the application writer assumed they were sequential. So for example, if the application thinks that it’s writing two disk blocks in sequential order, they may not appear sequentially on disk.

In many quarters this is perceived as violating an implicit contract between the storage array and the client of the storage array. And in many ways, this is at the heart of the Real vs Better Than Real FiberChannel debate.

The core question is: does this matter?

So let me, right off the bat, say: of course it does. If an application goes to the trouble to assume that two disk blocks are sequentially located on disk, then it’s assuming a certain kind of performance when it requests the data. If the storage array turns that sequential operation into several random operations, then the application performance will degrade. This is just a fact of life.

Case closed, then. WAFL sucks.

Not so fast there storage architect.

Let me first observe, that the traditional storage arrays are able to transform random write operations into sequential operations through the use of clever caching and architectures that rely on global shared memory.

In a similar vein, it is possible to do the same thing for read operations, you can apply clever algorithms to make what is a fragmented on disk layout, appear to be sequential by being clever.

Making this concrete

Suppose we have an application that writes out a file in the following order:

Where the data A was at offset 0, and B was sequentially located after A in terms of offset.

The application then has the expectation that if it operates on the file by sequentially accessing A, B and C, that it will get sequential performance.

On the other hand the application has the expectation that if it operations on the file by accessing A, D, F, that the performance will be that of random disk IO.

Last week I showed this picture that showed how WAFL did write operations:

Now let me show a different picture that shows how disk blocks could get laid out by WAFL.

The light green blocks represent free blocks. The blocks marked P are parity blocks. The blocks that are not green and have the letters A, B, C, D, E, F, G etc represent allocated blocks. A single allocated file on this set of disks is represented by a sequence of blocks share the same color. In this example we have four files.

In this example, a sequential read operation would start at a specific row, and then read the blocks in a column starting from the row. A sequential read for example would be K1, L1 and M1.

So now if we consider an application that wants to read sequentially, you can imagine how this turns into a sequence of random read operations.

For example suppose the applications wants to read blocks A, B, C and D of the blue file and then wants to read E, F and G. A simplistic implementation would do one random read to read blocks A and B, another random read for block C, and a third random read for D. A clever implementation however, would recognize that A, B, C and D although not sequentially located on disk, can be read sequentially if you willing to skip over uninteresting disk blocks. A clever implementation would also perform read-ahead to read blocks E, F and G, such that the request for the next set of blocks came from memory rather than from disk, minimizing the total IO to disk.

To be fair, constructing such algorithms that work well in general is a hard problem. As I said earlier, the relatively simple problem of a single processor scheduler that has been studied for almost 40 years still produces new insights, this is a relatively new problem and we’re still learning.

But here’s where things work in our favor. All of these algorithms benefit from increasing compute power because you can spend more time thinking, and increasing memory so you look further into the future.  Both of those trends have worked in our favor over the last 15 years.

CPU performance has increased so much faster than disk drive performance, that we are able to perform very sophisticated computations before we do any disk operations. Memory capacity has increased dramatically as well allowing us to store more things in memory before we need to actually perform an operation.

But it’s more than just hardware. It’s also having a brilliant engineering team that has been focused on solving this problem for almost fifteen years. And no amount of hardware can replicate that experience.

But don’t trust me trust benchmarks that have validated this approach. Alex McDonald in his blog shows how we were able to sustain our performance even if the face of daunting random IO.

But why does this matter, in general?

Fragmentation of the on-disk layout is not just a property of WAFL, it also is a property of things like deduplication, thin provisioning and Real Snapshots. And the reality is making those things perform well requires the same kind of algorithms and sophistication that you need to make Better Than Real FiberChannel work.

The good news is that we’ve been working on those problems for a long time, which is why our benchmarks always have used snapshots and enabled thin provisioning.

Oh that might be the reason our competitors don’t have those things enabled and recommended turning them off when you need good performance...

The one good (storage virtualization) and the one bad (fine grained HSM) idea of every recession

In a recession, the storage group like every corporate function is asked to do more with less. One of the particular challenges with storage is that once a byte is consumed, it's hard to unconsume it. You can't just delete disk blocks willy nilly.

So you take a deep breath

One approach is to look at the stuff you already have and see if you can make better use of it. For example, when times are good we don't ask our users to delete unnecessary data, when times are bad we send out "spring cleaning" emails, and crunch down quotas. And all of that works but to a point. This is the moral equivalent of holding your breath. It helps, but it is unsustainable.

And then you try to optimize

Once it's clear that you can no longer hold your breath you need to start looking at more system approaches to doing more with less. And that's when things like storage tiering get revisited. When you were flush, everybody got better storage than they needed. When you're short on cash, and you still need to grow your storage capacity it makes a lot of sense to re-examine every application and move things around to optimize for the right cost/performance/availability tier of each application.

And in that maelstorm of applications being moved around, the ugly step-child of storage, fine-grained hierarchical storage management (HSM), makes it's appearance. Fine-grained HSM is storage tiering taken to it's absurd conclusion. If moving large blobs of data to the right tier is good, then moving small files around must be better! And if moving large blobs of data transparently is a good idea, then moving small files transparently must be all that better!

And because re-configuration of your storage is hard, and because HSM is impossible, storage and NAS virtualization appliances make their appearance. The basic idea is that by inserting some kind of layer of indirection between the host and the storage, a virtualization device allows transparent migration of data between the storage tiers. And if you do enough migration and move things around just so the net effect will be an optimized storage infrastructure!

The other benefit the storage virtualization gang tout is that by using this new layer you are able to unify disparate storage islands in your infrastructure into bigger storage pools. Bigger storage pools should create better efficiences. Right?

Except that the market for storage virtualization has failed. The market failed because if you're inserting a bump-in-the-wire you better be adding enough value to over-come the market's aversion to bumps-in-the-wire (and if you want to understand what I said you might find this one-two-three-four part series interesting ). On top of which, the bump's in the wire were being inserted by companies that were unable to provide the right level of support to make their new infrastructure work.

And so along with the storage virtualization appliances, the term went down in flames.

Which is unfortunate, because storage virtualization is the only systemic way to actually reduce your storage costs. Storage virtualization, a term hijacked by products whose only function in life appears to be to simplify migration, is a technique that allows you to reduce the amount of storage you consume by exploiting data redundancy through indirection. Storage virtualization, furthermore, allows you to separate the logical storage container from the physical infrastructure the storage container is actually stored on simplify data migration.

Enter Better Than Real FiberChannel

So for example, WAFL, NetApp's storage virtualization layer, allows us to create space-efficient clones, eliminate data redundancy through dedup and take snapshots that are performance neutral, space efficient and plentiful. Furthermore, WAFL allows us to replicate data between different physical infrastructures (SATA to FC, low-to-high and back again). In fact, the unique capabilities that WAFL has are what lead to our last promotion.

If you choose to use Better Than Real FiberChannel you can systemically move to a lower cost basis for your storage.

The problem for most Real FC vendors and ultimatley their customers, is that because Real FC vendors consider storage virtualization to be a dirty word (or at least a dirty unsavory and performance killing layer of indirection, benchmarks be damned) they can not exploit the information in the array the way WAFL does. The net effect is more storage being poorly utilized.

MAD blog: Chuck demands customers stop buying frankenstorage

I never thought I would see the day where Chuck Hollis would admit that Frankenstorage was a term (a term I believe coined by NetApp to describe the NX4 and EMC VTL).

I never though that I would see Chuck tell customers to not buy Frankenstorage. Good for you Chuck, I thought. Finally calling a shovel a shovel instead of a Lexus.

I thought can it be? Can EMC's VP of Marketing (or Spin?) finally be telling customers something truthful about unified storage? Like how using four different technologies for provisioning and data protection does not unified storage make?

Nope.

It's the damnable v-series. The damnable v-series. That's the piece of technology he hates. A piece of technology he went out of his way to distort facts about. And it's not because of our support of the TMS device, but because you stick a v-series in front of a Symm or a CLARiON and you suddenly need less disk. And that saves money. Not funny money but real dollars and anything he can say or due to blunt that point he will say and do. And if all you're using the Symm for is disk drives, you need less of their software. And that has got to hurt.

Of course the v-series and RAMSAN combo is not a unified storage device. But EMC has yet to ship any device that meets a reasonable definition of such a device. And for Chuck to lecture about any term, after all he coined Real FiberChannel(tm), is darn funny. So unlike the the original Frankenstorage, other than hardware management, all of your day-to-day operations are done consistently in a uniform way. You know using the ONTAP tools. But you've read that rant of mine before.

So I suddenly realized what the issue was: Professional Jealousy. Dr. Frankenstein is pissed off that someone else knows how to put together a device made up of multiple hardware devices and add value to customers.

And not only can we do that, but we can make our systems simpler to manage and adminsister than what the good doctor produced.

Hmm...

Backup, the new storage tiers and real snapshots

When I first described my new terminology for storage tiers, Tim Burlowski, asked how backup and recovery fit into that model.

It turns out that that is an interesting question.

Backup, the only working ILM/HSM solution

Step 10 feet away, and you realize that backup is HSM.

The backup process creates cold data, the backup image. The backup software them moves the backup copy to cheaper tier of storage. When a restore is performed, the backup software moves the backup copy from the cheaper tier to a more expensive tier.

The question on the table, then, is where does this form of HSM still make sense?

Captive IOPS

One obvious place where HSM as part of backup process is for the backup of data on a Captive IOPS tier. The tier is built using very expensive storage. Storing cold data that will probably never get accessed seems odd and a waste of money.

But that's not necessarily true.

If the Captive tier is built using disk drives, and the application  has a high IOPS data density, a lot of the capacity of the disks is  unused. The cheapest place to store backup copies, in that case, would be on the same disks you're using for primary data.

If the Captive tier is built using SSD's like EMC storage, then it makes a lot of sense to use a backup solution to move backup images off of the flash onto cheaper storage. If the application has a low IOPS data density, then a better approach is to use flash cache in front of disk.

If the Captive tier is built using a flash cache in front of disk, the cheapest place to store the backup copies may be on the disk drives.

More generally, the backup methodology of a captive tier made up of flash cache and disk, will be the same as the backup methodology of the capacity tier.

So what about the capacity tier and backup?

In the general case, the capacity tier will be backed up much in the same way that it always has been.

A capacity tier that has real snapshots can eliminate the HSM part of the backup process.

In fact, in an earlier series on backup I proved that for a VMware infrastructure, the cheapest place to store backup copies was on the same disks that stored the active data.

Now I want to generalize that comment for all storage..

If you consider IOPS density, disk drives must contain large amounts of cold data. A subset of the cold data in a data center is the backup data. That data can either be stored on a separate set of disks or on the same disks that are being used to serve IOPS.

To store the data on the same disks the following things need to be true

1. Creating the backup image must consume minimal IOPS
2. The existence of the backup image must impose no performance penalty
3. Be able to store a large number of images.
4. The storage must be configured using some form of RAID-6 or RAID 1-0
5. The cost of the backup images must be cost effective

It turns out that for NetApp systems all five are true.

So as disk capacities expand, storage architects who have the option to use real snapshots will stop moving data off of the capacity tier to some cheaper tier.

Said in a slightly more poetic way:

The final resting place for data on disk will be the disk where the data first got created.

The net effect will be a much simpler and more cost effective storage infrastructure to support backup.

One minor addendunm added after I wrote this post.

As Martin G says in the comments below, having the backup copies on the local disks doesn't protect you from site failures. So, of course, some form of storage replication is required. Thankfully NetApp has a cool technology called SnapMirror that replicates all of the snapshots to a remote destination in a space efficient way.

In an earlier post on VMware and backup I show how all of this fits together.

I should have included this need to replicate the data in my original blog post. So I am rectifying that error now.

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...

Tape, Roman Chariots and Data Management

There is this great story that describes how the railroad gauge of 4 feet and 8.5 inches derives from the Imperial Roman War Chariot. The point behind that story is that we continue to do things long after the reason  for doing them has evaporated.

It turns out that tape is playing the same role in the data center.

So it turns out that backup is the only ubiquitous data protection policy.  Regardless of how small you are, protecting data from operational errors (human frailty and software frailty) is something you do. The first time you accidentally delete an important file and  the first time an upgrade fails is the moment a backup procedure gets created. And that tends to happen within a few months of acquiring a computer system.

Because backup is so ubiquitous it's very highly automated. There is an awe inspiring amount of software developed and sold for backup.

And once you have a backup policy, the next thing you start to worry about is disaster recovery. Namely, how do you recover all of the data after a system or site goes belly up. And the obvious thing to do is leverage the existing backup infrastructure to get the data back after a disaster. And then once you've got disaster recovery covered, the next thing to worry about is long term archival. And it's logical to just leverage the same backup infrastructure, just hold onto the backups for longer...

So far so good.

And back in the Ur-days when tape was a lot cheaper than disk, the only way to economically do backup was to use tape.

Okay so back when you just did backup and restore, you probably chose to do a single full backup followed by some large number of incremental backups. You probably held onto the tapes for a modest amount of time and you had a small number of tapes. Life was good.

But then you wanted to leverage Disaster Recovery so you had to start taking more full backups more frequently. Why? Because doing full recovery from a set of incremental backups is simply not practical. And all of a sudden you had more tapes, and things started to take more time, and you needed bigger pipes.

And then you wanted to do long term archival, and all of a sudden you had  even more tapes staying around for longer.And now you had a whole set of policies that you needed to put into place to recycle tapes.

So now what?

Well now we have a set of processes and procedures that exist because that's the way tape made things exist. We have servers that have enough IO bandwidth to be able to do full backups on a regular basis. We have network infrastructures that only exist to sustain the vast amount of data that needs to be moved for backup. We've created and sized an entire infrastructure predicated on moving all of the data very frequently because that's how tape made us do it...

Okay...

But here's where it gets insidious, we know look at the mess that tape has created, and instead of asking the question:

Is a data protection infrastructure predicated on creating whole copies on a regular basis flawed?

We ask the question:

How can I make creating and storing full copies more efficient?

Rather than use new technologies that make backup and restore efficient, disaster recovery efficient and archive efficient and effective, we continue to rely on creating vast amounts of redundant copies, keeping those copies around for a while and praying that deduplication will save our skins.

And when folks argue that deduplication really should be done on the primary, and that we should look at ensuring that as little data moves we act surprised. Data management is about managing thousands of whole copies, that's the way it's always been ...

At some point, 50 years from now people will ask why do we expend so much energy eliminating redundant data when we could have avoided creating the data in the first place. And someone will write an article about the role of tape and whole images and people will laugh...

A response to Chuck on Deduplication for EMC

Chuck's response to my post Introducing Deduplication for EMC was interesting. Val's response to Chuck's post was hilarious and informative.

And although I was sorely tempted to walk away from this one, I couldn't resist the temptation to respond to the first question.

Question #1 -- What's Involved In Installing One Of These?

Chuck's answer to this question was interesting, not for the answer, but the kind of answer.

Here's what he did:

Assert Something Outrageous: The only deployment model was rip-n-replace

Observe something reasonable based on outrageous assertion: Rip-n-replace is expensive

Conclude Outrageous Proposal is Outrage: The deployment of a V-Series is impractical.

Although a good rhetorical device, this line of reasoning is not a very useful way to have a thoughtful conversation.

Because I am technologist and not in product marketing, let me offer a different, and I hope more thoughtful, response.

Needing more storage

Any growing enterprise will have to store more data, and storing more data will require more storage. Now it turns out that in practice there are only two ways to get more storage:

1. Buy more
2. Use what you have more efficiently

The real story about V-Series and Deduplication is that it really addresses the second use case. Using our V-Series technology you can use the disk you've already got more efficiently.

Who needs storage

There are three basic consumers of storage

1. Existing applications whose needs expand and the storage is providing good enough service at the right cost
2. Existing applications that are going through a technology refresh
3. New applications

Existing applications that just need more storage, especially if the storage is good enough and at the right price, are rarely a good target for any new technology. It's always easier to just use more of the same.

Existing applications that are going through a technology refresh are a good target for introducing new technology. If you're going through a technology refresh, you are probably looking at the whole stack from application to OS to server to storage trying to figure out how to get more for less money. Depending on how thorough the refresh is, you've already budgeted some amount of disruption and you're willing to live with that disruption.

New applications are an obvious target for introducing new technologies in an infrastructure. Processes and procedures have not yet been deployed, so there is a lot of opportunity to look at the best and most cost-effective way to meet your business objectives.

So the answer to Chuck's question is?

If you are looking at increasing the utilization of your existing infrastructure as you deploy new applications or go through technology refreshes for existing applications, then the answer to what's involved in deploying one of these is not rip-n-replace but

The complexity of deployment is the complexity of setting up a V-Series to use an EMC device.

So that suggests that the ROI for the device is a lot lower than the catastrophic 20 hours of down-time per-server that Chuck suggested.