NetApp - Ask Dr Dedupe

Questions and Answers about Deduplication

I recently sat in on a webcast Q&A panel with a few other vendors that provide deduplication technology.  Representatives from EMC, Data Domain, NEC, and Exar were on the panel, along with me.  The discussion was lively and the questions came in from the listeners rapid-fire.  Based on the large number of attendees and questions, its clear that people still have a lot of concerns about deduplication.  The webcast was sponsored by SNIA, who continues to do good work in educating people about new technologies like dedupe.  Click here for a replay of this panel discussion.

DrDedupe

4 Things Frank Slootman Probably Wishes He Didn't Say

Dave Raffo interviewed Frank Slootman this week at SNW.  For those that don't know him, Frank is the former CEO at Data Domain and is now President of EMC's Data Backup Division.  During the interview Frank said things that were, well, un-presidential.  Unfortunately for Frank, we live in a world that records comments like his and preserves them longer than those unidentifiable objects tucked away in the back of your refrigerator.

Anyway, Frank made four comments during this interview that stood out as “huh? did he really say that?” types of comments.  I only wish I were in the room so I could have asked some follow up questions.  If I were invited into the room (like that would ever happen) here is the way the interview might have gone:

Raffo: But if Disk Library customers want dedupe, they have to buy a Data Domain box?

Slootman: Yes. If you have a car and now you want an airplane, are you going to put wings on your car? You have to buy an airplane.

DrDedupe:  Frank – I am not sure that is the correct analogy.  A car is designed to travel on the ground, and an airplane is designed to fly through the air.  I think what you meant to say is that if you have a car now and you want a more efficient car, EMC says you should just buy another car, right?

Raffo: EMC has sold Quantum's data deduplication software with Disk Library. Will you sell Data Domain software with Disk Library instead?

Slootman: No. Disk Library is a straight VTL [virtual tape library] , like it always should have been. It's a brute-force system, no finesse. That's the way it was when it first came out, then they tried to turn it into something it is not by adding deduplication and replication. They bastardized the product, so much so people don't even know what a VTL is anymore. People think VTL is a generic term for backup to disk. People think Data Domain is a VTL, but 90% of the systems we sell are IP-connected, not with a Fibre Channel protocol.

DrDedupe:  Thanks for the clarification Frank - EMC was selling a bastardized product and confusing the market.  Good luck in the next all-hands meeting at that division you run now.

Raffo: What about the Quantum software EMC has sold with Disk Library?

Slootman: We're swapping a lot of those boxes out at zero revenue. We've taken out about a dozen and we'll continue to take out a similar number this quarter. Customers don't want it.

DrDedupe:  Wow Frank this is a very interesting disclosure.  Would you care to point out any other products that EMC has sold that your customers don't want?

Raffo: Will you continue to work closely with Symantec Corp.'s OpenStorage (OST) API now that you're EMC?

Slootman: Yes. I'm not throwing my partners under the bus. We'll compete, but we're all competitors and partners these days. We won't screw them. We'll screw other companies, like CommVault. We {Data Domain] treated them as a good partner and they came after us.

DrDedupe:  This is another interesting disclosure Frank.  I am sure our viewers will want to know who the other companies are that you intend to screw.

So thats the way the interview might have gone with DrDedupe in the room.  I'm pretty sure that Frank will have some 'splaining to do next time he travels to Hopkinton.  But then again, isn't he starting to sound more like other EMC execs?

DrDedupe

You Lie! Who Coined the Term "Data Deduplication"

Today I came across this quote from Ed Walsh, former CEO of Avamar Technologies:

“Data deduplication was not a term until Avamar used it,” Walsh said. “Data Domain called it capacity optimized storage. At Avamar, we had to teach the market that data deduplication was something that you wanted. Now the market is rife. The technology has proven itself.”

As your purveyor of truth in dedupe, I feel it is my responsibility to get to the bottom of this issue.  I've always given Data Domain credit for coining the term that eventually became my alter-ego. 

So, no I am not going to try and build a case for NetApp.  As stated  out in our award-winning video - "we didn't invent, it we perfected it." - But who really did invent it?  Lets find out together.

First stop, the U.S. Patent Office.  Avamar is listed as the owner of 6 patents.  The oldest, from February 2001, is entitled "Hash File System and Method For Use In Commonality Factoring System."  Whew.  Glad that name didn't last, Dr Commonality Factoring doesn't have the same ring to it.  Well ,anyway, this patent mentions duplicate data but makes no mention of data deduplication.  In fact, after searching all 6 Avamar patents, we find that the word deduplication is never used.

OK, now lets examine Data Domain.  They are listed as the holder of 11 patents.  Their oldest patent is from January 2002 and is entitled "Archival Data Storage System and Method."    Does it mention data deduplication?  Alas, no - it describes an archival storage system with disk drives that spin up and down (an idea, by the way, that seems to have spun down on its own.)

But wait!  A Data Domain patent from July 2006 is entitled "Locality-based Stream Segmentation for Data Deduplication."  The first use of the term belongs to Data Domain!  Do we have a winner?  No, lets just say the score stands at Data Domain 1, Avamar 0, at halftime.  BTW here's some nice halftime entertainment courtesy of the Dr.

OK back to our task. Next - lets try some google searches to see what comes up.  Well, well. Here's an Earnings Release from Avamar dated October 2006, four months after Data Domain's deduplication patent - and an excerpt from that press release: 

"Avamar Technologies, Inc. is a leading provider of enterprise data protection software. Avamar's patented data reduction, single instance store and point-and-click restore technologies have revolutionized backup, recovery and DR strategies for global corporations."

Nary a mention of data deduplication anywhere in this press release.  Now, Ed, if Avamar was using the term data deduplication before Data Domain, wouldn't it have been used in something important like this press release?  Sorry, but I have to declare a shutout here and name Data Domain the winner over Avamar , 2-0.  Ed I believe you've been caught in a little white lie.  Remember, the Dr is watching...

The Dedupe 2.0 Pundits Are Still Swimming in Lake 1.0

I had to chuckle when I saw the concurrent announcements from Permabit and Storage Switzerland that dedupe 2.0 had arrived.  Apparently the acquisition of Data Domain by EMC had signaled the demise of dedupe 1.0, even though I failed to see any technology transformation as a result of that merger.  Data Domain is still doing the same thing they always did and EMC is still doing what they do.  There was no transformative event here guys.

Anyway, the people at Permabit seem to believe that hashing and deduplicating large archival stores is ushering a new era of deduplication and leaving point products behind.  But, er, isn't Permabit another point product?  Last time I looked in a data center there was more than archival data behind the blinking lights on those storage arrays.  In my estimation, deep archival consumes somewhere around 1% of the world's data storage.

Then there's Storage Switzerland.  I guess the Switzerland part is to have one believe that they are a neutral party amidst this vendor-against-vendor world.  But does anyone out there really think they are neutral?  I'll let my readers form their own conclusions.  Anyway, SS says that the core of dedupe 2.0 will be a foundational repository where all previously optimized data will come to rest from primary and secondary storage.  Hmmm, sounds strangely like an archival system to me.  Wait, doesn't Permabit sell archival systems, and they are also talking about dedupe 2.0?  Could there be a connection here?  Say it ain't so Storage Switzerland!

OK lets move back towards reality.  You know when dedupe 2.0 will really arrive?  When people stop talking about dedupe altogether.  Like many technologies that came before it, such as RAID, SCSI, Snapshots, and dozens of others, dedupe will become ubiquitous and acknowledged as a must-have feature from every serious storage vendor.  Inline or Post-process, source or target, local or global - it doesn't matter as long as it dedupes.  Dedupe will run silently in the background, eradicating those pesky duplicate data objects.  NetApp has proven that dedupe can run quite well on a unified platorm that services all tiers of storage (including archival), uses all standard protocols, and serves a wide breadth of applications.  Once a concept is proven, the next steps are constant refinement and optimization, and thats exactly what we are doing.  I can't tell you exactly the day dedupe 2.0 will arrive, but I am pretty sure that NetApp will be among the frontrunners.

DrDedupe

Who Is Your Favorite Storage Blogger (ahem) ?

One of my favorite blog sites (Storage Monkeys) is running a poll for people to select their top 5 storage vendor bloggers.  Each of you can vote once (I already tried stuffing the ballot box.)  Here's my promise, keep me in the top ten and I'll keep blogging (maybe even more often) to enlighen you with a regular dose of dedupe, as only the Dr can.

Cast your vote now!

EMC's Celerra dedupe calculator doesn't add up

I ran across this EMC dedupe calculator today.  Interested in the space savings EMC was advertising for primary storage with dedupe, thought I'd take a closer look.  I was surprised by the results.  The calculator asks you to input the "Size of Current Data" for seven different application categories.  So I ran the calculator 7 times, each time entering 10TB into the 7 categories, one at a time.  Here are the results I got:

EMC Celerra Data Duplication Calculator

(Data Duplication? I sure hope that’s a typo)

 

Potential Savings:

Active virtual machine files - 0TB (0%) saved

Active databases and messaging files - 0TB (0%) saved

Compressed files - .2TB (2%) saved

Media files - .4TB (4%) saved

Binaries – 1.7TB (17%) saved

Office documents - 3.3TB (33%) saved

Text files - 5.3TB (53%) saved

 

Wait a minute, this can't be right.  4 of the 7 categories offer between 0% and 4% savings.  Doesn't seem like its worth the effort.  Well how about Binaries at 17% savings?  But EMC only dedupes "inactive" files - which of my binaries are inactive?  Not sure, so I think I'll skip that one.  Office documents - bingo 33% saved - but again this only applies to inactive docs, so I guess my savings will be a little less.  Finally, there are text files with a whopping 53% savings - but I probably don't have 10TB in inactive .txt files though.  At this point, I am a little disappointed in EMC.

 

The way I see it, using Celerra dedupe in a real world scenario would produce maybe 5% overall savings on a good day.  The problem you see is that the majority of user data is tied up in those darned applications that Celerra dedupe can't touch.  The puzzling part is why EMC would include applications with zero percent savings in their calculator.  Trying to deceive users?  Nah there must be some other explanation.  Guess I'll leave that one up to them to answer.

 

Your voice in the search for dedupe honesty-

 

DrDedupe

 

My View On Data Domain

In a comment to my last blog “NetApp Dedupe Revisited” - Glenn asked “does NetApp really need Data Domain more than EMC for purely technical reasons and to fix the shortcomings of A-SiS?”

This is a question I’ve heard a few times since NetApp announcedits intention to acquire Data Domain a few weeks ago.  It’s an important question Glenn, one that warrants a blog of its own.  Before I answer your question though, I want to stress that the transaction between DDUP and NTAP is still in process and subject to scrutiny by regulators and shareholders.

Now, back to the question – is Data Domain’s technology superior to NetApp’s?  No, it’s just different.  Think of it this way – is Fibre Channel technology superior to iSCSI?  Well, depending on your needs, you may choose to use either FC or iSCSI, and either technology might be best suited to you.  Your decision would be based on several factors that are important to your particular needs – things like performance, experience, flexibility, and cost.  There is no “right” answer here – in fact you might decide the best solution is to use FC for some apps and iSCSI for others.  It’s the same with Data Domain and NetApp, let me explain.

NetApp deduplication is based on post-processing architecture.  This means that data is not deduplicated immediately upon arrival at the NetApp system, but rather deduplication is scheduled to run some point on the future, presumably when all your User’s have gone home to have dinner with their families and the storage array is just sitting around looking for some extra work to do - that’s the nice thing about machines, they don’t need to eat or sleep or do those other things we humans waste time doing.  Turns out that this post-processing method is perfect for production storage applications - let the applications do their thing, don’t slow them down, and then reduce physical storage when they aren’t watching.

Data Domain deduplication, on the other hand, is based on inline architecture.  This means that each time data arrives at the Data Domain system, a realtime decision is made.  Should I store or reference this data?  The only way to make this decision is to compare this new data to all the other data previously stored on the system.  Turns out this is a good way to handle D2D backup data.  The data doesn’t need to be stored first before deduplication, and if the comparison process takes a little time to complete – well, its still faster than waiting for a tape to (hopefully) load and get itself into position to accept data.

So you see – these two architectures really solve two different needs.  Sure, technically you can use a NetApp FAS system to dedupe your D2D backups or you could use a Data Domain system to dedupe your production data, but if you did this you wouldn’t be leveraging the strength of either product.  Together though, leveraging the design aspects of each system, you could achieve optimal efficiency across your entire storage enterprise, and leverage the attributes of each system – all while sitting down for you own family dinner…

NetApp Deduplication Revisited

It’s been quite a ride here at NetApp since we launched deduplication 2 years ago. Thought I’d take some time to reflect on what I’ve witnessed in this short period of time. As I said in an earlier blog, no one on the original dedupe launch team (at that time we were still calling it A-SIS) could have envisioned that today we’d be sitting here looking at an installed base of over 37,000 systems. Our customers are running deduplication in virtually all industry segments, across scores of workloads (including primary production workloads), with use cases offering tremendous space savings. There are very few times, if any, in one’s career where something you are responsible for becomes an "overnight" success, but this was one of those cases. Let’s look back and see how this all happened.

NetApp deduplication development began back in 2004; when our technical directors realized that we could create a catalog of “fingerprints,” and use those fingerprints to identify deduplicate data blocks. The rest was easy – once the duplicates were identified, we could use our existing system of multiple block referencing (used in all our snap features) and create many pointers to a single data block, while releasing all the duplicate blocks back to the system for re-use. NetApp deduplication was born.

Over the next year or so, the development work started. The coding was actually pretty quick, what took far greater time was the QA testing. You see, when you store some of the world’s most valuable data, you better be very careful you are not going to break anything with any new feature, especially one that removes blocks of data.  One thing I am grateful for was the fact that the early designers insisted that we do a full byte-level comparisons before any deduplication occured, that turned out to be a very wise decision indeed.

Finally, in October of 2006, we were ready for field tests. Luckily, time was on our side – deduplication was still nascent and people weren’t clamoring for it (yet.) So we decided to conduct an “early release” program with 25 of our closest friends, er customers. Everyone on the development team held their collective breath and waited for the results to come in. Were we going to see system crashes? panics? data corruption? As it turns out, over this 6-month period not a single bug report was filed. I was astounded. Having been around the block a few times at other companies I knew how these things usually go. Marketing wants the widget before Engineering is ready to give it up, and there are constant “I told you so” meetings during this rough-and-ready period. But none of that happened in this case – a Marketeer’s dream.

So in March of 2007, we released NetApp deduplication to the world. What I failed to understand in those early days was our customer’s propensity to take risks and ignore our advice. “Just use it for archival data”, we said, figuring that was safe enough – we wanted to walk before we ran. Our customers, however, seemed to be running the 100-yard deduplication dash. They immediately began to use deduplication on user files, virtual machines, and all those other areas we cautioned them against – and on thier primary production volumes no less! During this period, I got the feeling I was raising another teenage son, telling him not to do something and then watching him do exactly what I told him not to do.

Time for my second moment of astonishment. NetApp deduplication worked on these applications. And worked. And worked. To my amazement, I heard over and over again that our customers were reporting significant space saving and no performance impact. So I did what any good marketing man would do. I promoted these use cases, both within NetApp and to our other Users. You should have seen the icy stares I got in engineering meetings. But I persisted and eventually won over the mob - but it really wasn’t me, it was a very well designed feature that I was lucky enough to inherit just at the right time.

Now let’s fast forward to today’s scene. Anyone who does anything in the data storage industry knows that deduplication is hot, white hot. NetApp has proven that deduplication can be used anywhere, including primary production data. Newcomers and established storage vendors today are making a lot of noise trying to grab a chunk of the dedupe pie. Through all this clamoring, lets not forget one fact - NetApp was in the right place at the right time; and changed the way that people view deduplication and storage efficiency. Come to think of it, maybe deduplication really is the teenager of the IT family. Bordering on maturity - but still with plenty of years to grow. Time will tell…but I like our position at the table today.

NetApp Teams Up To Offer Even More Deduplication

Some of you might have noticed this press release from CommVault, where they announced support for block-level deduplication in their latest release of Simpana.  Curious to know how they viewed NetApp deduplication in light of this new feature, I contacted old friend and Marketing VP Dave West to get his viewpoint.  To my relief, Dave stated that he viewed NetApp dedupe as complimentary to CommVault's and sent along some documentation for me to review.

After looking over the docs, I agree with Dave's assessment.  CommVault user's can enable Simpana's deduplication and remove duplicate blocks at the source, the main advantage being a reduction in the amount of data transmitted over the wire.  CommVault recommends that the examined/comparison size be 50KB or larger, presumably so that Simpana can find a reasonable amount of duplicates without overtaxing media server resources - this granularity can be adjusted by the user for optimal reduction/performance.

Now here's the cool part.  If CommVault is backing up to a NetApp FAS system, Simpana recognizes this and fires off a second dedupe process, this time directly on the FAS volume.  This accomplishes two things: First, since NetApp dedupe occurs at a 4KB level, some additional duplicates will likely be found.  Second, if multiple Simpana backup jobs are aggregated into one FAS volume, NetApp dedupe will sweep through all those datasets and remove duplicates across all backup jobs. Pretty slick..

Who says storage vendors can't get along?

When Is Dedupe not Dedupe?

Ocarina has been creating some buzz lately, promoting their ECOsystem deduplication for primary storage.  Trouble is, they don't seem to actually offer deduplication.  Here's a comment from Marketing VP Carter George:  "Dedupe for primary storage wasn't our first choice for a working description, but that's what everybody's calling it."  Hmm...so lets not confuse people with the facts, right Carter?  Lets call it dedupe because that's what people "want" to call it?  I'm sure the marketing Gods are smiling down at this use of creative marketing , but DrDedupe wants to seek the truth, not appease the Gods.

So I started doing some research.  The first stop on my quest was Ocarina's own website.  Looking under "Resources" I found a white paper called "Data Deduplication For Primary Storage."  Ah, this should put the discussion to rest.  Unfortunately this paper seem outdated, with quotes like "Currently, Ocarina Networks reduces only still images with video support planned for the next release in early 2009."   Hmmm...I'll keep looking.  Found two other papers, "The Primary Storage Handbook," parts 1 and 2.  Darn, these are just various viewpoints on how primary storage can be optimized, no specific information about Ocarina to be found.

Finally, I found a presentation "Reduce Your Storage Footprint" on Ocarina's site.  Then I read this quote:  "Ocarina Networks’ approach to storage optimization is built on content-aware compression rather than the more commonly used block deduplication metholodogy"  - other than pointing out the obvious need for a better spell-checker (metholodogy?), the quote does seem to put the matter to rest, Ocarina does not utilize deduplication, by their own admission.

Am I missing something here?  Is Ocarina intentionally misleading the market by using deduplication in name only but not in practice, or are they just not telling us the whole story.  My search for truth continues...