NetApp - Ask Dr Dedupe

The Dedupe 2.0 Pundits Are Still Swimming in Lake 1.0

I had to chuckle when I saw the concurrent announcements from Permabit and Storage Switzerland that dedupe 2.0 had arrived.  Apparently the acquisition of Data Domain by EMC had signaled the demise of dedupe 1.0, even though I failed to see any technology transformation as a result of that merger.  Data Domain is still doing the same thing they always did and EMC is still doing what they do.  There was no transformative event here guys.

Anyway, the people at Permabit seem to believe that hashing and deduplicating large archival stores is ushering a new era of deduplication and leaving point products behind.  But, er, isn't Permabit another point product?  Last time I looked in a data center there was more than archival data behind the blinking lights on those storage arrays.  In my estimation, deep archival consumes somewhere around 1% of the world's data storage.

Then there's Storage Switzerland.  I guess the Switzerland part is to have one believe that they are a neutral party amidst this vendor-against-vendor world.  But does anyone out there really think they are neutral?  I'll let my readers form their own conclusions.  Anyway, SS says that the core of dedupe 2.0 will be a foundational repository where all previously optimized data will come to rest from primary and secondary storage.  Hmmm, sounds strangely like an archival system to me.  Wait, doesn't Permabit sell archival systems, and they are also talking about dedupe 2.0?  Could there be a connection here?  Say it ain't so Storage Switzerland!

OK lets move back towards reality.  You know when dedupe 2.0 will really arrive?  When people stop talking about dedupe altogether.  Like many technologies that came before it, such as RAID, SCSI, Snapshots, and dozens of others, dedupe will become ubiquitous and acknowledged as a must-have feature from every serious storage vendor.  Inline or Post-process, source or target, local or global - it doesn't matter as long as it dedupes.  Dedupe will run silently in the background, eradicating those pesky duplicate data objects.  NetApp has proven that dedupe can run quite well on a unified platorm that services all tiers of storage (including archival), uses all standard protocols, and serves a wide breadth of applications.  Once a concept is proven, the next steps are constant refinement and optimization, and thats exactly what we are doing.  I can't tell you exactly the day dedupe 2.0 will arrive, but I am pretty sure that NetApp will be among the frontrunners.

DrDedupe

NetApp Teams Up To Offer Even More Deduplication

Some of you might have noticed this press release from CommVault, where they announced support for block-level deduplication in their latest release of Simpana.  Curious to know how they viewed NetApp deduplication in light of this new feature, I contacted old friend and Marketing VP Dave West to get his viewpoint.  To my relief, Dave stated that he viewed NetApp dedupe as complimentary to CommVault's and sent along some documentation for me to review.

After looking over the docs, I agree with Dave's assessment.  CommVault user's can enable Simpana's deduplication and remove duplicate blocks at the source, the main advantage being a reduction in the amount of data transmitted over the wire.  CommVault recommends that the examined/comparison size be 50KB or larger, presumably so that Simpana can find a reasonable amount of duplicates without overtaxing media server resources - this granularity can be adjusted by the user for optimal reduction/performance.

Now here's the cool part.  If CommVault is backing up to a NetApp FAS system, Simpana recognizes this and fires off a second dedupe process, this time directly on the FAS volume.  This accomplishes two things: First, since NetApp dedupe occurs at a 4KB level, some additional duplicates will likely be found.  Second, if multiple Simpana backup jobs are aggregated into one FAS volume, NetApp dedupe will sweep through all those datasets and remove duplicates across all backup jobs. Pretty slick..

Who says storage vendors can't get along?

Oh Beth (Pariseau), what can I do?

I've been a fan of Beth Pariseau's "Storage Soup" blog for quite a while now.  Beth tends to write on the provocative side, always looking for an interesting scoop to set her apart from the other writers in an industry that can be a little, um, well, boring at times.  I like that.  Anyway, Beth recently wrote an article for searchstorage that described the need for storage efficiency in today's crazy world where a dollar is worth about seven pesos.  In the article, Beth mentioned things like:

Focus on storage efficiency grows as budgets shrink

Analysts predict broad trend toward efficiency

Right on Beth!  The Dr is with you - storage efficiency has suddenly become cool in the data center.  But wait - I must have read the article too fast, there is no mention of NetApp.  I reread the article 3 more times and sure enough, deduplication is mentioned several times, but NetApp is as absent as a beauty queen at a Star Trek convention.  What about our 20,000+ systems and and over 5,000 customers already running NetApp dedupe, and loving it - how could you have missed that Beth?  All I could do is think of the lyrics from the old Kiss song, "Beth what can I do, oh Beth what can I do?" 

After thinking about this a little more, I decided what I really needed to do was to try and appeal to Beth  through that great song that bears her name.

So here it is, my personal appeal to Beth Pariseau, sung to the tune of "Beth":

 

Beth I hear you calling

But I can't come by right now

Me and the boys are working

And we really got it right 

NetApp has the magic

To control your data growth

I think I hear them calling

NetApp has dedupe

NetApp has dedupe

 

You say you feel so sorry

That your data's overflowing

I'ts out of control  and you know it

And you're just there all alone

Just a few more hours

And I'll be right back to you

I think I hear them calling

NetApp has dedupe

NetApp has ... dedupe

 

Beth your data's growin'

But I know you'll be alright

'Cause me and the boys have been working all night...all night

Thank You Dave Simpson (I think)

A colleague of mine recently sent me this excerpt from Dave Simpson's blog:

"For a humorous (and, well, odd) look at data de-duplication, check out NetApp's Dr. Dedupe Blog."

I really appreciate you mentioning my blog Dave - but am I really "odd"?  Well OK maybe I do tend to come off as unconventional, but really this is just my attempt to do a blog that is interesting, humorous, and at times even insightful.  As for my youtube videos, ok well I will admit they are odd, but again just my attempt at creating a little unconventional awareness around dedupe.

Anyway, for those of you that haven't heard of Dave Simpson, he is sort of an icon in the data storage industry, having been the one and only editor of Infostor since its inception many years ago.  I have always respected Dave (and still do) and look forward to reading his columns.  So you can imaging my pleasure when I read the December issue of Infostor where in Dave's editorial he called out deduplication as "the dominant storage technology of 2008."

But my joy soon turned into despair when I read Dave's closing editorial comment: "And lets give credit to the early pioneers of data deduplication, such as [name withheld], that gave users a technology that was so useful that they, in turn, forced the big vendors to offer a technology that in a way was not in their near-term best interests."

Why the despair, Dr Dedupe, you ask?  Because your company was not the [Name Withheld]?  No, I can live with the fact that there were other companies dedupe-ing before us.  What got me was the reference to the big vendors (I think he means us) doing something "not in their best interests" - in other words, giving stuff away for free.

I'll try not to get  too passionate about my company here, but I'd like to remind Dave that NetApp is the company that pioneered features such as snapshots, thin provisioning, and dual-parity RAID.  We don't charge for any of these features and they all allow our users to increase their effective storage utilization rates.  NetApp also pioneered the use of ATA/SATA drives on enterprise workloads, again providing a key breakthrough to help our users drive down their cost/GB.  So it should not have come as any real surprise that we didn't charge for our deduplication feature; it was simply another way for us to show that we want our customers to be able to store the maximum amount of data for the lowest possible cost.  Looking back on our history, it appears that storage efficiency and now deduplication is in NetApp's blood.  Our best interests have always been the same as our customer's best interests, and I believe this has been the critical component enabling our success.