NetApp - Ask Dr Dedupe

The Cost of Efficient Storage can be ZIP

Storage efficiency is a funny thing.  Vendors these days are asking customers to spend money so that they can save money on their data storage.  Huh? This is a bit of a paradox.  Here's the example I use, lets say my wife comes home with a brand new pair of shoes.  "Hey honey, look at these great shoes I just bought!  Regular price was $200 but I got them for 50% off!"  My wife sees this transaction as saving us $100, but I see the transaction as $100 gone from the family checking account, never to be seen again.  Who has the correct perspective?  Well, that all depends on just how necessary those shoes were.  If the closet is overflowing with shoes just like the new ones, this was probably a wasted expense.  However if the shoes were bought to perform a specific function that no other shoe could accommodate (like this), then its probably a worthwhile investment.

Storage efficiency is kind of like that.  You shouldn't invest in it unless it provides some type of new functionality that you don't already own.  Plenty of vendors will offer to sell you cheaper storage, with heavy discounts to entice you, but if its just like all the other storage already in your closet, what are you buying?  More shoes just like the ones you already have?  When you invest in storage efficiency, you should look for something different, something that lets you do things you haven't been able to do before.

Why am I bringing this up?  This week NetApp rolled out a new promo called ZIP, or Zero Investment Promise.  Here's the concept:  instead of buying more shoes, er I mean storage, why not make your existing storage more efficient?  With ZIP, NetApp will loan you a shiny new V-Series Open Storage Controller, but before it hits your dock a promise is made that the V-Series will immediately save you x, (x is defined once we do an onsite evaluation, for free.)  We'll put this promise in writing and give you 90 days to run your tests.  If your tests deliver what we promised, you buy the V-Series (heck its probably already paid for itself by now anyway) - but if you believe we broke our promise, and we agree - the V-Series is yours to keep - gratis.

OK get ready, here comes the fine print.  You and your data center must be in North America (some countries strangely have a problem with promos like this), you cannot already be using V-Series (duh), you have to be a current user of EMC Clariion or HP EVA systems, and you have to agree to use 5 key features on your EMC and HP system, provided via the V-Series (Snapshot, FlexClone, Thin Provisioning, Thin Replication, and of course Deduplication).  That's about it.

So check it out.  And don't assume you have pay alot to bring efficient storage into your world.  In this case, you'll pay ZIP.

DrDedupe

NetApp Replaces OR with AND

I spent last week in New York City, where I was asked to brief many of our largest customers on the merits of deduplication and how it fits into our storage efficiency story.  I also walked the streets of Manhattan in my Dr's garb with a film crew and conducted spontaneous "Man on the Street" interviews, but more on that in another blog...

Anyway back to my briefings.  Most, but not all, of the dozen or so customers I talked to were already implementing deduplication and praised its effectiveness.  A few of the customers I briefed were not actually customers, but rather perspective customers that wanted to learn more about the technology behind NetApp storage efficiency and how it could help them improve their current storage environment.  One thing I've mentioned before but bears repeating is that NetApp deduplication is not limited to NetApp storage systems.  Our V-Series open storage controller brings deduplication and storage efficiency to HP, Sun, EMC and many other arrays.  We didn't make those storage systems, we just made them better.

This brings me to the title of today's blog and the message I always convey during my briefings.  At NetApp, we never ask you to sacrifice performance or resiliency for the sake of reduced storage capacity.  In fact, we'll improve your ability to quickly and safely serve data to your users and applications while at the same time reducing your physical storage footprint.  Thats the message behind our new ad campaign - Replace "Or" with "And" - a clever play on words but one we take to heart.  If you are interested in seeing more - here's a link to our new ad page - including a video I recorded on the topic, yes I do serious videos too.  Also, heres a link to our revamped storage efficiency page featuring yours truly again.  I wonder if the Dr runs the risk of being over-exposed?  Nah you can never see too much of DrDedupe.  Enjoy-

Exchange 2010 Dismisses SIS?

Microsoft Exchange revolutionized the way we communicate.  Its hard to imagine life without email.  Sure there are other email apps out there but aren't they all ladies-in-waiting to the true Queen of email - Microsoft Exchange?

Since version 4.0 was released (about the time the B-52's were singing "Love Shack") Exchange had a cool feature called Single Instance Storage, SIS for short.  SIS was the granddaddy of deduplication, and went something like this:

Email from Joe in Accounting:  "Hey check out this cool bootleg copy of "Love Shack!"

Email from Sally in Marketing, Fred in Shipping, Adam in Sales and about 10 other people "Hey check out what Joe sent me!" 

Well what Joe sent you and you sent to 10 friends and they sent to 10 friends was basically the same email with the same attachment sent over and over again.  The folks in Redmond figured this was going to be a regular thing, so they invented SIS.  With SIS, all those emails were reduced to a single instance, as long as they were held in the same Exchange Store.

Single Instancing saved storage capacity and was free - what's not to like?  Well fast-forward about 15 years and Microsoft had a problem.  More mailboxes, more attachments, and not only bootlegged audio files but attachments of all shapes and sizes - Word Docs, Powerpoints, PDF's, JPEGs, MPEGs, where will it stop?  So Microsoft did what any self-respecting software company would do, they said "you know, storage hardware is cheap these days, and all this SIS stuff is really slowing us down - so please don't use SIS anymore and just buy more storage, OK?

That's right.  Microsoft needed to destroy the monster it had created.  As of Exchange 2007 the message from Redmond was "Given current trends, we expect the value of single instance storage to continue to decline over time. It's too early for us to say whether SIS will be around in future versions of Exchange. However, we want everyone to understand that it is being deemphasized and should not be a primary factor in today's deployment or migration plans."  Worse yet, the word on the streets is that in Exchange 2010, SIS will disappear altogether.

Now before you come down with an acute case of ESISPTD (Exchange SIS Post-Traumatic Disorder) the Dr has some good news for you.  You can use NetApp deduplication to get back what Microsoft taketh away.  On tests run with Exchange 2010, without doing much optimization, we've seen seen space reductions of around 30% via dedupe.  Better yet, since NetApp dedupe is run in the background as a post-process procedure, performance impact on those users sharing their favorite files should be nominal.  I guess you could say that NetApp dedupe lets you go Back to the Future - It's free and it saves storage capacity - whats not to like?

DrDedupe

The Evolution of the Storage Brain - Applications Run Faster With Deduplication

It seems a contradiction - improving system performance with dedupe?  Usually, when someone thinks about implementing dedupe on a storage system, the first thought is "hmmmm...how much is this going to slow down my applications?"  In this blog I am going to tell you how dedupe can actually speed up your applications.

In Data ONTAP 7.3.1, NetApp introduced "Intelligent Caching."  Intelligent Caching refers to FAS and V-Series system memory, as well as secondary memory.  By secondary memory I am referring to the Performance Acceleration Module, or "PAM" for short.  NetApp introduced PAM II in this press release, although it was buried in with a bunch of other important announcements so it was easily missed.  I'll  come back to PAM II later.

So how does system memory, secondary memory, and Intelligent Caching relate to dedupe?  Simple.  When a NetApp storage system holds a dedupe'd data block in its memory, it also holds all the information about the data pointers that reference that particular block.  If this block has been deduped, say, 100 times, and is currently held in memory, there is no need to access the block from disk again, regardless of who is requesting the data block.  The result is faster access to data via reduced disk I/O latency.

The ramifications of this become very interesting.  The higher the dedupe ratio, the more dedupe'd blocks are placed into memory and the faster the system becomes.  We've already run benchmarks of this during VDI boot storms (when everyone turns on their virtual desktops at the same time and the storage system is inundated with read requests) and the results with dedupe and Intelligent Caching show orders of magnitude improvement in read response times.  With PAM II, the total system cache can exceed 2TB, and as Foxy Cleopatra might say, "Now that's a WHOLE lot of memory."  With this larger cache, the laws of probability begin to take affect - a larger cache means more blocks can be cached, and frequently requested blocks will remain in cache longer.  Meanwhile, since dedupe'd blocks have multple pointers they have a higher probability of being cached ...you get the idea.

To me, the combination of intelligent caching and deduplication is a signal that we are entering a new era in the maturity of deduplication, where the intelligence of deduplication is combined with other intelligences (intellligenci?) of the storage system to bring breakthrough results - the evolution of the storage brain.  I'll be exploring other thoughts on this topic in upcoming blogs...

Here Comes The EPA

First a little background - in December 2006 our Congress passed Public Law 109-431, directing the Environmental Protection Agency to perform a study of energy consumption in U.S. data centers.  Dutifully, in August of 2007 the EPA responded with this report, outlining an alarming trend in electricity consumption within these power hungry bits of real estate.

In their report, the EPA estimated that in 2006, data center storage devices consumed 3.6 Billion kilowatt hours of electricity. This gave storage systems the dubious role of being the #2 power hog of all data center device categories (volume servers took top honors.)  Furthermore, in their policy recommendations, the EPA suggested "ENERGY STAR specifications for data center products

— Label products that have the best energy performance on the market, to make it easy for purchasers to identify and buy these products."

As a result of these findings, for the past year the EPA has been working with SNIA'a Green Storage Initiative to define an Energy Star Rating for storage systems.  A few weeks ago, the EPA released the Energy Star Enterprise Storage Draft Specification and opened the spec to public comment.  These comments will be discussed in San Jose next week at aSNIA Symposium.

So why bring this up in my dedupe blog?  Because the EPA recognizes the value of deduplication (and other software efficiencies) in contributing to the greening of storage.  From the EPA draft specification:

"EPA understands that there are many software-based approaches to improving the energy efficiency of storage products. The benefits of virtualization, data deduplication, and other software-based data management techniques are well documented. These software solutions, perhaps even more so than the hardware itself, are heavily customized for specific customers and applications. Achieving maximum efficiency gains is highly dependent upon proper software architecture, implementation, operation, and maintenance by individual users. A key objective of the ENERGY STAR specification is to identify and reward storage solutions that seamlessly integrate software and hardware efficiency strategies that provide verifiable benefits without user intervention."

I'll be reporting periodically on the progress of this initiative as new developments warrant.  Some day you just might see an Energy Star logo on your favorite storage system...

My View On Data Domain

In a comment to my last blog “NetApp Dedupe Revisited” - Glenn asked “does NetApp really need Data Domain more than EMC for purely technical reasons and to fix the shortcomings of A-SiS?”

This is a question I’ve heard a few times since NetApp announcedits intention to acquire Data Domain a few weeks ago.  It’s an important question Glenn, one that warrants a blog of its own.  Before I answer your question though, I want to stress that the transaction between DDUP and NTAP is still in process and subject to scrutiny by regulators and shareholders.

Now, back to the question – is Data Domain’s technology superior to NetApp’s?  No, it’s just different.  Think of it this way – is Fibre Channel technology superior to iSCSI?  Well, depending on your needs, you may choose to use either FC or iSCSI, and either technology might be best suited to you.  Your decision would be based on several factors that are important to your particular needs – things like performance, experience, flexibility, and cost.  There is no “right” answer here – in fact you might decide the best solution is to use FC for some apps and iSCSI for others.  It’s the same with Data Domain and NetApp, let me explain.

NetApp deduplication is based on post-processing architecture.  This means that data is not deduplicated immediately upon arrival at the NetApp system, but rather deduplication is scheduled to run some point on the future, presumably when all your User’s have gone home to have dinner with their families and the storage array is just sitting around looking for some extra work to do - that’s the nice thing about machines, they don’t need to eat or sleep or do those other things we humans waste time doing.  Turns out that this post-processing method is perfect for production storage applications - let the applications do their thing, don’t slow them down, and then reduce physical storage when they aren’t watching.

Data Domain deduplication, on the other hand, is based on inline architecture.  This means that each time data arrives at the Data Domain system, a realtime decision is made.  Should I store or reference this data?  The only way to make this decision is to compare this new data to all the other data previously stored on the system.  Turns out this is a good way to handle D2D backup data.  The data doesn’t need to be stored first before deduplication, and if the comparison process takes a little time to complete – well, its still faster than waiting for a tape to (hopefully) load and get itself into position to accept data.

So you see – these two architectures really solve two different needs.  Sure, technically you can use a NetApp FAS system to dedupe your D2D backups or you could use a Data Domain system to dedupe your production data, but if you did this you wouldn’t be leveraging the strength of either product.  Together though, leveraging the design aspects of each system, you could achieve optimal efficiency across your entire storage enterprise, and leverage the attributes of each system – all while sitting down for you own family dinner…

NetApp Teams Up To Offer Even More Deduplication

Some of you might have noticed this press release from CommVault, where they announced support for block-level deduplication in their latest release of Simpana.  Curious to know how they viewed NetApp deduplication in light of this new feature, I contacted old friend and Marketing VP Dave West to get his viewpoint.  To my relief, Dave stated that he viewed NetApp dedupe as complimentary to CommVault's and sent along some documentation for me to review.

After looking over the docs, I agree with Dave's assessment.  CommVault user's can enable Simpana's deduplication and remove duplicate blocks at the source, the main advantage being a reduction in the amount of data transmitted over the wire.  CommVault recommends that the examined/comparison size be 50KB or larger, presumably so that Simpana can find a reasonable amount of duplicates without overtaxing media server resources - this granularity can be adjusted by the user for optimal reduction/performance.

Now here's the cool part.  If CommVault is backing up to a NetApp FAS system, Simpana recognizes this and fires off a second dedupe process, this time directly on the FAS volume.  This accomplishes two things: First, since NetApp dedupe occurs at a 4KB level, some additional duplicates will likely be found.  Second, if multiple Simpana backup jobs are aggregated into one FAS volume, NetApp dedupe will sweep through all those datasets and remove duplicates across all backup jobs. Pretty slick..

Who says storage vendors can't get along?

Deduplication and The Long Arm of the Law

Check out this press release and you'll see a couple interesting facts:

• NetApp has delivered deduplication to over 7,200 customers and over 37,000 systems
• Cohasset evaluated deduplication with SnapLock and found that it meets SEC regulations for electronic records management

Each of these items is a milestone of sorts.  Having been part of the original dedupe launch team back in the Spring of 2007, I can tell you that none of us would have imagined that within two years we would be approaching 40,000 deployments - in fact our "stretch" goal was around a thousand systems in the first year - we beat that by about 10,000 systems.

The success of deduplication at NetApp is a testament to the vision of our development team - they recognized yet another way to leverage the intelligence of Data ONTAP, and to our customers, who were willing to put their faith in our design and prove that production data could be dedupe'd just as easily as backup data.

The second part of today's announcement is Cohasset's evaluation of dedupe for compliance records.  Cohasset Associates is the acknowledged authority in electronic records management.  For decades, storage vendors have turned to Cohasset to make sure their archival and compliance products passed the sniff test. 

To validate products, Cohasset leans heavily on SEC rule 17a, probably the most stringent test of any government agency for legal record compliance.  The SEC is interested in two key criteria - A) Is the record stored in a state that cannot be altered by users? and B) Can the record always be recovered back to its original state?  When Cohasset evaluated NetApp deduplication, there were two things that stood out that eventually resulted in two thumb's up from them:

1) Before deduplication - a byte-level comparison takes place to eliminate data corruption as a result of hash collisions

2) After deduplication, the only change to the original data are re-addressed data pointers, which cannot be altered or otherwise manipulated by users, and does not involve any external hash tables or containers.

Deduplication of legal records has been a controversial topic for some time.  Hopefully our announcement today makes this topic is somewhat less controversial...and before I close I'd to make sure I say thanks to the thousands of NetApp dedupe users that have made this the fastest-growing feature in NetApp history!

Any Wiki-ers out there?

A friend of mine emailed me the other day to let me know a new Wikipedia entry had been created for "Storage Efficiency."  I took a look and sure enough there it was -

Storage Efficiency

From Wikipedia, the free encyclopedia

The ability to store and manage data that consumes the least amount of space with little to no impact on performance; resulting in a lower total operational cost. Efficiency addresses the real-world demands of managing costs, reducing complexity and limiting risk. Storage efficiency may be measured by a simple formula: Storage Efficiency = Allocated Capacity / Raw Capacity. Ultimately, efficient solutions must go beyond simple hardware and software offerings and connect at a human level.

Thats it??  Er, ahem (imagine the sound of me clearing my throat)..does this writer happen to know that deduplication is the major contributor to all storage efficiency?  Connect at the human level?  Aren't we humans the ones that created all this duplicate data to begin with?  We need to connect at the non-human level to improve the efficiency of stored data.  Go beyond simple hardware and software?  Dear Wikipedia: deduplication is not a weekend science fair project.

My friends, this definition is precisely why the Dr can never rest.  Millions (ok hundreds) of people will read this Wiki and never associate deduplication with storage efficiency.  I'd love to go on the Wiki myself and set the record straight, but my company for some reason frowns on me blatantly using Wikipedia as a way to get free advertising for deduplication (who me??) - but that doesn't mean that (sound of me clearing mythroat again) ummm, that you can't help set to record straight on storage efficiency!

So put your editorial caps on, log onto Wikipedia, and help make the world a better place through deduplication. The Dr is counting on you!

And the winner is...NetApp - No, wait its VMware. Oh heck we are both winners.

In what continues to be a torrid week for deduplication, Storage Magazine announced its winners for the 2008 Storage Products of the Year.  In the category of "Storage Management Tools," eleven tools were evaluated by the judges.  After a hotly contested battle that probably involved multiple backroom deals by the judges (ok I made that part up) the Gold Medal winner was VMware's Storage VMotion followed closely by Silver Medal winner NetApp deduplication (ok I made that up too, I am not really sure how close the voting was), and the Bronze Medal was won by Nirvanix CloudNAS (not sure exactly who they are but I bet they have a great garage band).

Anyway, much as I was hoping we'd grab the Gold, finishing second to VMware is not all that bad, especially when you consider that we were competing with something that's a free part of our operating system and is fundamentally one system command (sis start).

Hats off to all the finalists and especially the other two companies standing on the  podium with us.  It is always great to receive industry honors and we humbly accept the Silver Medal from Storage Magazine on behalf of the 5,000 customers running NetApp deduplication today.

DrDedupe