NetApp - Ask Dr Dedupe

Deduplication and Hyper-V are bringing NetApp and Microsoft closer together

I don't want to say that dedupe was the only reason that NetApp was named Microsoft's Storage Solutions Partner of the Year, but I can dream can't I ?....

Steve Ballmer:  DrDedupe, on behalf of our bazillion Microsoft employees, I'd like to personally thank you for your contribution to our success.

DrDedupe:  Aw shucks Steve, it was nothing really.  Just a couple hundred lines of code the team wrote over a long weekend.  We were kind of bored you see and it seemed like a fun project...

OK seriously, as Hyper-V begins to gain in popularity, there is some real synergy going on between NetApp and Microsoft.  Check out this video, where can you see a demonstration of a Hyper-V LUN being created and then easily re-sized after being dedupe'd.

Using NetApp storage with Microsoft apps seems to makes alot of sense not only for Hyper-V but also for some other apps you might have heard of such as Exchange, SQL, and Sharepoint.

SQL and its nephew Sharepoint offer some interesting (and largely unexplored) use cases for dedupe that I will discuss in upcoming blogs.  Exchange, as I noted in a previous blog, has jettisoned single instance storage (SIS) in 2010, presenting yet another compelling opportunity for NetApp dedupe.  Yes - I know that Microsoft often espouses server-attached storage for Exchange, but sooner or later they'll realize that people stopped doing that around a decade ago - for many good reasons.  (Microsoft are you listening?)

Speaking of blogs, here's a good one that Nick Triantos posted about one of our newer features, SnapManager for Hyper-V.  This feature brings automation to Hyper-V for DR and D2D Backups and is integrated with Microsoft's Volume Shadow Service (VSS).  Very cool indeed, and OK maybe the real reason we won the partner award...

Anyway, if you are a Microsoft app user (sorry Sun) you owe it to yourself to check out what NetApp can do for you.  Our latest issue of Tech OnTap also features an article on the 5 best practices for Hyper-V.

DrDedupe

Exchange 2010 Dismisses SIS?

Microsoft Exchange revolutionized the way we communicate.  Its hard to imagine life without email.  Sure there are other email apps out there but aren't they all ladies-in-waiting to the true Queen of email - Microsoft Exchange?

Since version 4.0 was released (about the time the B-52's were singing "Love Shack") Exchange had a cool feature called Single Instance Storage, SIS for short.  SIS was the granddaddy of deduplication, and went something like this:

Email from Joe in Accounting:  "Hey check out this cool bootleg copy of "Love Shack!"

Email from Sally in Marketing, Fred in Shipping, Adam in Sales and about 10 other people "Hey check out what Joe sent me!" 

Well what Joe sent you and you sent to 10 friends and they sent to 10 friends was basically the same email with the same attachment sent over and over again.  The folks in Redmond figured this was going to be a regular thing, so they invented SIS.  With SIS, all those emails were reduced to a single instance, as long as they were held in the same Exchange Store.

Single Instancing saved storage capacity and was free - what's not to like?  Well fast-forward about 15 years and Microsoft had a problem.  More mailboxes, more attachments, and not only bootlegged audio files but attachments of all shapes and sizes - Word Docs, Powerpoints, PDF's, JPEGs, MPEGs, where will it stop?  So Microsoft did what any self-respecting software company would do, they said "you know, storage hardware is cheap these days, and all this SIS stuff is really slowing us down - so please don't use SIS anymore and just buy more storage, OK?

That's right.  Microsoft needed to destroy the monster it had created.  As of Exchange 2007 the message from Redmond was "Given current trends, we expect the value of single instance storage to continue to decline over time. It's too early for us to say whether SIS will be around in future versions of Exchange. However, we want everyone to understand that it is being deemphasized and should not be a primary factor in today's deployment or migration plans."  Worse yet, the word on the streets is that in Exchange 2010, SIS will disappear altogether.

Now before you come down with an acute case of ESISPTD (Exchange SIS Post-Traumatic Disorder) the Dr has some good news for you.  You can use NetApp deduplication to get back what Microsoft taketh away.  On tests run with Exchange 2010, without doing much optimization, we've seen seen space reductions of around 30% via dedupe.  Better yet, since NetApp dedupe is run in the background as a post-process procedure, performance impact on those users sharing their favorite files should be nominal.  I guess you could say that NetApp dedupe lets you go Back to the Future - It's free and it saves storage capacity - whats not to like?

DrDedupe

The Evolution of the Storage Brain - Applications Run Faster With Deduplication

It seems a contradiction - improving system performance with dedupe?  Usually, when someone thinks about implementing dedupe on a storage system, the first thought is "hmmmm...how much is this going to slow down my applications?"  In this blog I am going to tell you how dedupe can actually speed up your applications.

In Data ONTAP 7.3.1, NetApp introduced "Intelligent Caching."  Intelligent Caching refers to FAS and V-Series system memory, as well as secondary memory.  By secondary memory I am referring to the Performance Acceleration Module, or "PAM" for short.  NetApp introduced PAM II in this press release, although it was buried in with a bunch of other important announcements so it was easily missed.  I'll  come back to PAM II later.

So how does system memory, secondary memory, and Intelligent Caching relate to dedupe?  Simple.  When a NetApp storage system holds a dedupe'd data block in its memory, it also holds all the information about the data pointers that reference that particular block.  If this block has been deduped, say, 100 times, and is currently held in memory, there is no need to access the block from disk again, regardless of who is requesting the data block.  The result is faster access to data via reduced disk I/O latency.

The ramifications of this become very interesting.  The higher the dedupe ratio, the more dedupe'd blocks are placed into memory and the faster the system becomes.  We've already run benchmarks of this during VDI boot storms (when everyone turns on their virtual desktops at the same time and the storage system is inundated with read requests) and the results with dedupe and Intelligent Caching show orders of magnitude improvement in read response times.  With PAM II, the total system cache can exceed 2TB, and as Foxy Cleopatra might say, "Now that's a WHOLE lot of memory."  With this larger cache, the laws of probability begin to take affect - a larger cache means more blocks can be cached, and frequently requested blocks will remain in cache longer.  Meanwhile, since dedupe'd blocks have multple pointers they have a higher probability of being cached ...you get the idea.

To me, the combination of intelligent caching and deduplication is a signal that we are entering a new era in the maturity of deduplication, where the intelligence of deduplication is combined with other intelligences (intellligenci?) of the storage system to bring breakthrough results - the evolution of the storage brain.  I'll be exploring other thoughts on this topic in upcoming blogs...

Deduplication and The Long Arm of the Law

Check out this press release and you'll see a couple interesting facts:

• NetApp has delivered deduplication to over 7,200 customers and over 37,000 systems
• Cohasset evaluated deduplication with SnapLock and found that it meets SEC regulations for electronic records management

Each of these items is a milestone of sorts.  Having been part of the original dedupe launch team back in the Spring of 2007, I can tell you that none of us would have imagined that within two years we would be approaching 40,000 deployments - in fact our "stretch" goal was around a thousand systems in the first year - we beat that by about 10,000 systems.

The success of deduplication at NetApp is a testament to the vision of our development team - they recognized yet another way to leverage the intelligence of Data ONTAP, and to our customers, who were willing to put their faith in our design and prove that production data could be dedupe'd just as easily as backup data.

The second part of today's announcement is Cohasset's evaluation of dedupe for compliance records.  Cohasset Associates is the acknowledged authority in electronic records management.  For decades, storage vendors have turned to Cohasset to make sure their archival and compliance products passed the sniff test. 

To validate products, Cohasset leans heavily on SEC rule 17a, probably the most stringent test of any government agency for legal record compliance.  The SEC is interested in two key criteria - A) Is the record stored in a state that cannot be altered by users? and B) Can the record always be recovered back to its original state?  When Cohasset evaluated NetApp deduplication, there were two things that stood out that eventually resulted in two thumb's up from them:

1) Before deduplication - a byte-level comparison takes place to eliminate data corruption as a result of hash collisions

2) After deduplication, the only change to the original data are re-addressed data pointers, which cannot be altered or otherwise manipulated by users, and does not involve any external hash tables or containers.

Deduplication of legal records has been a controversial topic for some time.  Hopefully our announcement today makes this topic is somewhat less controversial...and before I close I'd to make sure I say thanks to the thousands of NetApp dedupe users that have made this the fastest-growing feature in NetApp history!

Any Wiki-ers out there?

A friend of mine emailed me the other day to let me know a new Wikipedia entry had been created for "Storage Efficiency."  I took a look and sure enough there it was -

Storage Efficiency

From Wikipedia, the free encyclopedia

The ability to store and manage data that consumes the least amount of space with little to no impact on performance; resulting in a lower total operational cost. Efficiency addresses the real-world demands of managing costs, reducing complexity and limiting risk. Storage efficiency may be measured by a simple formula: Storage Efficiency = Allocated Capacity / Raw Capacity. Ultimately, efficient solutions must go beyond simple hardware and software offerings and connect at a human level.

Thats it??  Er, ahem (imagine the sound of me clearing my throat)..does this writer happen to know that deduplication is the major contributor to all storage efficiency?  Connect at the human level?  Aren't we humans the ones that created all this duplicate data to begin with?  We need to connect at the non-human level to improve the efficiency of stored data.  Go beyond simple hardware and software?  Dear Wikipedia: deduplication is not a weekend science fair project.

My friends, this definition is precisely why the Dr can never rest.  Millions (ok hundreds) of people will read this Wiki and never associate deduplication with storage efficiency.  I'd love to go on the Wiki myself and set the record straight, but my company for some reason frowns on me blatantly using Wikipedia as a way to get free advertising for deduplication (who me??) - but that doesn't mean that (sound of me clearing mythroat again) ummm, that you can't help set to record straight on storage efficiency!

So put your editorial caps on, log onto Wikipedia, and help make the world a better place through deduplication. The Dr is counting on you!

Is Hashing Legal?

 

Hashing, Hacking, Phishing, Spoofing, Spamming - they all sound like bad things, right?  Not necessarily.  Don't be fooled by a name, hashing is one of the good guys of computer technology, and shouldn't be associated with the dark side of computing like those other terms above.  Hashing can be your friend, but what is it anyway?

Hashing algorithms have been around since the dawn of computing:  Think of the dead languages - Pascal, Cobol, Fortran, take your pick they all used hashing.  Simply stated, a hash is an easy way to look up data in a large logical table, or more importantly can be used to calculate a small number that represents a much larger number.

Here's an example:  take the number 1,050,235,750.  If I asked you to memorize this number and then came back a day later and asked you to repeat it, I bet you'd have a hard time recalling it from memory.  But if we add all the integers together, we get the number 28 (1+0+5+0+2+3+5+7+5+0=28.)  28 is a much easier number to remember, and in this example the number 28 is actually a hash value representing the larger number 1,050,235,750.  Of course, the problem here is that a lot of numbers have integers that add up to 28 (for instance the number 8,488) so if you use this simple hashing system you'll quickly find yourself seeing constant hash "collisions" where many different numbers have integers that hash to the same value.

What does hashing have to do with deduplication, Dr Dedupe?  Well, everything, my friend.  Nearly every implementation of deduplication depends on hashing functions to identify data "chunks" that are stored on disk storage systems. 

Depending on the flavor of deduplication (source or target, see my previous blog for more on that) hash calculations are performed either as the data leaves the source or when it arrives at the destination.  These hash values are stored as digital "fingerprints" that identify each piece of data written to the system.

Hopefully now you are starting to get the picture.  As new data enters the scene, it is hashed and compared to every other hash (or fingerprint) on the system.  If the same fingerprint already exists on the system, the new data can be discarded and simply referenced to the previously written data.  If the new fingerprint does not match with any other fingerprint, then the data is considered unique and must be retained on disk.

But wait, Dr Dedupe.  What about those hash collisions?  If you depend solely on hash calculations, doesn't that mean there is a chance of removing the wrong data by mistake?  Can two completely different data chunks hash to the same value?  Why yes they can, my perceptive reader.  No matter how sophisticated the hash algorithm, there is a mathematical probability of hash collisions.  And that is precisely why NetApp uses a two-step approach to deduplication - we use hashes to identify "candidates" but always do a full byte-for-byte comparison before deleting any data - this guarantees that we will not accidentally corrupt your data due to a false match.  Too bad all deduplication vendors don't go to the same lengths to protect your data.

Well there you have it, my traveling companion, the complex world of hashing hopefully made easy.  And we've only just begun in our journey to unravel the mysteries of deduplication.  Hey I see that there's a pub just up the road, lets take a short refreshing pause before continuing...

The great debate - source or target deduplication, which is better?

 

One of the ongoing debates about deduplication of D2D backups is where the deduplication process should actually occur.  There are two established camps:

1.  The Red Team - Source deduplication.  Their position is that it is always better to dedupe before the data is sent to the backup target because this reduces network traffic and also substantially reduces your backup window - you are sending only non-redundant data during each backup session.  Think of it as "incrementals-forever", only better, since each backup still "looks" like a full backup image.

2. The Blue Team - Target deduplication.  Their position is that deduplication should always be performed at the backup target, since this way all backups can be aggregated into a single storage system.  Heck, everyone knows that the more data you have, the more chance to find redundant data, right?  And yes you are still sending all the data for each backup over the network but you were doing that with tape for years, weren't you?.

Hmmm....which team is right?  They both seem to make valid points.  What is more important, network optimization and reduced backup windows or storage savings on my backup disks?  And by the way, why do I always have to make trade-offs like this?

Enter NetApp SnapVault combined with NetApp Deduplication - the white team (because good guys always wear white)

3.  The White Team - Combined source and target deduplication.  The NetApp position is that you don't have to chose between red and blue, you can have the best of both worlds.  Lets examine how SnapVault and deduplication work together for optimal D2D backups.

Step One.  Perform your SnapVault baseline transfer.  This is a full D2D backup of your source volume.  Once the backup is complete, SnapVault checks to see if deduplication is enabled on the destination volume.  If it is, deduplication commences automatically and removes data redundancy from this baseline copy.

Step Two.  Set up your schedule for subsequent SnapVault backups.  SnapVault uses Snapshot technology to take a "differential" view of the data in the source volume and only transfers data to the destination that has changed since the last SnapVault session.  This dramatically reduces the amount of data sent over the network, and reduces backup windows.  Some people call this "delta deduplication" or "differential deduplication", but more on that in a later blog... the important point is that each time a SnapVault session completes, deduplication is also automatically performed.  Think of it as two layers of storage reduction that work together harmoniously.

Step Three.  OK well actually there is no step three, unless of course you need to actually recover any data that you backed up.  SnapVault provides a full filesystem view for each backup session, even if only one file changed between backups.  This makes it very easy for users to retrieve point-in-time copies of files, folders, or entire directories with a few simple clicks or commands.

There you have it, this great debate is now over.  Both the Red team and the Blue team have proven their points, and after the mandatory handshakes, a solution has been agreed to - combining the best ideas from both teams.  Exactly the same way our upcoming presidential debates will conclude (maybe?)