The Evolution of the Storage Brain - Applications Run Faster With Deduplication

It seems a contradiction - improving system performance with dedupe?  Usually, when someone thinks about implementing dedupe on a storage system, the first thought is "hmmmm...how much is this going to slow down my applications?"  In this blog I am going to tell you how dedupe can actually speed up your applications.

In Data ONTAP 7.3.1, NetApp introduced "Intelligent Caching."  Intelligent Caching refers to FAS and V-Series system memory, as well as secondary memory.  By secondary memory I am referring to the Performance Acceleration Module, or "PAM" for short.  NetApp introduced PAM II in this press release, although it was buried in with a bunch of other important announcements so it was easily missed.  I'll  come back to PAM II later.

So how does system memory, secondary memory, and Intelligent Caching relate to dedupe?  Simple.  When a NetApp storage system holds a dedupe'd data block in its memory, it also holds all the information about the data pointers that reference that particular block.  If this block has been deduped, say, 100 times, and is currently held in memory, there is no need to access the block from disk again, regardless of who is requesting the data block.  The result is faster access to data via reduced disk I/O latency.

The ramifications of this become very interesting.  The higher the dedupe ratio, the more dedupe'd blocks are placed into memory and the faster the system becomes.  We've already run benchmarks of this during VDI boot storms (when everyone turns on their virtual desktops at the same time and the storage system is inundated with read requests) and the results with dedupe and Intelligent Caching show orders of magnitude improvement in read response times.  With PAM II, the total system cache can exceed 2TB, and as Foxy Cleopatra might say, "Now that's a WHOLE lot of memory."  With this larger cache, the laws of probability begin to take affect - a larger cache means more blocks can be cached, and frequently requested blocks will remain in cache longer.  Meanwhile, since dedupe'd blocks have multple pointers they have a higher probability of being cached ...you get the idea.

To me, the combination of intelligent caching and deduplication is a signal that we are entering a new era in the maturity of deduplication, where the intelligence of deduplication is combined with other intelligences (intellligenci?) of the storage system to bring breakthrough results - the evolution of the storage brain.  I'll be exploring other thoughts on this topic in upcoming blogs...

Comments

[Steven Schwartz]

Um....so basically what you are saying is that a heavy caching model(or specialized nvram or flash based "PAM") improves static data access times? Like the spin!

[DrDedupe]

Hi Steven, yes heavy caching will of course improves access times on its own - but combined with dedupe-aware cached blocks you get another layer of improvement. i.e. An app calls for a block and its pulled into cache. Then another app calls the same block but from a different pointer and hey! its already in cache! More efficient use of cache, would you agree?

[Chuck Hollis]

I think you guys have a lot to learn regarding the real-world effects of read caching in storage subsystems.

I think your breathless euphoria will wear off pretty soon.

Here's why: LOR (locality of reference) read caching plays out in multiple levels of the stack -- for example, the Oracle database has its read cache and the UNIX operating system of course has its buffers as well.

Servers have lots of memory these days, dontcha know? Not only that, applications tend to have good knowledge of what needs to be buffered, and what doesn't.

By the time the read I/O stream gets to the storage subsystem, the vast majority of "easy pickings" has been taken by upper levels of caching. Not much left for the storage cache to go work with -- there's some benefit, but not as much you'd think.

Now, as with everything else, there are exceptions, but unless you're contemplating a brute force approach of sucking the majority of a file system or LUN group into cache, it won't be as compelling.

Write caching is a different animal entirely, but it's my understanding that PAM can't be used for writes as it's not considered nonvolatile.

I have formed this opinion from my experiences at EMC working with a storage architecture with a very large cache and some very smart algorithms for over a decade, namely Symmetrix.

Now, if y'all would get around to putting some of those STEC drives on your array, you'd see some *real* random read performance!

-- Chuck

[DrDedupe]

Chuck I couldn't help detect a note of cynicism in your comment. Remember, we can disagree without being disagreeable.

Anyway, regarding your point about write caching, ummm, NetApp has been doing that since the first Toaster rolled off the assembly line, and PAM extends the use of write caching. Although this is a good topic for a future post, it has nothing to do with my post, which discusses caching dedupe'd blocks to improve application performance. As far as LOR, I believe your argument lacks merit. Yes OS's and applications do prefetch, cache, and use buffers, so then why all the fuss about Flash and SSD? Because benchmarks from many storage vendors point out the concrete benefits of read and write caching, and my blog simply pointed out how NetApp added a little more intelligence...

[kostadis roussos]

@chuck

Someday, maybe, you'll understand why NetApp storage systems don't need a write cache.

Maybe.

But in the meantime, for the rest of the world I wrote an explanation on why a FAS system does not require additional write cache, unlike Traditional Legacy Arrays.

http://blogs.netapp.com/extensible_netapp/2009/04/understanding-wafl-performance-the-f-word.html

And while you're reading about why a write cache isn't necessary you might want to check out my series on the DAS disruption.

http://blogs.netapp.com/extensible_netapp/2009/08/the-das-disruption-a-summary-so-far.html

The SSD is a technological dead end for the TLA. Even the symm architects figured that out which is why they are so aggressively pushing FAST.

But all of that is irrelevant vendor chuck-kostadis FUD bashing.

The real point is that intelligent caching is goodness and Dedup + PAM is a new kind of intelligent caching that we've never seen before in the storage industry.

kostadis

[Mark Woods]

Hi Chuck,

Following the logic of your argument, one might conclude that there really isn’t much need for fast storage.

“By the time the read I/O stream gets to the storage subsystem, the vast majority of 'easy pickings' has been taken by upper levels of caching. Not much left for the storage cache to go work with -- there's some benefit, but not as much you'd think.”

So, why is EMC pushing SSDs so hard? Ah … that must be the exceptions part.

“Now, as with everything else, there are exceptions, but unless you're contemplating a brute force approach of sucking the majority of a file system or LUN group into cache, it won't be as compelling.”

So, you are saying don’t bother to make your storage system fast unless you go at it in a big way to handle what is not “easy pickings” for upper levels of caching?

Okay, right. You have a Ph.D. from Symmetrix University. So, I’ll pick up on your lecture points about large cache size, write caching, etc.

We can go big. PAM II is available in 256GB and 512GB size modules. You can configure up to 2TB of unified read cache per controller; up to 4TB total cache per storage system. With deduplication in play, the effective cache sizes can be even larger.

PAM caches hot data from ALL storage behind the controller, including SATA drives. You don’t have to tier the data. The administrator doesn’t have to figure out what is hot, what is not, and move it around.

Yes, I know you are promising F.A.S.T. We are delivering administration-free performance acceleration NOW! And we do it at block level. Let me say that again: Now.

By the way, PAM works with our FlexShare quality-of-service software. So, data from LUNs for critical applications can be given preference for retention when a PAM cache gets full and needs to start evicting data.

Regarding write caching, NetApp has been using NVRAM as a write cache for long time. We don’t need PAM to do that.

Since we’re talking about writes, let me point out that PAM can be configured to cache writes when they are committed to disk. So, you can get a fast response on the first read. This is one of many “smart algorithms” that Data ONTAP uses with PAM.

Bottom line is that PAM cards are a great alternative to SSDs as a way to improve response time and, for a disk-bound system, increase I/O throughput. Borrowing your words, Chuck, PAM delivers “real random read performance”!

Best of all, you don’t have to tier your storage. You don’t have to move hot data into or out of our large read caches. It’s all automatic.

- Mark

[Chris Lionetti]

Hey Chuck, missed a few points.
1). Servers know what needs to be buffered and what doesn't. The OS doesn't have the luxury of holding back its flush no matter how much memory you install. As for reads, most App servers will not cache intelligently as you state see. http://blogs.msdn.com/ntdebugging/archive/2007/11/27/too-much-cache.aspx
2). As for the brute force method, That is exactly what he IS saying. By De-duping the OS bits, you can do just that. That was kind of the whole point of the article.

4). Your experiences in EMC Symm land may not relate. The steps you take to optimize a Symm are very different from those to optimize a NetApp. In fact that steps to optimize a Clariion are significantly different than a Symm. There are optimal ways to get the best out of any array, they are not common methods.
5). Wait, are you saying that a set Stec Drives is the way to go? if I was enhancing a Symm, that is exactly what I would do, of course that mostly has to do with the fact that it is easier to insert a STEC drive then it is to invent a new type of FA/DA board for a Symm. Our controllers however have PCI-e slots, so we are already closer to integrating a controller upgrade. Got to go for the best bang for the buck.

Must respect the differences in array architectures.

[Mike Riley]

@Chuck

As far as real world use of read cache goes, you really need to pick up a Symm manual every once in a while. When installing a limited (limited by EMC) number of SSDs, you disable that large chunk of Symm cache for the SSD drives. You can overrun Symm cache with SSDs so, your points on how cache works and using SSD as a validation point is nonsense. However, I do agree with your point that the Symm really does use a cache approach that is over a decade old.

EMC (per your best practice manuals) recommends SSD for the same environments where we would recommend PAM II: highly random read I/O. If you're bashing the PAM II applicability, you're bashing the documented EMC approach as well. We're both targeting the same problem area. If you're harangue on the multi-levels of caching were valid, why does EMC ship SSDs again? Why target the highly random I/O environment? Apparently this could all be solved by a large Symm cache and some exta dimms in the server. The only efficiency I see here is the fact that you didn't wait until a follow-on post to contradict yourself. You did it all in the same post to save space, I guess.

EMC has also limited software functionality for SSDs. For example, you do not support virtual pools (thin provisioning) on SSDs. Second, you limit the useful space of the Symm cache. Yes, you can have 1TB of cache but that's mirrored so you're left with 512GB. Of that 512GB you then have to figure out how much you want to allocate as read and write cache which further reduces your effective read and write cache. With NetApp, customers can leverage 100% of the cache for reads (and writes - BTW, NVRAM is NOT a write cache) and PAM II can be used across all protocols with all of our software - dedupe, cloning, thin provisioning, etc.

I think a key difference here is NetApp leverages flash, while EMC imposes limitations on it's use. For the Symm, SSDs were simply a disk replacement strategy. That wasn't innovation. That was merely integration (with a list of caveats). What Larry is pointing out is an innovation coupled with hardware integration. The result is far more powerful than tossing some extra hardware at a problem and harumphing about how long we've has been tossing hardware at problems. Innovation may require some hardware integration but hardware integration alone is not innovation.

[kostadis roussos]

@kostadis @chuck

It's not that NetApp storage systems don't use write-caching, of course they do. What I meant to say was that our need for write caching is not as great as symm or a clariion or other traditional legacy arrays.

To improve performance you need to expand the read cache not the write cache.

kostadis