Extensible NetApp: Thinking outside of the box

FAS the new DAS

When the Forrester report came out with its report on DAS, the response from the storage industry was predictable:

1. Same old, same old
2. Immature, unreliable, inefficient technology
3. Whole product has value

And when I read those responses, I could not but feel a little bit like every disruptee feels when a new disruptive technology shows up.

The funny thing with the Forrester guys is, that they are right. DAS is disruptive to the traditional storage array, in a way that the storage industry has not owned up to. And that unless the storage industry responds, DAS may win out.

The problem with the report is that they knew there was fish in one of the ponds, they just went to the wrong pond, and their report was dismissed.

So instead, I am going to offer a different point of view about the disruption of storage by DAS.

John Fulbright, has an excellent analysis where he shows how the DAS disruption is tied to RAID-5 SAN and how NetApp RAID-DP is different.

This disruption, because it is a storage disruption will take years to shake out. Storage infrastructures change at glacial rates. Once a trend, however, sticks it sticks for a long long time.

This is going to be a long series, but the short version is below.

The disruption in a nutshell

In the late 90’s and early part of this decade, Data Center Architects, reduced costs in their data center by moving away from expensive reliable server architectures to cheap and unreliable server architectures. To address application availability, the Data Center Architects moved to applications that achieved high availability by leveraging clustering for CPU and Memory availability and highly available storage for data availability.

The new DAS disruption is about providing data availability by moving availability out of the storage infrastructure. The new DAS disruption is made possible because traditional legacy arrays are wasting vast amounts of capacity because they were built to optimize the performance of servers that were disrupted last decade. The waste is so significant that the cost advantage of DAS outweighs the value of improved utilization through centralization.

What does this mean for NetApp?

Ultimately shared storage must always compete with DAS. If you can’t deliver superior value at a reasonable cost, then you don’t belong in this industry. Arguing for total customer value is a quick way to being a disrupted technology.

NetApp storage systems were optimized, interestingly enough, for the server infrastructures that are being built today. Which means, that uniquely in the storage industry, we are able to deliver performance and efficient use of capacity.

If you compare the performance, capacity and utilization of our low-end storage systems against a DAS system, it turns out that we are cost competitive and offer superior utilization, performance and availability. This is exactly what John shows in his blog.

More prosaically it means that you can offer external storage in a DAS configuration and deliver value, a surprising conclusion.

And if you can deliver value for DAS, then you can deliver the value of centralized storage if you can combine that DAS into shared storage. And that’s where our ONTAP 8 product architecture fits in.

So unlike the rest of the storage industry, my reaction to DAS is: we can do that too, and we can do that better.

Ex-chain smokers

Right after the acquisition of DDUP was announced, NetApp had it's quarterly all hands.

Dan Warmenhoven invited Frank Slootman, CEO of DDUP, to come and speak. His speech was recorded and is now available on youtube.

Frank Slootman talks about NetApp, Data Domain, the rationale for the acquisition, and culture.

And the rationale is still pretty obvious, Data Domain allows NetApp to penetrate into more accounts faster by picking a specific entry point where they have compelling technology.

But there is one funny moment when he describes ex-EMC employees as ex-chain-smokers that needed rehabilitation.

FAS Deduplication is here for the long haul

The current tussle between NetApp and EMC has, predictably, created a dust-up amongst the corporate blog-o-sphere. After all the storage industry believes in death-match-competitive-blogging, none of this polite, civilized discourse, here we reach for the knives and hack!

And it would be tempting to point out that Chuck and Storagezilla are like Aesop’s fox, but that would be too easy.

However, there is a point of confusion in the storage and analyst community about the relationship between FAS dedup and inline dedup technologies like the one provided by Data Domain. There is some opinions that say that this proposed acquisition is driven by flaws in our technology. There are some customers who may be wondering whether this reflects some flaws in our technology.

So let me state again, what I said earlier:

The rest of the storage industry has adopted inline because they lack NetApp's storage virtualization layer that enables primary deduplication.

But because that confusion is real, I wanted to remind folks about why FAS dedup is unique. And more importantly, why NetApp FAS dedup is different and must be different to meet the market requirements our storage plays in than other inline dedup solutions.

I went over this in a post several months ago in a post titled A little digression on Inline Dedup.

The basic points I made are the following:

1. Deduplication is a compute intensive operation
2. Single core performance is stagnating
3. Latency isn’t something it’s the only thing
4. Backup workloads are latency insensitive but throughput sensitive
5. Primary storage workloads are latency sensitive

And I observed that because of 1, 2 and 5 any inline process is problematic if not deeply flawed.

And then in another post I observed that the important question for dedup isn’t how efficient the dedup process is, but how well do you perform after the dedup process is complete.

Our deduplication process performs well because of it is integrated into our core WAFL data structures and because, unlike everyone else in the storage industry, we’ve solved the random IO performance problem introduced by on-disk layout fragmentation.

FAS dedup is the most widely deployed dedup solution, the only workable primary storage dedup solution, realizes tremendous cost savings, and continues to provide compelling value to our customers.

Additional Information and Where to Find It

As mentioned above, in connection with the proposed acquisition of Data Domain, on June 4, 2009, NetApp filed with the SEC a Registration Statement on Form S-4 (Commission File Number 333-159722) containing a Proxy Statement/Prospectus for Data Domain’s stockholders, and NetApp and Data Domain each plan to file with the SEC other documents regarding the proposed transaction. The definitive Proxy Statement/Prospectus will be mailed to the stockholders of Data Domain. BEFORE MAKING ANY INVESTMENT OR VOTING DECISION, INVESTORS AND SECURITY HOLDERS OF DATA DOMAIN ARE URGED TO READ THE PROXY STATEMENT/PROSPECTUS AND OTHER DOCUMENTS FILED WITH THE SEC CAREFULLY IN THEIR ENTIRETY BECAUSE SUCH DOCUMENTS CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION.

Investors and security holders will be able to obtain free copies of the Registration Statement, Proxy Statement/Prospectus and other documents filed with the SEC by NetApp and Data Domain through the web site maintained by the SEC at www.sec.gov and by contacting NetApp Investor Relations at (408) 822-7098 or Data Domain Investor Relations at (408) 980-4909. In addition, investors and security holders will be able to obtain free copies of the documents filed with the SEC on NetApp’s website at www.netapp.com and on Data Domain’s website at www.datadomain.com.

Participants in the Acquisition of Data Domain

NetApp, Data Domain and their respective directors, executive officers and certain other members of management and employees may be deemed to be participants in the solicitation of proxies in respect of the proposed transaction. Information regarding these persons is set forth in the Proxy Statement/Prospectus described above. Additional information regarding NetApp’s executive officers and directors is included in NetApp’s definitive proxy statement on Schedule 14A, as filed with the SEC on July 14, 2008, and additional information regarding Data Domain’s executive officers and directors is included in Data Domain’s Annual Report on Form 10-K and Form 10-K/A, as filed with the SEC on March 13, 2009 and April 30, 2009, respectively. You can obtain free copies of these documents from NetApp or Data Domain using the contact information above.

How Flash killed the TLA

I come not to praise the traditional legacy storage array (TLA), but to bury it. The architectural model that served it so well for so long, has reached the end of it’s useful lifespan. Killed ironically by the technology that it thought would prolong it, Flash.

Sadly for many customers, the end of the TLA will be more like  a long slow death spiral, rather than an abrupt demise. Sad because those customers will spend money and time trying to make the wrong architecture work.

Why?

The traditional storage array is optimized for read operations. The array minimizes the amount of effort it has to do on a read by putting the disk block exactly where the client thinks it is. This means on a read, the array has to do a trivial lookup to get the disk block.

However, the downside is that the array makes the write operations more expensive.

When you factor in RAID and the impact of RAID on write performance, the TLA offers a poor value proposition.

But the poor write performance was okay as long as the read operations dominated.

Enter bigger memories

The problem with the TLA architecture was that it was designed in a pre-64 bit era, and in a pre-flash era. In that era, the servers connected to the storage array were memory starved. Because they were memory starved, the buffer cache on the server was quite small, resulting in more IO operations for reads.

In the new 64 bit era, and especially with the availability of Flash, the servers are no longer memory starved. You can now have more volatile memory than you ever had before. What that means is that the average server is rarely doing a read operation, but instead is mostly doing write operations.

The impact on the TLA is that the workload shifts from being dominated by read operations to a bigger mix of write operations.

The problem, then, is that the poorly optimized path is now a bigger piece of the overall workload mix.

Or more to the point, that poor write behavior  around RAID-6, suddenly becomes a very big issue.

Which is why, after all, the TLA vendor is recommending RAID-10.

Their poor write performance is forcing them to throw more hardware at the problem.

And their poor write performance is forcing application administrators to look at alternative storage architectures.

Enter write optimized storage arrays

Write optimized storage arrays, like ONTAP, are designed for this new world order. The downside, of course, is that the read operations are more expensive, but if the mix shifts between read and write, then that’s a reasonable tradeoff.

And as I’ve said before you can solve the read performance problem …

And why do I pick this date?

Because it has been almost a year and a half since the time the Better Than Real Fibre Channel guys demonstrated that their architecture outperformed the TLA.

An architecture optimized for read was beaten by an  architecture optimized for writes, and the response continues to be silence.

So goodbye TLA, you are so last decade…

10 years at NetApp!

Between writing the requiem of the 15K RPM drive and dealing with a broken sewage pipe at home, I forgot that June 1st, was my 10 year anniversary at NetApp.

The coolest part of my time here has been the people. Listing them all here would be problematic. After all I don't want to encourage recruiters.

The most surprising part of my tenure here has been two things:

1. Everybody keeps saying we're going to go out of business because we're a niche NAS player, and we don't.
2. EMC is still our most ruthless competitor. I mean c'mon guys, just keel over and die already :-)

I've said this before, but the executive team at NetApp continues to be the single most important reason I've stayed here over the years. When I joined in 1999, I wasn't sold on NetApp for the long haul. And after the thespian troupe (acting CEO and acting CFO and acting COO) that mismanaged SGI, I wasn't sold on executives adding any value. But after a day long training session on NetApp where Tom Mendoza and Dan Warmenhoven spoke, I was sold on them, and the company. And nearly 10 years later, the team they've built and continue to build is probably the best retention policy they could have.

Some random fun facts:

1. I spent 9 years working on anything but Data ONTAP.
2. When I joined NetApp I thought SCSI was a cable not a protocol
3. There were fewer than 500 employees when I joined.
4. I remember when we thought it was Network Appliance (s) not Appliance
5. I've been through bubble 1.0, recession 1.0 and now bubble 2.0 followed by recession 2.0... and I am praying for bubble 3.0
6. When I joined 90.8 million in a quarter seemed like a lot of money.
7. Dave's book is still the best and most accurate history of NetApp
8. I managed to serve in the Greek military and debug code at the same time.
9. It took me only 9 years to figure out what Tom Mendoza meant when he said that attitude was everything.
10. I've changed jobs in NetApp 3 times going from Streaming Media, to Storage Management and now data protection, demonstrating that I lack any ability to focus.
11. I've had way too Dr. Evil moments: well let's say we have 100 MBs (crowd laughs), okay 100 GB (crowd laughs), fine folks, throw me a bone, how about 100 PB (crowd stops laughing)
12. I've been here long enough to see disk replace tape (oops not yet)
13. There was a point in time when an outrageous opinion was that Flash would replace disk. Which is especially funny, because now it's outrageous to argue that Flash won't replace disk.

After 10 years, it's tempting to want to jump off the train, but man, if I can forget that this was my tenth anniversary, who knows, maybe there's another ten in me...

15K RPM Drives, Not dead yet?

EMC, to their credit, were the first storage vendor to put an SSD into their array.

One point of view, was that the emergence of SSD was the end of the high performance disk drive. After all SSD’s offer vastly improved performance over 15K RPM drives, and on IOPS/$ basis are cheaper along every meaningful dimension.

Except, I have this nagging doubt.

SSDs are a great technology. They are fast, they are cheap and they look like disk drives. The problem with an SSD is that data at rest is vastly more expensive if it’s on an SSD and not on spinning rust.

So?

As I have said earlier, the reason disk drives have been so successful in the market place is that application IOPS data density maps very well to IOPS density of disk drives.

Most of the data an application has is cold data. So paying for high performance for cold data is a very expensive proposition. If the array can provide high performance to hot data even if the data is stored on slow storage, then the value of Flash as a medium for storing the data is lessened.

In other words, yes 15K RPM drives don’t provide uniformly excellent performance, but they may provide good enough performance for enough of the data to make SSD’s a niche.

Now, I’m not arguing that SSD’s are not the future, nor am I arguing that 15K RPM drives are the past, I am just observing that the future is murky.