Extensible NetApp: Thinking outside of the box

Snake Oil, Object Stores and Dinosaurs

On Monday I called File Virtualization Snake Oil, and Martin called me on that.

And before that I told Chuck that his object store pontifications were wrong, and he called me a dinosaur.

So let me defend myself...

Let me explain Snake Oil

My objection to File Virtualization is to the set of products in the market that don't work. Products like F5 Acopia and Rainfinity. They simply do not work. They are bumps in the wire that don't scale. They break data protection, they make backup harder, and they create all sorts of new failure modes. 

Furthermore, their main value prop is the ability to move data between tiers, and frankly there are simpler, more robust and more elegant ways to do that, even before NetApp introduced Data Motion

Object Stores

The proponents of object stores are the new wave of file system bigots. A traditional file system consists of an object store (a series of inodes with a sequence of blocks) and a namespace (a directory structure that points to a series of inodes). To find an object you do a look up into the namespace and then retrieve the data from the object store. That's how they all work, and have worked since Kernighan and Ritchie wrote their original file system.

Now I am told, that embedded name look up through directories is *flawed*. That it's in the past.

Let's be clear, extracting the namespace from the object store, and declaring the death of file systems is surprising.

The user-view of a file system is that they do a name lookup and then retrieve the object from the object store. The only question we are asking is whether the name lookup must be hierarchical and must be enforced in the storage system.

And guess what, it doesn't to both. But just because I removed the namespace look up from being embedded with the object store, doesn't mean I suddenly don't have a file system. I just have a file system that consists of multiple layered elements.

And I think having those layered elements is goodness. Because embedding the name space lookup in the storage device is too constricting. And guess what we've been doing that since 1994 when we invented this http protocol. Hmm... coincidentally NAS was invented then as well...

The 75$ VDI disruption

Vaughn Stewart wrote today about how NetApp is able to deliver VDI at a storage cost of 75$ per seat.

Chuck Hollis wrote, in reply to Vaughn, with what is the essence of the problem with being on the wrong end of a disruption:

I'm actually glad that you've decided to focus exclusively on "cost per desktop". That's what VDI is all about, right? Saving money without worrying about delivering other forms of business value?

You've decided to compete with $50 desktop drives from BestBuy.

Best of luck with that.

-- Chuck

The reality of a disruptive technology is that it provides enough value at the right price point. Not the whole value, but enough of the value. And yes, the price point for VDI is the damn 50$ disk drive at BestBuy (or Frys for those of us in California ...).

If you can't deliver at that price point you're making a TCO argument. About how over time the customer will see real savings. Sort of how Sun competed with Linux for years, and customers kept saying but I want to pay less..

EMC claims to have figured this out from the early part of this decade: that either you deliver the price and value or you might as well not show up ...

Part of the point on my whole series on the DAS disruption is that you have got to be able to hit the price point. TCO is funny money. Real dollars are what people care about.

The fact that NetApp is able to compete with the cheapest alternative and deliver compelling value is nothing short of amazing. This may, just may, represent the tipping point for VDI in the enterprise.

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

The DAS Disruption part 12: How available does storage have to be?

For a complete recap read: The DAS Disruption the story so far.

The end of big iron storage, describes how EMC was able to navigate the transition from monolithic storage to modular storage systems, and how that transition fundamentally changed what storage systems were.

The old Tier 1 storage array was expected to provide continuous uptime and predictable performance even in the face of failures. In effect, the Tier 1 storage array was the perfect disk drive with a whole bunch of layered features like SRDF.

The new modular storage exposed to the OS and application that the storage system consisted of a set of components that could fail and when they did the performance was unpredictable.

If a little unpredictability could save you big dollars…

Once the cat was let out of the bag, consumers of storage had to ask if a little bit of less reliability could save some dollars, what about a whole lot of unreliability?

And before you know it, very smart technology vendors were looking at using less reliable storage infrastructures as a way to cut costs.

Google started the trend with their GFS infrastructure. GFS demonstrated that a highly available service did not require highly available storage.

But Google’s economics were assumed to be the corner case, not the general case.

They were able to store multiple redundant copies in a globally distributed infrastructure.

Exchange makes the end of available storage the general case

When Microsoft’s Exchange team argues that their customers don’t need highly available storage, if you’re in the highly available storage business you should pay attention.

Exchange’s availability was increasingly tied to the underlying storage infrastructure.This is not surprising given the application architecture of Exchange 2003. The problem, if you’re Microsoft, is that there were an increasingly large number of vendors who were offering alternative email services that did not require the same kind of expensive storage infrastructure. 

Essentially what Microsoft observed is that if they could move the availability out of the storage into the application software layer, they could make do with significantly cheaper storage hardware and therefore compete more effectively.

The potential fly in the ointment was that high end storage also provides better performance.

But the Exchange folks were studying the same technology trends we’ve been studying, and observed that they could make do with cheaper slower storage if they could use memory more efficiently.

And so that’s what they did. They moved to Windows 64, re-did their internal algorithms, delivered a good-enough replication solution in the application layer, and bid a fond fair well to high end storage.

I cover this entire trend in The Assault on RAID. Which I recommend as a good read.

 

Technorati Tags: netapp,emc,wafl,storage,google,microsoft,exchange,das,tier 1

The DAS Disruption part 11: The end of big iron storage

For a summary of the story so far, I suggest you start here.

Regular readers of this blog have to endure the periodic “EMC sucks, NetApp rocks” post. Today, I want to applaud EMC for what was a remarkable transformation from selling exclusively big iron storage to selling a mix of big iron and modular storage. That transformation, however, sowed the seeds of the destruction of highly available tightly coupled storage infrastructures.

In the 1990’s, EMC’s Symmetrix was the benchmark for available storage. EMC, the company, provided exceptional service, and the hardware did the same.

Like the server big iron to which it was connected, the Symm allowed application and OS developers to assume that every disk block had the same degree of worst case latency, even when hardware started to fail.

The Symm was able to do that because it was engineered to hold a certain amount of capacity and the infrastructure necessary for the performance and availability was baked into the hardware and cost structure.

In 2002, however, well into the disruption of server big iron with x86 servers,  EMC hit a brick wall. Customers weren’t buying monolithic highly available servers, and they didn’t understand why their cheap servers needed to be connected to big storage iron.

One of the beneficiaries of that disruption was NetApp. We convinced many customers to replace their x86 windows file servers or Solaris file servers connected to Symm’s with our FAS systems. 

Faced with what was essentially a customer revolt, EMC responded with the CLARiiON. The CLARiiON unlike the Symm has a modular storage system design that has less availability and fewer performance guarantees than a Symm. But it is cheaper.

It’s cheaper because there is a tradeoff between uniform resource latency and availability and cost and modularity.

The transition, watched from the outside, is nothing short of breathtaking. Within a few short years, the big iron company became the modular company while still maintaining it’s lead in big iron.

The seeds of destruction

The problem with that transition from big iron to modular storage is that it exposed to the world the notion that storage wasn’t a uniform resource.

With the Symm, application availability was tied to the Symm. And since the Symm provided uniform performance, no one cared what was going underneath the Symm. With modular storage, we introduced all sorts of preferred, and optimized and unoptimized notions of data access, making it obvious that storage wasn’t quite so monolithic ..

Well once you tell people that it’s no longer cost effective for storage to solve the whole problem yourself, folks have to ask: how much of the problem do they really need storage to solve?

Put differently, modular storage systems provide high availability but not uniform access to the underlying storage.  The question then becomes, how non-uniform can the access be? In other words, how coupled does the storage have to be? And how highly available does the storage have to be for the services layered on top of it to be available?

Technorati Tags: storage,emc,netapp

Minor formatting error.

The DAS Disruption: The story so far

Over the last two months (starting in June 2009) I’ve been exploring the whole DAS disruption. One of my more loyal readers remarked that the series had become unwieldy, and hard to follow if you missed the beginning. On top of all that, I tended to jump around and worse, stop in mid sentence to talk about other things like Unified Storage.

And I agreed with assessment. .

So Lost, my favorite television series, has episodes that are recaps of the story so far such that the new viewers and old viewers can have their memories refreshed.

Setting the stage

The whole discussion about DAS as a disruption in the storage infrastructure began with a meditation on WAFL performance. That series  culminated in this post explaining WAFL performance, The F Word. As part of the meditation I discovered the interesting correlation between RAID-6 implementations and the assault on RAID, or what I realized was a disruptive trend in the data center. 

The basic outline of the DAS disruption is discussed in FAS the new DAS where I make the point that the DAS disruption is real, and that storage vendors who ignore it are dead.

In the late 90’s and early part of this decade, Data Center Architects, reduced costs in their data center by moving away from expensive reliable server architectures to cheap and unreliable server architectures. To address application availability, the Data Center Architects moved to applications that achieved high availability by leveraging clustering for CPU and Memory availability and highly available storage for data availability.

The new DAS disruption is about providing data availability by moving availability out of the storage infrastructure. The new DAS disruption is made possible because traditional legacy arrays are wasting vast amounts of capacity because they were built to optimize the performance of servers that were disrupted last decade. The waste is so significant that the cost advantage of DAS outweighs the value of improved utilization through centralization.

Key themes: Trends, Margin and Performance

There are three themes in this series, technology trends matter,  if you invest in the wrong areas you end up dying and if you aren’t faster than the alternative you end up dying.

Brian Pawlawski, our CTO, in Innovation, does a really good job explaining what technology trends are and how they fit into how NetApp thinks about innovation.

Points of Gross Margin explains how gross margin affects investment and the cataclysm that occurs when gross margin goes south.

The 90’s assault on DAS makes the case that shared storage came into existence because it was a more cost effective way of getting performance than the alternative.

Understanding servers, applications and the DAS disruption

The next three posts, Server Collapse, The Role of Storage in the Server Collapse, and The Server Collapse in Pictures, explore how server vendors who built big iron found themselves outflanked by low-end x86 servers and by application vendors who figured out how to get RASM without requiring big iron.

As I say in The Server Collapse in Pictures:

As each individual server provided an increasingly smaller share of the overall performance and availability applications evolved to solve those in software.

As an example, consider Exchange and availability.

In 2000 Exchange leveraged clustering for CPU and Memory availability, but relied on highly available storage with data protection. By 2009, with Exchange 2007, Microsoft was arguing that hardware availability was entirely passe and that all of the availability was to be provided by the application software.

 

A little bit later, I observed in How application availability became service availability that focusing on application availability was the wrong story, I was really talking about service availability

Trends: CPU, Memory, and Disks

A key principle at NetApp in our exploration of technology is the notion of technology trends. And the most important technology trends are hardware trends, because hardware begets software which begets new markets.

So in that spirit, I next explored, in AMD64 and Mother boards beget larger memories which beget fewer reads, how the x86 64 bit processor enabled emergence of mother boards, that could handle large memories and how that results in fewer read operations to storage.

Which lead to a set of posts on how big disks and RAID 5 were on a collision path. In. Big Disks and RAID, I made the case that RAID 5 just didn’t work and that the alternative conventional storage arrays with RAID-6 had a serious performance problem. In, How Big Disks Killed RAID 5 and RAID 10, I brought some quantitative analysis to bear to explain why RAID and RAID-10 just don’t work if you care about data availability.

The DAS Disruption part 10: How application availability became service availability

In my series so far, I’ve talked about application availability. And I wish I had not. Because it’s not about application availability but service availability.

So what is a service?

In IT a service is something likes messaging, or file sharing, or collaboration. A messaging service is the set of software components that you need to make it possible for folks to do things like send email and receive email. For MS Exchange this includes the Exchange server, but it also includes the outlook client, the set of authentication services, the backup infrastructure etc.

In the past, let’s say the 1980’s, a service mapped to a single instance of software with a single instance of hardware. Availability of the service was tied almost entirely to the availability of the hardware.

It was this tie-in between service availability and hardware availability that made the RASM features of high-end UNIX servers so valuable.

How application availability stopped being synonymous with service availability

Before clustering, applications assumed that hardware never failed. This dramatically simplified application development. If you assume the hardware never breaks, and provides uniform resource access, life is very good.

The best analogy I can give, is that we assume that water will flow from the tap reliably. If water did not, then we would make plans to keep water around just in case. We would have buckets, and bottles in the fridge, we would have a schedule to monitor water levels in our deposit, we would think about how much water we could drink every time we drank water, etc.

The fact is life is easier if we have clean water flowing from our taps.

Similarly life is easier if hardware never fails

But the emergence of cheaper hardware made it worthwhile to explore alternative architectures for building applications. And in spite of the pain. application developers tried to see if they could survive with less reliable hardware.

What they realized, in the end, was that customers didn’t care about the application up-time but the service up-time. So if you could deliver service up-time without application up-time then you could have less reliable components delivering a high quality service.

It was the decoupling of service uptime from application uptime that was crucial in the DAS disruption.

The DAS Disruption part 9: Mother boards beget larger memories which beget fewer reads

The continuing story of the DAS disruption takes us from the processor to the mother board.

Before the emergence of a 64 bit standard, a 64 bit system cost more money because of the extra infrastructure required. Up until the rest of the ecosystem surrounding a processor had 64 bit capabilities, we lived in a small memory world.

When both Intel and AMD adopted an x86 64 bit standard, the investments in the commodity space for  chip sets and mother board that could support large memories happened.

Motherboards beget larger memories

When the system infrastructure necessary to support 64 bit memories became a commodity, server systems started to have a lot more memory than they used to.

But what about the applications?

It turns out that making a 32 bit program directly leverage 64 bit memory is hard. Variables that assumed 32 bits have to be changed.

And what if I could leverage bigger memories transparently?

It’s very easy to transparently leverage larger memories, especially as a bigger file system buffer cache, if the OS does it.

So superficially it would seem you had to wait for the applications to be modified, but practically you could still get an application performance boost by from having more read cache in the server.

And because of the underlying economics, it became increasingly cost effective to put more memory on a server.

So to recap

AMD and Intel’s historic agreement to make x86 64 bit aware, made it possible for the hardware ecosystem necessary to make 64 bit systems cost-effective to emerge, which made it possible to make servers that could have very large memories at very attractive cost points.

Although it would take time for applications to adapt to directly use 64 bit memories, operating systems were able to leverage those 64 bit memories for bigger read caches.

Because applications were serving more of the read operations from the local server memory, the storage systems were seeing fewer and fewer reads.

DAS Disruption part 8: How Big Disks killed RAID-5 and RAID-10

When I first identified this DAS disruption I was wondering if I was all alone. Interestingly, one of our top guys in the Microsoft space and an MVP Exchange architect, John Fulbright, identified the same trend starting in a different place. Whereas I started from first principles, he started from looking at competitive pricing.

What we both figured out was that the DAS Disruption was triggered by the failure of the dominant storage vendors to build an effective and efficient RAID-6 implementation.

In this series on the DAS Disruption, and earlier, I’ve remarked how RAID-5 and RAID-10 are dead because of Big Disks.

The Microsoft team has today written an excellent post that puts numbers to the difference in the risk profile between RAID-5, RAID-10 and RAID-6 or RAID-DP.

They produced this nice graphical visual:

 

that shows exactly what I’ve been saying about RAID-6 for some time.

You should check out their whole blog and the analysis.