Extensible NetApp: Thinking outside of the box

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

The DAS Disruption part 12: How available does storage have to be?

For a complete recap read: The DAS Disruption the story so far.

The end of big iron storage, describes how EMC was able to navigate the transition from monolithic storage to modular storage systems, and how that transition fundamentally changed what storage systems were.

The old Tier 1 storage array was expected to provide continuous uptime and predictable performance even in the face of failures. In effect, the Tier 1 storage array was the perfect disk drive with a whole bunch of layered features like SRDF.

The new modular storage exposed to the OS and application that the storage system consisted of a set of components that could fail and when they did the performance was unpredictable.

If a little unpredictability could save you big dollars…

Once the cat was let out of the bag, consumers of storage had to ask if a little bit of less reliability could save some dollars, what about a whole lot of unreliability?

And before you know it, very smart technology vendors were looking at using less reliable storage infrastructures as a way to cut costs.

Google started the trend with their GFS infrastructure. GFS demonstrated that a highly available service did not require highly available storage.

But Google’s economics were assumed to be the corner case, not the general case.

They were able to store multiple redundant copies in a globally distributed infrastructure.

Exchange makes the end of available storage the general case

When Microsoft’s Exchange team argues that their customers don’t need highly available storage, if you’re in the highly available storage business you should pay attention.

Exchange’s availability was increasingly tied to the underlying storage infrastructure.This is not surprising given the application architecture of Exchange 2003. The problem, if you’re Microsoft, is that there were an increasingly large number of vendors who were offering alternative email services that did not require the same kind of expensive storage infrastructure. 

Essentially what Microsoft observed is that if they could move the availability out of the storage into the application software layer, they could make do with significantly cheaper storage hardware and therefore compete more effectively.

The potential fly in the ointment was that high end storage also provides better performance.

But the Exchange folks were studying the same technology trends we’ve been studying, and observed that they could make do with cheaper slower storage if they could use memory more efficiently.

And so that’s what they did. They moved to Windows 64, re-did their internal algorithms, delivered a good-enough replication solution in the application layer, and bid a fond fair well to high end storage.

I cover this entire trend in The Assault on RAID. Which I recommend as a good read.

 

Technorati Tags: netapp,emc,wafl,storage,google,microsoft,exchange,das,tier 1

The DAS Disruption part 11: The end of big iron storage

For a summary of the story so far, I suggest you start here.

Regular readers of this blog have to endure the periodic “EMC sucks, NetApp rocks” post. Today, I want to applaud EMC for what was a remarkable transformation from selling exclusively big iron storage to selling a mix of big iron and modular storage. That transformation, however, sowed the seeds of the destruction of highly available tightly coupled storage infrastructures.

In the 1990’s, EMC’s Symmetrix was the benchmark for available storage. EMC, the company, provided exceptional service, and the hardware did the same.

Like the server big iron to which it was connected, the Symm allowed application and OS developers to assume that every disk block had the same degree of worst case latency, even when hardware started to fail.

The Symm was able to do that because it was engineered to hold a certain amount of capacity and the infrastructure necessary for the performance and availability was baked into the hardware and cost structure.

In 2002, however, well into the disruption of server big iron with x86 servers,  EMC hit a brick wall. Customers weren’t buying monolithic highly available servers, and they didn’t understand why their cheap servers needed to be connected to big storage iron.

One of the beneficiaries of that disruption was NetApp. We convinced many customers to replace their x86 windows file servers or Solaris file servers connected to Symm’s with our FAS systems. 

Faced with what was essentially a customer revolt, EMC responded with the CLARiiON. The CLARiiON unlike the Symm has a modular storage system design that has less availability and fewer performance guarantees than a Symm. But it is cheaper.

It’s cheaper because there is a tradeoff between uniform resource latency and availability and cost and modularity.

The transition, watched from the outside, is nothing short of breathtaking. Within a few short years, the big iron company became the modular company while still maintaining it’s lead in big iron.

The seeds of destruction

The problem with that transition from big iron to modular storage is that it exposed to the world the notion that storage wasn’t a uniform resource.

With the Symm, application availability was tied to the Symm. And since the Symm provided uniform performance, no one cared what was going underneath the Symm. With modular storage, we introduced all sorts of preferred, and optimized and unoptimized notions of data access, making it obvious that storage wasn’t quite so monolithic ..

Well once you tell people that it’s no longer cost effective for storage to solve the whole problem yourself, folks have to ask: how much of the problem do they really need storage to solve?

Put differently, modular storage systems provide high availability but not uniform access to the underlying storage.  The question then becomes, how non-uniform can the access be? In other words, how coupled does the storage have to be? And how highly available does the storage have to be for the services layered on top of it to be available?

Technorati Tags: storage,emc,netapp

Minor formatting error.

The DAS Disruption: The story so far

Over the last two months (starting in June 2009) I’ve been exploring the whole DAS disruption. One of my more loyal readers remarked that the series had become unwieldy, and hard to follow if you missed the beginning. On top of all that, I tended to jump around and worse, stop in mid sentence to talk about other things like Unified Storage.

And I agreed with assessment. .

So Lost, my favorite television series, has episodes that are recaps of the story so far such that the new viewers and old viewers can have their memories refreshed.

Setting the stage

The whole discussion about DAS as a disruption in the storage infrastructure began with a meditation on WAFL performance. That series  culminated in this post explaining WAFL performance, The F Word. As part of the meditation I discovered the interesting correlation between RAID-6 implementations and the assault on RAID, or what I realized was a disruptive trend in the data center. 

The basic outline of the DAS disruption is discussed in FAS the new DAS where I make the point that the DAS disruption is real, and that storage vendors who ignore it are dead.

In the late 90’s and early part of this decade, Data Center Architects, reduced costs in their data center by moving away from expensive reliable server architectures to cheap and unreliable server architectures. To address application availability, the Data Center Architects moved to applications that achieved high availability by leveraging clustering for CPU and Memory availability and highly available storage for data availability.

The new DAS disruption is about providing data availability by moving availability out of the storage infrastructure. The new DAS disruption is made possible because traditional legacy arrays are wasting vast amounts of capacity because they were built to optimize the performance of servers that were disrupted last decade. The waste is so significant that the cost advantage of DAS outweighs the value of improved utilization through centralization.

Key themes: Trends, Margin and Performance

There are three themes in this series, technology trends matter,  if you invest in the wrong areas you end up dying and if you aren’t faster than the alternative you end up dying.

Brian Pawlawski, our CTO, in Innovation, does a really good job explaining what technology trends are and how they fit into how NetApp thinks about innovation.

Points of Gross Margin explains how gross margin affects investment and the cataclysm that occurs when gross margin goes south.

The 90’s assault on DAS makes the case that shared storage came into existence because it was a more cost effective way of getting performance than the alternative.

Understanding servers, applications and the DAS disruption

The next three posts, Server Collapse, The Role of Storage in the Server Collapse, and The Server Collapse in Pictures, explore how server vendors who built big iron found themselves outflanked by low-end x86 servers and by application vendors who figured out how to get RASM without requiring big iron.

As I say in The Server Collapse in Pictures:

As each individual server provided an increasingly smaller share of the overall performance and availability applications evolved to solve those in software.

As an example, consider Exchange and availability.

In 2000 Exchange leveraged clustering for CPU and Memory availability, but relied on highly available storage with data protection. By 2009, with Exchange 2007, Microsoft was arguing that hardware availability was entirely passe and that all of the availability was to be provided by the application software.

 

A little bit later, I observed in How application availability became service availability that focusing on application availability was the wrong story, I was really talking about service availability

Trends: CPU, Memory, and Disks

A key principle at NetApp in our exploration of technology is the notion of technology trends. And the most important technology trends are hardware trends, because hardware begets software which begets new markets.

So in that spirit, I next explored, in AMD64 and Mother boards beget larger memories which beget fewer reads, how the x86 64 bit processor enabled emergence of mother boards, that could handle large memories and how that results in fewer read operations to storage.

Which lead to a set of posts on how big disks and RAID 5 were on a collision path. In. Big Disks and RAID, I made the case that RAID 5 just didn’t work and that the alternative conventional storage arrays with RAID-6 had a serious performance problem. In, How Big Disks Killed RAID 5 and RAID 10, I brought some quantitative analysis to bear to explain why RAID and RAID-10 just don’t work if you care about data availability.

The DAS Disruption part 10: How application availability became service availability

In my series so far, I’ve talked about application availability. And I wish I had not. Because it’s not about application availability but service availability.

So what is a service?

In IT a service is something likes messaging, or file sharing, or collaboration. A messaging service is the set of software components that you need to make it possible for folks to do things like send email and receive email. For MS Exchange this includes the Exchange server, but it also includes the outlook client, the set of authentication services, the backup infrastructure etc.

In the past, let’s say the 1980’s, a service mapped to a single instance of software with a single instance of hardware. Availability of the service was tied almost entirely to the availability of the hardware.

It was this tie-in between service availability and hardware availability that made the RASM features of high-end UNIX servers so valuable.

How application availability stopped being synonymous with service availability

Before clustering, applications assumed that hardware never failed. This dramatically simplified application development. If you assume the hardware never breaks, and provides uniform resource access, life is very good.

The best analogy I can give, is that we assume that water will flow from the tap reliably. If water did not, then we would make plans to keep water around just in case. We would have buckets, and bottles in the fridge, we would have a schedule to monitor water levels in our deposit, we would think about how much water we could drink every time we drank water, etc.

The fact is life is easier if we have clean water flowing from our taps.

Similarly life is easier if hardware never fails

But the emergence of cheaper hardware made it worthwhile to explore alternative architectures for building applications. And in spite of the pain. application developers tried to see if they could survive with less reliable hardware.

What they realized, in the end, was that customers didn’t care about the application up-time but the service up-time. So if you could deliver service up-time without application up-time then you could have less reliable components delivering a high quality service.

It was the decoupling of service uptime from application uptime that was crucial in the DAS disruption.

The DAS Disruption part 9: Mother boards beget larger memories which beget fewer reads

The continuing story of the DAS disruption takes us from the processor to the mother board.

Before the emergence of a 64 bit standard, a 64 bit system cost more money because of the extra infrastructure required. Up until the rest of the ecosystem surrounding a processor had 64 bit capabilities, we lived in a small memory world.

When both Intel and AMD adopted an x86 64 bit standard, the investments in the commodity space for  chip sets and mother board that could support large memories happened.

Motherboards beget larger memories

When the system infrastructure necessary to support 64 bit memories became a commodity, server systems started to have a lot more memory than they used to.

But what about the applications?

It turns out that making a 32 bit program directly leverage 64 bit memory is hard. Variables that assumed 32 bits have to be changed.

And what if I could leverage bigger memories transparently?

It’s very easy to transparently leverage larger memories, especially as a bigger file system buffer cache, if the OS does it.

So superficially it would seem you had to wait for the applications to be modified, but practically you could still get an application performance boost by from having more read cache in the server.

And because of the underlying economics, it became increasingly cost effective to put more memory on a server.

So to recap

AMD and Intel’s historic agreement to make x86 64 bit aware, made it possible for the hardware ecosystem necessary to make 64 bit systems cost-effective to emerge, which made it possible to make servers that could have very large memories at very attractive cost points.

Although it would take time for applications to adapt to directly use 64 bit memories, operating systems were able to leverage those 64 bit memories for bigger read caches.

Because applications were serving more of the read operations from the local server memory, the storage systems were seeing fewer and fewer reads.

DAS Disruption part 8: How Big Disks killed RAID-5 and RAID-10

When I first identified this DAS disruption I was wondering if I was all alone. Interestingly, one of our top guys in the Microsoft space and an MVP Exchange architect, John Fulbright, identified the same trend starting in a different place. Whereas I started from first principles, he started from looking at competitive pricing.

What we both figured out was that the DAS Disruption was triggered by the failure of the dominant storage vendors to build an effective and efficient RAID-6 implementation.

In this series on the DAS Disruption, and earlier, I’ve remarked how RAID-5 and RAID-10 are dead because of Big Disks.

The Microsoft team has today written an excellent post that puts numbers to the difference in the risk profile between RAID-5, RAID-10 and RAID-6 or RAID-DP.

They produced this nice graphical visual:

 

that shows exactly what I’ve been saying about RAID-6 for some time.

You should check out their whole blog and the analysis.

The DAS Disruption part 7: Big Disks and RAID

In part 5, 90's assualt on DAS, I focused on the role memory played in enabling storage on the network.

What I did not discuss was the role of RAID.

As I described two weeks ago, RAID made it possible to get disk drive resiliency without requiring 2x the number of components.

So let's assume you have to protect yourself against a single disk drive failure. If you use mirroring, then you have 2x the disk drives. If you use RAID 5 you have 20% more disk drives. 

RAID also made it effective to combine a bunch of disk drives to improve resiliency, which had the nice side effect of improving performance and capacity of your storage infrastructure.

Adding memory to the storage system made storage a very, very attractive device.

But hiding in the garden was a snake, and the snake was increasing disk capacities.

Now it turns out that there are multiple ways in which disks can fail. The problem with RAID is that after a disk drive has failed, you need to rebuild the disk group. And the rebuild process exposes you to every single disk block on the system. And that's when a second disk failure can be catastrophic.

Failures are a probability game. The more likely a failure, the more you have to protect against it. The less likely, the less important protecting yourself from it becomes.

The problem with the kinds of failures that RAID protects you from, is that the probability is related to the size of the disk. Increasing disk capacities make certain kinds of failures increasingly likely which means you have to start protecting yourself.

Where trends converge.

In the world of clustered servers, every application has a set of servers for resiliency. Which means that if I have server A and it's cluster pair server B, then the question is whether it's cheaper and better to have disk drives in A and B or whether I'm better off putting the disk drives on the network?

In the 1990's and early part of this century, the answer was obvious: the network. RAID 5 performance was good enough, and most workloads on storage arrays were reads, for which the big memory caches were ideal.

But the big disks ...

The problem is that as the disks got bigger, the probability of failures during a reconstruct increased. Which meant that that rare event, started becoming increasingly common. Which meant you had to resort to a different kind of scheme to protect the data from a disk failure. NetApp invented RAID-DP, EMC promotes RAID1-0 or RAID-5 with active sparing or RAID-6. 

And superficially, this is just a small change, after all it's just RAID... except that if you're EMC, RAID-6 performance on writes is poor, and RAID1-0 imposes a 2x cost penalty and latency penalty on writes. Which means that the write performance of your storage controller goes down.

So as long as the IOP mix was 80% read, 20% write, a 10% percent degradation in write performance shouldn't matter... Applying Amdhal's law that meant a 2% overall drop in performance, big but not too serious

And after all those 32 bit processors were a firewall that precluded the emergence of big server memories that could change that mix.

The DAS Disruption part 6: AMD64

The beginning of the DAS disruption doesn't begin in storage but begins with the introduction of the AMD64 instruction set through the AMD K8 family of processor.

It's funny to think about it now, but in the late 1990's the question of how mainstream x86 servers would deliver 64bit memory architectures was unclear.

For the UNIX servers built on top of 64 bit architectures, this represented a well defined limit of the x86 system architecture, acting as a firewall between the high and low-end servers.

Intel's strategy, which was in retrospect wrong, was to introduce a new processor architecture the Itanium. The new architecture had a new instruction set built on a new instruction set paradigm (EPIC). The problem was that for applications and OS's to leverage the new 64 bit architecture everything would have to change.

The problem was that the commodity space wasn't going to move to the new system architecture until the market moved, which meant that 64 bit computing was out of the cost/performance sweet spot.

If you go back to my earlier posts, and remember this picture:

the lack of a standard 64 bit processor  meant that theindividual boxes could not use 64 bit processor architectures. And more importantly the individual servers could not have memory system designs that assumed very large memories. Which meant that memory on a per server basis was low.

If you were in the storage business this created a firewall that protected your business.

So while the market tried to figure itself out, with demand for increasing memories pushing more and more Frankenstein solutions or requiring major rewrites of software, AMD, pretty much out of nowhere does exactly what everyone wished Intel had done: extend the 32 x86 architecture to support 64 bit addressing.

When Intel saw AMD's traction, Intel released a similar processor, and the death warrant of boutique processors whose only differentiation was 64 bit addressing was signed.

Very quickly it became possible to put 64 bit processors in those small server boxes, and the memory system design necessary to support them effectively also became cost effective.

Which meant that now you could have servers with lots and lots of memory at low cost.