Extensible NetApp: Thinking outside of the box

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

How Flash killed the TLA

I come not to praise the traditional legacy storage array (TLA), but to bury it. The architectural model that served it so well for so long, has reached the end of it’s useful lifespan. Killed ironically by the technology that it thought would prolong it, Flash.

Sadly for many customers, the end of the TLA will be more like  a long slow death spiral, rather than an abrupt demise. Sad because those customers will spend money and time trying to make the wrong architecture work.

Why?

The traditional storage array is optimized for read operations. The array minimizes the amount of effort it has to do on a read by putting the disk block exactly where the client thinks it is. This means on a read, the array has to do a trivial lookup to get the disk block.

However, the downside is that the array makes the write operations more expensive.

When you factor in RAID and the impact of RAID on write performance, the TLA offers a poor value proposition.

But the poor write performance was okay as long as the read operations dominated.

Enter bigger memories

The problem with the TLA architecture was that it was designed in a pre-64 bit era, and in a pre-flash era. In that era, the servers connected to the storage array were memory starved. Because they were memory starved, the buffer cache on the server was quite small, resulting in more IO operations for reads.

In the new 64 bit era, and especially with the availability of Flash, the servers are no longer memory starved. You can now have more volatile memory than you ever had before. What that means is that the average server is rarely doing a read operation, but instead is mostly doing write operations.

The impact on the TLA is that the workload shifts from being dominated by read operations to a bigger mix of write operations.

The problem, then, is that the poorly optimized path is now a bigger piece of the overall workload mix.

Or more to the point, that poor write behavior  around RAID-6, suddenly becomes a very big issue.

Which is why, after all, the TLA vendor is recommending RAID-10.

Their poor write performance is forcing them to throw more hardware at the problem.

And their poor write performance is forcing application administrators to look at alternative storage architectures.

Enter write optimized storage arrays

Write optimized storage arrays, like ONTAP, are designed for this new world order. The downside, of course, is that the read operations are more expensive, but if the mix shifts between read and write, then that’s a reasonable tradeoff.

And as I’ve said before you can solve the read performance problem …

And why do I pick this date?

Because it has been almost a year and a half since the time the Better Than Real Fibre Channel guys demonstrated that their architecture outperformed the TLA.

An architecture optimized for read was beaten by an  architecture optimized for writes, and the response continues to be silence.

So goodbye TLA, you are so last decade…

15K RPM Drives, Not dead yet?

EMC, to their credit, were the first storage vendor to put an SSD into their array.

One point of view, was that the emergence of SSD was the end of the high performance disk drive. After all SSD’s offer vastly improved performance over 15K RPM drives, and on IOPS/$ basis are cheaper along every meaningful dimension.

Except, I have this nagging doubt.

SSDs are a great technology. They are fast, they are cheap and they look like disk drives. The problem with an SSD is that data at rest is vastly more expensive if it’s on an SSD and not on spinning rust.

So?

As I have said earlier, the reason disk drives have been so successful in the market place is that application IOPS data density maps very well to IOPS density of disk drives.

Most of the data an application has is cold data. So paying for high performance for cold data is a very expensive proposition. If the array can provide high performance to hot data even if the data is stored on slow storage, then the value of Flash as a medium for storing the data is lessened.

In other words, yes 15K RPM drives don’t provide uniformly excellent performance, but they may provide good enough performance for enough of the data to make SSD’s a niche.

Now, I’m not arguing that SSD’s are not the future, nor am I arguing that 15K RPM drives are the past, I am just observing that the future is murky.

Flash Point: Missing the point on TLAs

Martin G, as he does so well, cut through the recent Flash brouhaha by pointing out that stuffing an SSD with a FiberChannel connector into a storage array is more a statement about the sad state of storage innovation than any miracle of modern engineering.

So what about NetApp?

NetApp's plans were announced in a whitepaper that described our strategy for integrating flash into our storage systems.

The PAM announcement, even though it's not Flash, is a proof point, another execution point on our roadmap to our destination. .

NetApp's recent announcement on TMS RAMSAN, doesn't really count as the great leap forward either. In my mind it's just more proof/evidence of that the roadmap.

But in spite of what  I just said, I think the TMS RAMSAN announcement is interesting in and of itself.

Why?

Because of the value of our software.

Our software lets you use less storage, make faster copies and keep more backup images than any other storage software out there. And flash just makes it all go faster. And by making it go faster, you're allowing customers who never thought they could benefit from our storage software benefit from it. And all of a sudden they are keeping more backups, using less storage and making faster copies than they were before. And all of a sudden that creates a new set of opportunities for them.

Because flash isn't a destination. Flash is an enabling technology.

The question is, enables what?

The short answer is it enables much faster reads at a fraction of the cost of DRAM.

But that isn't that interesting either.

What's really interesting is how does software adapt itself to Flash.

And no one knows the answers to those questions.

But here's what I do know, everytime you make something faster and cheaper and more plentiful, innovative people figure out a way to exploit that plenty to do new things.

So let me give you an example of what more faster reads enables.

Add more memory to our systems, and all of a sudden you don't need a FiberChannel disk tier in some infrastructures. And that suddenly frees up more money because you're buying cheaper disk, and it suddenly frees up time because you're no longer doing tiering. And that suddenly is very very interesting.

You make reads more plentiful, and guess what things change.

So focusing on the speed of Flash or how it's packaged is, in my mind, about as boring a discussion as you can imagine.

But hey, leave it to a storage customer to point out when storage vendors are too busy examining their plumage instead of focusing on the real value to the customer.

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

MAD blog: Chuck demands customers stop buying frankenstorage

I never thought I would see the day where Chuck Hollis would admit that Frankenstorage was a term (a term I believe coined by NetApp to describe the NX4 and EMC VTL).

I never though that I would see Chuck tell customers to not buy Frankenstorage. Good for you Chuck, I thought. Finally calling a shovel a shovel instead of a Lexus.

I thought can it be? Can EMC's VP of Marketing (or Spin?) finally be telling customers something truthful about unified storage? Like how using four different technologies for provisioning and data protection does not unified storage make?

Nope.

It's the damnable v-series. The damnable v-series. That's the piece of technology he hates. A piece of technology he went out of his way to distort facts about. And it's not because of our support of the TMS device, but because you stick a v-series in front of a Symm or a CLARiON and you suddenly need less disk. And that saves money. Not funny money but real dollars and anything he can say or due to blunt that point he will say and do. And if all you're using the Symm for is disk drives, you need less of their software. And that has got to hurt.

Of course the v-series and RAMSAN combo is not a unified storage device. But EMC has yet to ship any device that meets a reasonable definition of such a device. And for Chuck to lecture about any term, after all he coined Real FiberChannel(tm), is darn funny. So unlike the the original Frankenstorage, other than hardware management, all of your day-to-day operations are done consistently in a uniform way. You know using the ONTAP tools. But you've read that rant of mine before.

So I suddenly realized what the issue was: Professional Jealousy. Dr. Frankenstein is pissed off that someone else knows how to put together a device made up of multiple hardware devices and add value to customers.

And not only can we do that, but we can make our systems simpler to manage and adminsister than what the good doctor produced.

Hmm...

MAD Blog: Flash and the 'storage randomizers'

updated to make sure that people realize this is a MAD blog

Chuck Hollis remarked the other day that EMC's approach to flash was, at an architectural level, superior to NetApp's. He dismissively described everyone else's approach as storage randomization and that storage randomization was inherently wrong-headed.

I was intrigued by that comment because the rationale he gave did not mesh with my understanding of the physical properties of flash.

First let me attempt to explain the difference Chuck draws between EMC and NetApp.

NetApp's approach to storage can be described with this diagram:

Essentially we take a collection of similar disks, FC or SATA, combine them into a bigger container, an aggregate and then provision volumes from which applications consume storage.

As data gets written we allocate physical blocks on demand.

EMC's approach can be described with this diagram

EMC's logical storage unit has blocks that are allocated up front to the physical disk block. In effect, there is no layer of indirection between the disk block on disk and the logical block that is presented to the application.

The argument Chuck made was that if you consider Flash storage, the performance of Flash would make any latency introduced by the virtualization layer of the array unacceptable.

The problem with that argument is that what you can do with disk, namely permanently associate a logical block of storage to a physical block of storage is not how flash must be used.

Now it turns out that an annoying property of Flash is that a particular disk block can only be written to a fixed number of times.

In an EMC context, if you directly mapped block 500 on the flash disk, to a storage unit at some point in time the writes to the disk block would fail because of the inherent physical properties of flash.

Flash disk drive vendors know about this so here's what they do:

What they do is they add their own layer of virtualization above the raw flash so that the logical block they present to the array can be remapped under the covers of the array. Given how traditional array's work, they have to do this because the direct mapping technology would destroy flash drives at too fast a rate.

In some corners this technology is called wear-leveling.

So if we go back to the EMC architecture Chuck was talking about we have the following picture:

The logical storage container that the storage admin has assigned to a physical flash disk block is actually a logical disk block sitting on top of the disk drive vendors virtualization layer.

1. If there is no layer of virtualization between the raw  flash and the logical container then you will wear out the flash too fast.
2. If you don't have your own, you have to rely on the FC SSD (or enterprise flash drive) vendors to provide it.
3. If the Flash Vendors are providing the virtualization layer what is your value? Are you just a bump in the wire between the disk and the client?
4. There is no direct mapping like there is in disk, so it's unclear what the value of providing that direct mapping is.

If we look at NetApp's approach our goal is to integrate the flash into our virtualization layer directly by exploiting the fact that WAFL can do much of the same function that flash virtualization does. You can check out Val's post for more detail on that.

So I don't know whose architecture is necessarily superior, but I am not sure I buy into the argument that NetApp's architectural approach is by definition inferior and by definition poorly suited to Flash like Chuck claimed.

Backup, the new storage tiers and real snapshots

When I first described my new terminology for storage tiers, Tim Burlowski, asked how backup and recovery fit into that model.

It turns out that that is an interesting question.

Backup, the only working ILM/HSM solution

Step 10 feet away, and you realize that backup is HSM.

The backup process creates cold data, the backup image. The backup software them moves the backup copy to cheaper tier of storage. When a restore is performed, the backup software moves the backup copy from the cheaper tier to a more expensive tier.

The question on the table, then, is where does this form of HSM still make sense?

Captive IOPS

One obvious place where HSM as part of backup process is for the backup of data on a Captive IOPS tier. The tier is built using very expensive storage. Storing cold data that will probably never get accessed seems odd and a waste of money.

But that's not necessarily true.

If the Captive tier is built using disk drives, and the application  has a high IOPS data density, a lot of the capacity of the disks is  unused. The cheapest place to store backup copies, in that case, would be on the same disks you're using for primary data.

If the Captive tier is built using SSD's like EMC storage, then it makes a lot of sense to use a backup solution to move backup images off of the flash onto cheaper storage. If the application has a low IOPS data density, then a better approach is to use flash cache in front of disk.

If the Captive tier is built using a flash cache in front of disk, the cheapest place to store the backup copies may be on the disk drives.

More generally, the backup methodology of a captive tier made up of flash cache and disk, will be the same as the backup methodology of the capacity tier.

So what about the capacity tier and backup?

In the general case, the capacity tier will be backed up much in the same way that it always has been.

A capacity tier that has real snapshots can eliminate the HSM part of the backup process.

In fact, in an earlier series on backup I proved that for a VMware infrastructure, the cheapest place to store backup copies was on the same disks that stored the active data.

Now I want to generalize that comment for all storage..

If you consider IOPS density, disk drives must contain large amounts of cold data. A subset of the cold data in a data center is the backup data. That data can either be stored on a separate set of disks or on the same disks that are being used to serve IOPS.

To store the data on the same disks the following things need to be true

1. Creating the backup image must consume minimal IOPS
2. The existence of the backup image must impose no performance penalty
3. Be able to store a large number of images.
4. The storage must be configured using some form of RAID-6 or RAID 1-0
5. The cost of the backup images must be cost effective

It turns out that for NetApp systems all five are true.

So as disk capacities expand, storage architects who have the option to use real snapshots will stop moving data off of the capacity tier to some cheaper tier.

Said in a slightly more poetic way:

The final resting place for data on disk will be the disk where the data first got created.

The net effect will be a much simpler and more cost effective storage infrastructure to support backup.

One minor addendunm added after I wrote this post.

As Martin G says in the comments below, having the backup copies on the local disks doesn't protect you from site failures. So, of course, some form of storage replication is required. Thankfully NetApp has a cool technology called SnapMirror that replicates all of the snapshots to a remote destination in a space efficient way.

In an earlier post on VMware and backup I show how all of this fits together.

I should have included this need to replicate the data in my original blog post. So I am rectifying that error now.

Flash and the new storage classification model

So I'll be out-of-the-office  this week visiting some customers, which may mean that my blog writing may slow down.

After defining some new storage tiers, repeated here for clarity of exposition

Two IOPS tiers:

• Captive (used to be Captive IOPS)
• Shared (used to be Shared IOPS)

And a capacity tier called

• Capacity (used to be Capacity Efficient)

If you followed my post from last week, you'll remember that I made the case for disk being the media of choice for any capacity tier, because the IOPS density of disk matches the economics of this tier.

Flash, unlike disk, has an IOPS density, that is closer to 1 than to 0. That suggest that Flash belongs elsewhere ...

Captive and Shared Tiers Go Flash?

Chuck Hollis has made the case that Captive and Shared tiers will go Flash. Ironically enough, I agree with Chuck but I disagree on significant details.

EMC has positioned flash only as a stable storage. Chuck has argued that the entire dataset of Captive and Shared tiers belongs on SSDs. I think that's wrong.

NetApp has positioned flash as both a new layer in the memory hierarchy and stable storage and intends to deliver both, and I, rather unsurprisingly, think that's right.   

Why the difference?

If you read my post on IOPS data density, you'll recall that I argued that some pretty important datasets actually have a very low IOPS data density. What that means is that the dataset requires a medium that has an IOPS density that is closer to zero than to one.

But flash has an IOPS density that is closer to one than to zero...

Following that logic, you would  have to conclude that disk not flash is the right medium for those datasets..

Which suggests that Flash as Stable Storage is not the right answer for every byte of every dataset that gets deployed on Captive storage tiers!   

So where does Flash fit in?

If you read my post on The Principle of Uniform Resource Latency, you'll see that I argue that it's not the absolute performance that matters but the worst case performance. Storage arrays provide significant amount of memory to cache read and write operations. If an entire dataset fits into the memory of either the client system or the storage array, then the performance of the system is defined by the memory operations. On the other hand, if the dataset does not fit into main memory, then performance is constrained by the ability to push data to and from disk.

So now we have three kinds of datasets;

1. Low IOPS data density
2. High IOPS data density but fit into memory
3. High IOPS data density but don't fit into memory

And only one of those datasets, the third, appears to be a candidate for (Enterprise Flash Drives) EFD. But it's not that simple.

Remember the goal is to improve performance. Putting the dataset on an EFD is an approach another is to put enough memory, like a PAM module, in the storage array to avoid having to go to disk.

Which brings me to NetApp's flash strategy.

1. Provide Flash as an alternative to disk for applications that really require that kind of storage.
2. Provide Flash as a cheaper read cache for every other application

So the role of Flash?

In my opinion, I believe that flash will play two roles:

1. Be a memory cache
2. Be stable storage

And that because of the economics of (1), will probably play a role in all storage tiers, including capacity tiers. As for (2), I believe, that the role will be a lot more limited than we imagine today.

Which wins out, of course, is a matter of conjecture today, but we'll definitely know the answer in a couple of years.