Extensible NetApp: Thinking outside of the box

Go NetApp!

After a decade at NetApp, I am moving on to a different adventure.  Which means that my blog and my twitter feed (kostadis_netapp) will go permanently dark…

I believe NetApp has the most compelling product, vision and technology out there. And our continued success in the market, in spite of a very focused program of unrelenting FUD by our competitors is proof of that. Customers buy high value product, and they reward vendors who build high value product, and they continue to reward NetApp.

Why leave?

My decision to leave is about my need to do different things, and I am very excited about those different things.

Over the last decade I learned that if you have a great team that is focused on adding value to customers, regardless of what the market does, you have a great chance of winning. And in the past decade I have had the chance of working with one of the greatest teams out there.

If I could borrow a phrase, Dave Hitz said in his book that he believes in Magic. And seeing what the teams at NetApp have done over the years, I too believe in Magic. I had and have an unshakeable belief in this company’s ability to create incredible things.

One last word of thanks…

Thanks to everyone who helped create a model company.

Snake Oil, Object Stores and Dinosaurs

On Monday I called File Virtualization Snake Oil, and Martin called me on that.

And before that I told Chuck that his object store pontifications were wrong, and he called me a dinosaur.

So let me defend myself...

Let me explain Snake Oil

My objection to File Virtualization is to the set of products in the market that don't work. Products like F5 Acopia and Rainfinity. They simply do not work. They are bumps in the wire that don't scale. They break data protection, they make backup harder, and they create all sorts of new failure modes. 

Furthermore, their main value prop is the ability to move data between tiers, and frankly there are simpler, more robust and more elegant ways to do that, even before NetApp introduced Data Motion

Object Stores

The proponents of object stores are the new wave of file system bigots. A traditional file system consists of an object store (a series of inodes with a sequence of blocks) and a namespace (a directory structure that points to a series of inodes). To find an object you do a look up into the namespace and then retrieve the data from the object store. That's how they all work, and have worked since Kernighan and Ritchie wrote their original file system.

Now I am told, that embedded name look up through directories is *flawed*. That it's in the past.

Let's be clear, extracting the namespace from the object store, and declaring the death of file systems is surprising.

The user-view of a file system is that they do a name lookup and then retrieve the object from the object store. The only question we are asking is whether the name lookup must be hierarchical and must be enforced in the storage system.

And guess what, it doesn't to both. But just because I removed the namespace look up from being embedded with the object store, doesn't mean I suddenly don't have a file system. I just have a file system that consists of multiple layered elements.

And I think having those layered elements is goodness. Because embedding the name space lookup in the storage device is too constricting. And guess what we've been doing that since 1994 when we invented this http protocol. Hmm... coincidentally NAS was invented then as well...

So why is this, Data Motion, thing so cool?

chuck hollis in a tweet asked:

@kostadis_netapp how is this different/better than, say storage vMotion, or external file virtualization, or other alternatives? Not clear.

And that was a legitimate question, and I was going to reply in a tweet, but 140 characters is not a great way to explain anything.

In many ways, storage vmotion and file virtualization and NetApp Data Motion, are all trying to solve the same fundamental problem of separating physical storage from a logical storage container.

When evaluating technologies the question is: what are the tradeoffs?

Storage Vmotion

Storage Vmotion, vmware's product, is great if you're only using vmware and you want to move your data one virtual machine at a time. The limitations of storage vmotion are

1. It doesn't work for hyperv, and it doesn't work for physical environments.
2. It moves the data by schlepping the data through the physical server. NetApp Data Motion moves the storage without bringing the data back into the server
3. It can't move your snapshots. If you're a NetApp customer using NetApp snapshots, storage vmotion will not move the vmdk's locked in the snapshots. 
4. Losing your data protection relationships. If you move a vm from one data store to another data store, any storage based data protection relationships get lost.
5. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

File Virtualization aka Snake Oil

File Virtualization is, in my not so humble opinion, snake oil. The premise of file virtualization is that you can track millions of files in a meta-database outside of the file system and move individual files. But let's assume it can be made to work, a lofty and unproven claim thus far.

What are the limitations of this technology

1. Scale. Moving millions of files one at time and tracking the information.
2. It can't move files in snapshots
3. Data protection. If a file gets moved between different storage tiers, then the data protection relationship gets gummed up.
4. Any dedup'ed storage get's undeduped as it gets transferred. This means you have to move more data for longer. You also have to pay the penalty of re-deduping after the fact.

NetApp Data Motion

NetApp Data Motion is, of course, perfect in every way (not!). NetApp Data Motion is great if you want to move a whole bunch of data and all of the storage based snapshots and preserve any mirroring relationships. However, there are limitations

1. Requires either NetApp storage or a NetApp virtualization appliance
2. The granularity of movement is large (a volume). So if you want to move a single file or vm it's not quite the right technology
3. Dedup'ed data stays deduped. This means faster data transfer and no dedup after the fact.

So why was I so excited?

The reason I was so excited was because NetApp Data Motion is the culmination of a journey to build a management infrastructure that divorced logical from physical storage. Not because this is the only way to skin the storage data motion cat, even though I think the advantages of this approach are significant.

Late breaking updates...

jpwarren reminded me about dedup. Do'h!

Data Motion: the next stop in storage management

Data Motion is a particularly proud moment for me.

In 2005, a small number of folks, myself included, in a conference room in a hotel in Palo Alto were looking into how NetApp could transform storage management.

Up until that point, NetApp had a small set of interesting products that provided reasonable, if not great, monitoring and modest active management. For long time NetApp hands, the phrase DataFabric Manager might be familiar.

In that room we asked: would it be possible to do something radically different?

Being arrogant, and ignorant, we thought: couldn't we dramatically simplify storage management through policy based automation? The intent was to separate logic storage from the underlying physical infrastructure. To turn storage management from a hardware problem to a software problem.

The intuition was the following, NetApp storage virtualization layer allowed dynamic reconfiguration of storage. There was nothing that said software couldn't do that dynamic reconfiguration. And if software could do that, then we could make storage truly dynamic.

The first product we shipped that embodied that vision was Protection Manager. The vision itself was articulated in a paper we published in LISA 2007, and in an article in ACM Queue.

Today we announced Data Motion. A solution that is built on top of many of the foundational technologies we dreamed up four years ago. This solution demonstrates the power of a truly virtualized storage infrastructure that allows true dynamic reconfiguration. Is it complete? Of course not, but it's real.

The vision we had of being able to dynamically adjust your storage through software with no disruption using end-to-end tools is finally a reality.

I could not be more proud of this moment.

The 75$ VDI disruption

Vaughn Stewart wrote today about how NetApp is able to deliver VDI at a storage cost of 75$ per seat.

Chuck Hollis wrote, in reply to Vaughn, with what is the essence of the problem with being on the wrong end of a disruption:

I'm actually glad that you've decided to focus exclusively on "cost per desktop". That's what VDI is all about, right? Saving money without worrying about delivering other forms of business value?

You've decided to compete with $50 desktop drives from BestBuy.

Best of luck with that.

-- Chuck

The reality of a disruptive technology is that it provides enough value at the right price point. Not the whole value, but enough of the value. And yes, the price point for VDI is the damn 50$ disk drive at BestBuy (or Frys for those of us in California ...).

If you can't deliver at that price point you're making a TCO argument. About how over time the customer will see real savings. Sort of how Sun competed with Linux for years, and customers kept saying but I want to pay less..

EMC claims to have figured this out from the early part of this decade: that either you deliver the price and value or you might as well not show up ...

Part of the point on my whole series on the DAS disruption is that you have got to be able to hit the price point. TCO is funny money. Real dollars are what people care about.

The fact that NetApp is able to compete with the cheapest alternative and deliver compelling value is nothing short of amazing. This may, just may, represent the tipping point for VDI in the enterprise.

Shed a tier with PAM II: The SSD tier

The DAS disruption series has been driven by a belief that expanding memory sizes change the fundamental behavior of storage systems. What I’ve argued is that expanding memory sizes put an increasing premium on the ability to do efficient write operations and an increasingly low premium on the ability to do read operations. And I have argued that the future belongs to the Better Than Fibre Channel Array vendors who have optimized their systems for write performance.

It’s always gratifying when your company releases technology that buttresses your point :-)

The tier to shed isn’t FC, it’s SSD!

And just to be clear, the point of this post isn’t to argue FC is dead. because that’s too easy. The point of this post is to argue that Flash as a stable storage medium is dead.

The PAM II is NetApp’s Flash memory card. Unlike SSD’s that can only accelerate data that is stored on the SSD, the PAM II allows NetApp to cost effectively extend the buffer cache for all data in a system. Because the PAM II is a buffer no expensive complex data distribution system like FAST is required.

The PAM II allows a customer to add 256 512 GB of Flash memory per card to the system buffer cache, reducing the total IOPS that the disk subsystem has to perform across the entire dataset stored in the system. The PAM II and the surrounding software architecture allows NetApp to leverage flash for hot data, while leaving disk to store cold automatically.

There are lots of ways to make the point, but this particular set of graphs makes it quite eloquently.

Using hardware that has worse latency than FC, the latency of the storage system was about the same with fewer spindles for all of the data.

What’s going on is that the although the total number of write operations has stayed the same, the total number of read operations from disk have gone down. Because the read operations are being served from high speed flash, the total latency actually delivered from the storage device is better than a system that uses FC drives.

For those  who followed our previous announcements on the PAMvI, this is no surprise.

And now for more mission critical workloads…

What should be more interesting to folks is that we’re able to achieve similar results for OLTP workloads (aka mission critical).

In a recently produced white paper Using the Performance Accelerator Module II in Online Transaction Processing, NetApp demonstrates that a PAMvII is equivalent to doubling the number of disks:

 

And at the same time improving the system response time:

I’ve said it before, and I’ll say it again, Flash is a part of the volatile memory hierarchy, not the storage hierarchy. Flash is most efficiently and effectively used as a cache not a solid state device.

Technorati Tags: emc,netapp,flash,ssd,emc fast,storage performance,tier 0,tier 1,pamvii

The DAS Disruption part 13: The end of the IOPS tier?

For a complete recap, read The DAS Disruption, the story so far.

The whole DAS disruption series has made me realize that I got it wrong with the separation of the IOPS tier from the Capacity tier. Before I explain the mistake, let me give some background.

Background

In an earlier series, I explained how the whole discussion about Tier 1 vs Tier 2 vs Tier 3 was incomprehensible, and that instead there were only two tiers: IOPS and Capacity.

I also described how the optimal storage system would use the capacity tier for storing data, and the IOPS tier for responding to server requests for data.

In a separate series, I defined two terms:  IOPS data density and IOPS density. IOPS data density was the number of IOPS over a dataset. IOPS density was the number of IOPS over the capacity of the device. What I observed was that the IOPS data density was actually pretty well aligned with the IOPS density of disk drives, suggesting that the SSD was uninteresting.

Interestingly, EMC is promoting a new technology called FAST. FAST, in my mind, is their solution to the mismatch between IOPS density and IOPS data density. Inside the array, EMC moves data between the disk and flash tier to better match cost with data access.

Along the way, I had a series about WAFL performance, where I observed that unlike most storage systems WAFL performance was tied to the read cache not the write cache.

IOPS and Capacity redux

The IOPS tier was the part of the storage infrastructure that serviced random IOPS to applications. Because any modern storage array can turn a write into a sequential operation, the IOPS tier was really a random read IOPS tier.

But with increasing memory sizes on both the storage array and the server, the value of an IOPS tier built out of disk  is shrinking fast. In other words, the more memory you add the less you need disk drives to service random read because more of the read operations are coming from memory.

So if disk drives are not going to be doing random read operations, then the IOPS tier disappears entirely into the buffer cache of the sever and the storage array.

Which means the IOPS tier as a well defined layer in the storage infrastructure disappears as well.

So perhaps, what I should have said is that the future is a large memory pool and a single storage tier which is only used for capacity.

Technorati Tags: emc,netapp,wafl,storage,flash,ssd,iops data density,iops density,EMC FAST,DAS

The DAS Disruption part 12: How available does storage have to be?

For a complete recap read: The DAS Disruption the story so far.

The end of big iron storage, describes how EMC was able to navigate the transition from monolithic storage to modular storage systems, and how that transition fundamentally changed what storage systems were.

The old Tier 1 storage array was expected to provide continuous uptime and predictable performance even in the face of failures. In effect, the Tier 1 storage array was the perfect disk drive with a whole bunch of layered features like SRDF.

The new modular storage exposed to the OS and application that the storage system consisted of a set of components that could fail and when they did the performance was unpredictable.

If a little unpredictability could save you big dollars…

Once the cat was let out of the bag, consumers of storage had to ask if a little bit of less reliability could save some dollars, what about a whole lot of unreliability?

And before you know it, very smart technology vendors were looking at using less reliable storage infrastructures as a way to cut costs.

Google started the trend with their GFS infrastructure. GFS demonstrated that a highly available service did not require highly available storage.

But Google’s economics were assumed to be the corner case, not the general case.

They were able to store multiple redundant copies in a globally distributed infrastructure.

Exchange makes the end of available storage the general case

When Microsoft’s Exchange team argues that their customers don’t need highly available storage, if you’re in the highly available storage business you should pay attention.

Exchange’s availability was increasingly tied to the underlying storage infrastructure.This is not surprising given the application architecture of Exchange 2003. The problem, if you’re Microsoft, is that there were an increasingly large number of vendors who were offering alternative email services that did not require the same kind of expensive storage infrastructure. 

Essentially what Microsoft observed is that if they could move the availability out of the storage into the application software layer, they could make do with significantly cheaper storage hardware and therefore compete more effectively.

The potential fly in the ointment was that high end storage also provides better performance.

But the Exchange folks were studying the same technology trends we’ve been studying, and observed that they could make do with cheaper slower storage if they could use memory more efficiently.

And so that’s what they did. They moved to Windows 64, re-did their internal algorithms, delivered a good-enough replication solution in the application layer, and bid a fond fair well to high end storage.

I cover this entire trend in The Assault on RAID. Which I recommend as a good read.

 

Technorati Tags: netapp,emc,wafl,storage,google,microsoft,exchange,das,tier 1

The DAS Disruption part 11: The end of big iron storage

For a summary of the story so far, I suggest you start here.

Regular readers of this blog have to endure the periodic “EMC sucks, NetApp rocks” post. Today, I want to applaud EMC for what was a remarkable transformation from selling exclusively big iron storage to selling a mix of big iron and modular storage. That transformation, however, sowed the seeds of the destruction of highly available tightly coupled storage infrastructures.

In the 1990’s, EMC’s Symmetrix was the benchmark for available storage. EMC, the company, provided exceptional service, and the hardware did the same.

Like the server big iron to which it was connected, the Symm allowed application and OS developers to assume that every disk block had the same degree of worst case latency, even when hardware started to fail.

The Symm was able to do that because it was engineered to hold a certain amount of capacity and the infrastructure necessary for the performance and availability was baked into the hardware and cost structure.

In 2002, however, well into the disruption of server big iron with x86 servers,  EMC hit a brick wall. Customers weren’t buying monolithic highly available servers, and they didn’t understand why their cheap servers needed to be connected to big storage iron.

One of the beneficiaries of that disruption was NetApp. We convinced many customers to replace their x86 windows file servers or Solaris file servers connected to Symm’s with our FAS systems. 

Faced with what was essentially a customer revolt, EMC responded with the CLARiiON. The CLARiiON unlike the Symm has a modular storage system design that has less availability and fewer performance guarantees than a Symm. But it is cheaper.

It’s cheaper because there is a tradeoff between uniform resource latency and availability and cost and modularity.

The transition, watched from the outside, is nothing short of breathtaking. Within a few short years, the big iron company became the modular company while still maintaining it’s lead in big iron.

The seeds of destruction

The problem with that transition from big iron to modular storage is that it exposed to the world the notion that storage wasn’t a uniform resource.

With the Symm, application availability was tied to the Symm. And since the Symm provided uniform performance, no one cared what was going underneath the Symm. With modular storage, we introduced all sorts of preferred, and optimized and unoptimized notions of data access, making it obvious that storage wasn’t quite so monolithic ..

Well once you tell people that it’s no longer cost effective for storage to solve the whole problem yourself, folks have to ask: how much of the problem do they really need storage to solve?

Put differently, modular storage systems provide high availability but not uniform access to the underlying storage.  The question then becomes, how non-uniform can the access be? In other words, how coupled does the storage have to be? And how highly available does the storage have to be for the services layered on top of it to be available?

Technorati Tags: storage,emc,netapp

Minor formatting error.

The DAS Disruption: The story so far

Over the last two months (starting in June 2009) I’ve been exploring the whole DAS disruption. One of my more loyal readers remarked that the series had become unwieldy, and hard to follow if you missed the beginning. On top of all that, I tended to jump around and worse, stop in mid sentence to talk about other things like Unified Storage.

And I agreed with assessment. .

So Lost, my favorite television series, has episodes that are recaps of the story so far such that the new viewers and old viewers can have their memories refreshed.

Setting the stage

The whole discussion about DAS as a disruption in the storage infrastructure began with a meditation on WAFL performance. That series  culminated in this post explaining WAFL performance, The F Word. As part of the meditation I discovered the interesting correlation between RAID-6 implementations and the assault on RAID, or what I realized was a disruptive trend in the data center. 

The basic outline of the DAS disruption is discussed in FAS the new DAS where I make the point that the DAS disruption is real, and that storage vendors who ignore it are dead.

In the late 90’s and early part of this decade, Data Center Architects, reduced costs in their data center by moving away from expensive reliable server architectures to cheap and unreliable server architectures. To address application availability, the Data Center Architects moved to applications that achieved high availability by leveraging clustering for CPU and Memory availability and highly available storage for data availability.

The new DAS disruption is about providing data availability by moving availability out of the storage infrastructure. The new DAS disruption is made possible because traditional legacy arrays are wasting vast amounts of capacity because they were built to optimize the performance of servers that were disrupted last decade. The waste is so significant that the cost advantage of DAS outweighs the value of improved utilization through centralization.

Key themes: Trends, Margin and Performance

There are three themes in this series, technology trends matter,  if you invest in the wrong areas you end up dying and if you aren’t faster than the alternative you end up dying.

Brian Pawlawski, our CTO, in Innovation, does a really good job explaining what technology trends are and how they fit into how NetApp thinks about innovation.

Points of Gross Margin explains how gross margin affects investment and the cataclysm that occurs when gross margin goes south.

The 90’s assault on DAS makes the case that shared storage came into existence because it was a more cost effective way of getting performance than the alternative.

Understanding servers, applications and the DAS disruption

The next three posts, Server Collapse, The Role of Storage in the Server Collapse, and The Server Collapse in Pictures, explore how server vendors who built big iron found themselves outflanked by low-end x86 servers and by application vendors who figured out how to get RASM without requiring big iron.

As I say in The Server Collapse in Pictures:

As each individual server provided an increasingly smaller share of the overall performance and availability applications evolved to solve those in software.

As an example, consider Exchange and availability.

In 2000 Exchange leveraged clustering for CPU and Memory availability, but relied on highly available storage with data protection. By 2009, with Exchange 2007, Microsoft was arguing that hardware availability was entirely passe and that all of the availability was to be provided by the application software.

 

A little bit later, I observed in How application availability became service availability that focusing on application availability was the wrong story, I was really talking about service availability

Trends: CPU, Memory, and Disks

A key principle at NetApp in our exploration of technology is the notion of technology trends. And the most important technology trends are hardware trends, because hardware begets software which begets new markets.

So in that spirit, I next explored, in AMD64 and Mother boards beget larger memories which beget fewer reads, how the x86 64 bit processor enabled emergence of mother boards, that could handle large memories and how that results in fewer read operations to storage.

Which lead to a set of posts on how big disks and RAID 5 were on a collision path. In. Big Disks and RAID, I made the case that RAID 5 just didn’t work and that the alternative conventional storage arrays with RAID-6 had a serious performance problem. In, How Big Disks Killed RAID 5 and RAID 10, I brought some quantitative analysis to bear to explain why RAID and RAID-10 just don’t work if you care about data availability.