NetApp - On The Edge

Supporting Open Source Is A Privilege

I was particularly happy to see NetApp well represented in a recent paper describing contributions to the Linux kernel. NetApp is in the top twenty companies contributing code to the Linux kernel - much of it through our support of the Linux NFS Client. I got involved personally in getting NetApp more involved in Linux kernel development as our customers in 1999 started to increasingly use Linux in their data center operations. What I love about the Linux community and the people that comprise it is their openness to contributions - both bug fixes and significant feature enhancements.

NetApp has a long history of using open source, and I firmly believe that the use of open source should be followed by investing in the communities developing it. Indirectly, our membership in The Linux Foundation is part of that support. In the past, we've directly funded separate programs for NFS testing and SAN attach scalability in Linux through The Linux Foundation.

We use parts of FreeBSD in our products, and we're trying to support the work of that community also. Again, direct code contributions from our developers is our greatest contribution in my mind. But donations to organizations is also important - with a lot of these contributions coming from my CTO Office - such as our support for the FreeBSD Foundation (gotta check into status of our 2008 donation I notice!). Being a storage company we've found that sometimes providing highly reliable storage for development activities is the best way to help out some projects.

It is a privilege to be able to give back to these communities - important not only to NetApp but more importantly to our customers.

And personally, I enjoy the time I spend with the open source developers. Always engaging. And while I'm definitely not trying to start an open source war, my personal experience is that the Linux community does really know how to celebrate. I'm looking forward to the next Linux Kernel Summit! (And where are we with the sponsorship on that this year? Another thing to check up on:-)

NetApp raises the bar in SAN performance reporting

On Tuesday, January 29, 2008, NetApp published a couple of SPC-1 benchmark results  for the mid-range FAS3040 storage system.  With our highly efficient Snapshot(TM) feature on the FAS3040 with RAID-6 (RAID-DP) enabled, and our thin provisioning FlexVol architecture, the results clearly show that enabling data protection comes at a very modest cost in performance. The  features enabled on the NetApp storage are recommended in our best practice deployments and represent a realistic customer configuration. By using the independently audited, industry standard benchmark SPC-1 to report performance effects of these features, we hope to raise the level of conversation around storage system performance measurement and reporting.

Characterizing the performance impact of NetApp's Snapshot technology is the significant result - there is only a 3% drop in throughput from a base of 30,985 ops/s.

By way of comparison, as part of this study we measured a competitive system from EMC, the  CX3-40 system. In sharp contrast to the NetApp result, the EMC CX3-40 system configured in a standard mirrored RAID configuration - following EMC best  practices - suffers more than a 60% decrease in performance when their snapshots are enabled.

You can find more information about the results here.

The team at NetApp that did this work (thanks everyone!), and the product development group, are justifiably proud of these results. Such information on the costs and tradeoffs around standard storage management technologies is needed to make informed decisions about  storage deployments.

Benchmarks by their nature  tend to be controversial. But they are simply tools - some better, some worse - used to measure performance and compare two or more systems. A benchmark result is a combination of several factors. First, all benchmarks define a specific workload - a model, sometimes actually more often synthetic - to subject the test system to. Better benchmarks reflect attention to calibrating the workload with actual application workload results. The next factor is the configuration of the system under test - here is where a lot of controversy lies in benchmark results. "Tuning a configuration" to achieve good results is expected - it's when the tuning gets increasingly unrealistic - or forbid, benchmark specific - that things can get confusing. Which brings me to uniform run and reporting rules. Good benchmarks strive for useful comparisons by having run and reporting rules sufficient to allow others to reproduce the result. Finally,  peer review or auditing of the results prior to publication is required to maintain the integrity of their respective benchmarks. The SPEC SFS and SPC-1 benchmarks represent two of the more useful and rigorously defined benchmarks for storage performance measurement.

What is very cool about NetApp results like this is that we have striven to configure the system in much the same way as a customer would (following our best practice recommendations).  These results use much of our core feature set (RAID-DP, Flexible Volumes, Snapshot feature).  They are run using an industry standard workload, and audited by an independent auditor.  They are real.

The FAS 3040 result is the only SPC-1 published result using a RAID-6 implementation at this time. In fact, it's one of a small handful of published results using anything other than (the more costly) traditional RAID 1+0 mirror. NetApp's RAID-DP is an extremely efficient RAID 6 implementation that is enabled by default on our storage systems - including our StoreVault S300 product for small and medium businesses. (In the interest of full disclosure, I must say I am not only an employee of Network Appliance, but a fervent customer of one of our resellers and proud owner of an S300 - with a two year support contract:-) Further, this result is on our industry leading storage software stack that provides a Unified Storage Architecture (with the ability to support NAS, iSCSI and FC SAN access), and sophisticated thin provisioning capabilities that allow you to right size your LUNs and data containers for your application deployment. Because we have a single unified storage stack, it is possible to extrapolate these results to other deployments (say iSCSI or NAS connectivity) for a similar workload.

The other aspect of the reported result is the characterization of performance overhead when snapshots are enabled.  Snapshots, for online backups and application recovery points, are a commonly supported feature today in many storage offerings - but is usually implemented as a copy out operation  of unmodified data to a separate volume as opposed to NetApp's Snapshot in-place implementation. Snapshots were done more frequently on the NetApp FAS 3040 (every 15 minutes) vs. the EMC array (every 1 hour). Further, the NetApp  storage retained three snapshots rolling the oldest one off as a new one was created, versus one snapshot retained in the EMC measurement. Given the performance cost of the copy out snapshot mechanism in the CX-3 40 it was felt this was a more realistic frequency and depth.

Our belief is that the NetApp performance under SPC-1 load (NetApp result dropped only 3% vs. the nearly 60% performance drop in the measurement in the EMC result) reflects  that the Snapshot features in  Data ONTAP are more efficient because they  were designed into the product from the start as a fundamental feature of our solution.

So, what does all this mean to you? Realize that your mileage will vary - your workload will undoubtedly have different characteristics than that defined by SPC-1. But broadly speaking, the results provide apple-to-apple comparisons of two  competitive mid-range storage arrays and show the viability of an efficient RAID 6 solution vs. a standard RAID mirror configuration, and the dramatic performance overhead difference that you are likely to encounter with different snapshot architectures and implementations.

So what happens next?  There are a few additional aspects of these results that I will discuss in this blog in the near future. Longer term, we'll continue to improve our products, to make them more effective at solving more customers problems. And we'll publish benchmarks from time to time, allowing customers to understand the competitive context. Other critical storage features, such as remote mirroring and disaster recovery, bear characterization. We will publish when we have interesting things to say - and work with standard benchmarks and the organizations that develop them to encourage measurement methodologies around realistic customer configurations and concerns.

A conversation with Jim Zemlin

What started out as a face to face conversation with Jim Zemlin, Executive Director of the Linux Foundation, morphed into an e-mail interview as weather and San Francisco parking tickets reared their ugly heads in the New Year. I am on the board of the Linux Foundation (NetApp is a member company). Jim and I were catching up - and he helped me write my next blog entry.

Brian: Good afternoon, Jim. Great weather we're having in San Francisco this New Year.

Jim: The weather here is terrible. I suspect it is due to global warming and in the spirit of relating everything to the Linux Foundation we have a Green Linux initiative to enhance power management of Linux and make the world a better place.

Brian: So, NetApp is a member of the Linux Foundation, before that, we were a member of OSDL [one of the predecessors of the Linux Foundation]. NetApp, as you know, is the most innovative storage vendor in the industry, and our product is based on proprietary software for the most part. I know why we are a member, but I'm curious from your perspective, where do we fit in to the Linux Foundation and open source?

Jim: NetApp, like many other companies in the world, depend on Linux for a variety of things. First, it is important for NetApp's customers to understand that their products will work effectively with Linux. Because Linux is the fastest growing platform on the planet, I suspect there are a lot of NetApp customers that fall into this camp. Second, NetApp uses Linux in some of their products and by doing so benefits from the collective R&D that goes on in key open source projects such as the Linux Kernel or other components of open source software. Keeping a close eye on these projects and building up relationships with the other participants in these projects is something that is important to NetApp. At the Foundation we provide a forum within which NetApp can work with their industry partners and the community to benefit all of those working on making Linux and open source better.

Brian: Yes. The other place we're playing is in direct support and contribution to the Linux code base. We employ Trond Myklebust (the Linux NFS client code maintainer), and have a couple projects underway in NFS and other storage technologies that are generating code for inclusion in Linux. And certainly Linux is used by many of our development engineers as a development platform. What's it like to be the Executive Director of the Linux Foundation? Your day to day?

Jim: Lots of different things. First, I make sure that the people that work at the Foundation, including Linus Torvalds, get paid on time and have the best possible work environment we can provide. Second, I work with our members in order to act as a spokesperson on behalf of the industry, whether that takes the form of debunking competitors FUD or promoting new aspects of growth for Linux. Third, I make sure our technical initiatives to support Linux through open standards provide benefit to ISV's looking to target the platform. Finally, I work with our legal team to make sure that Linux and the work of the key developers of the platform is unfettered from legal threats. There are lots of other things I do at the Foundation day to day, but with dozens of people on staff working on these initiatives it keeps me pretty busy.

Brian: Yes, by the way, the Legal Summit that the Linux Foundation held in Boston in October was pretty informative from what I gather from our lawyer who attended. Part of the benefits of being a member! The Collaboration Summit was a great success it seems - I plan on attending the next one. So Jim, besides hanging out with me, what do you enjoy most about your job?

Jim: I enjoy meeting smart people. I get to travel the world and see how Linux is used in innovative new ways, whether that is a cutting edge device manufactured in Taipei or the One Laptop Per Child initiative which uses Linux to help children in poverty.

Brian: Actually, I feel the same way - I've met some cool people through my involvement with Linux and open source. The conversations are thought provoking. A lot of talent there - and some of that talent is at the Linux Foundation itself. So tell me, what's your impression of Linus Torvalds?

Jim: Linus is one of the smartest and most humble people I know. I am truly impressed with how much he has accomplished with Linux. His ability to work with people to create consensus, to write phenomenal code, and do it all while raising a wonderful family is truly impressive. His analytical mind is without question, but what impresses me the most about Linus is his emotional intelligence and his true humility. It is inspiring.

Brian: I've mused on the management style around Linux development. It seems very unstructured - yet works as a collaboration in ways that many much smaller projects I've seen haven't. There's a tremendous amount of trust it seems amongst the development community, and an underlying common purpose - where contributions weigh most in the voice you have. So, Linux is fully accepted in the enterprise today, it is embedded in many of our customer environments, and it is embedded in many products. By all measures it has been an outstanding success for the Linux community. What challenges do you see facing Linux in the next couple years? What role will the Linux Foundation play?

Jim: I believe that as Linux grows and expands there will be several challenges. First will be one of continuing to allow more people to participate in the development process. Large parts of the world have not yet joined in the development process in China, India, Eastern Europe and other areas due to language barriers or low awareness of Linux. We hope to help that by providing forums where key developers can spread the word about Linux and open source in these regions. Another challenge for Linux as a platform will be consistency. The Unix wars are evidence of what can happen when an operating system fragments. Through our work at the Foundation with the Linux Standard Base we hope to provide a degree of consistency across the various versions of Linux so that developers can easily target the platform. Finally, I believe we will need to continue to respond to competitive FUD about legal risks around Linux or other ridiculous claims and I believe the Foundation is in a good role to make that response.

Brian: Well, I look forward to working with you in the coming year within the Linux Foundation! The weather hasn't improved much during this conversation... I'm wondering if I need to go out and get some sandbags? Have a Great New Year! Gotta run!

FAST '08 Conference - approaching ... fast!

The File and Storage Technologies conference is a pre-eminent conference for storage designers and researchers held once a year. The FAST '08 conference is nearly upon us! It will be held again in San Jose, CA, February 26–29, 2008.

I have to say I'm particularly excited about this year's conference - 6 of the 21 refereed papers to be presented are from Network Appliance and our university research partners. As a technologist at the most cutting edge storage company in the industry, this really puts a spring in my step.

Check it out:

The program:

Pergamum: Replacing Tape with Energy Efficient, Reliable, Disk-Based Archival Storage

AWOL: Adaptive Write Optimizations in Linux

Is Disk the Dominant Contributor for Storage Subsystem Failures? A Comprehensive Study of Failure Characteristics

Parity Lost and Parity Regained

An Analysis of Data Corruption in the Storage Stack

SWEEPER: An Efficienty Disaster Recovery Point Identification Mechanism

What's really great about these six papers is that the authors are comprised of NetApp developers, some very bright students from several universities who interned at NetApp and university researchers. We have a great intern program at NetApp - and the students always bring a sense of enthusiasm and vibrancy to the Advanced Technology Group reporting into me.

I really hope to see you at the conference - it's always a great time - with an opportunity to meet with storage engineers and discuss where we are - and more importantly - where we are going.

See you there!

P.S. In response to availability of papers. As far as I understand it, full access to the papers on the Usenix site requires a Usenix membership for one year following the conference - which can be as low as $45 per year for a student membership. After one year the papers are available for free to one and all (for example - all the FAST 2007 papers seem to be online now). Usenix needs membership money to run, and I assume the one year incentive to become a member helps them make ends meet.

So, expect the same for the FAST 2008 conference. That said, I would note that there are often related papers on the university sites for the papers submitted to FAST. And there are other conferences more aggressive about posting their papers quickly such as SOSP 2007 - and if you follow the links off the technical program page - you can eventually access such gems as this paper on Dynamo.

Finally, the ATG Group is organizing NetApp submitted papers to conferences over the next couple months to post as quickly as possible (per the various agreements with refereed conferences and journals) on our web site - I'll update this information when we go live with that publication effort.

P.P.S. FAST is this coming Monday - and I would note that we just had John Strunk start at NetApp, co-author of a seventh FAST paper written while at CMU - "Using Utility to Provision Storage Systems".