NetApp - On The Edge

Fascinations

My apologies. I promised to write a series of blog entries reflecting on WAFL. And I have been writing them - several in my head. Editing. Discussing them with a few people at NetApp.

I believe this is known as writer's block.

I have been reading Designing Type by Karen Cheng. I have to make a confession - I am totally fascinated by typography. As a kid, I bought a small Excelsior printing press with three drawers of fonts from a friend for $25 - it unfortunately is long gone. Early in high school I had the opportunity to go to the Newark Star Ledger (a local newspaper) and watch what I recall to be their Linotype machine in action. Later in life I took an interest in small edition books from presses like Trillium (now Electric Works) - books such as Sandow Birk's and Marcus Sander's modern reimagining of Dante's Divine Comedy.

I'm slowly reading Cheng's illuminating text on the "general issues of type design." Because it is quite an enjoyable read. Spare in exposition, dense in information, and a model of clarity. An inspiration shall I say for me to overcome my writer's block?

I got stuck on the letter 'O'. I never considered it carefully. To quote from Cheng's chapter:

Designing a capital O today involves both objective and subjective rules. First and foremost, the capital 'O' in a serif typeface should always be circular or oval. Rectangular, square, diamond, triangular or 'free' O forms are not legible, since readers of the Roman alphabet have long been conditioned to recognize an O by its symmetry or roundness.

It proceeds from there for several pages with examples from the major families of serif fonts that illustrate the variety within the design constraints of the letter. It is then onto the letter 'E':

The E is the logical letter to design after the O. The E sets several critical factors for the entire font: the proportional system, visual centre, vertical stem width and serif and bracket style.

As I made my way to 'D', I was struck by the near inevitability of the design of a typeface once you choose the starting point. And the conceptual integrity that arises as you innovate within the chosen constraints of the design rules.

It struck me that my attempt at distraction from my writer's block led me back to consideration of the basis for the original design of WAFL. Dave Hitz reflected back ten years later that Mike Malcolm laid out three critical components that WAFL had to enable:

• RAID
  
• Write Optimized Layout (reallocate on write)
  
• Delayed Write Allocation (battery backed RAM)
  

Within these constraints, Dave proceeded to design a new file system. Years later, looking back, I was struck by the inevitability of what Dave created. And the integrity of the design.

Is this how you slip past a writer's block? Let me curl up with the serif capital 'B'.

Mobile Me - Projekt Revolution

So, finally a software upgrade to Apple iPhone app for Typepad allows me to blog while out and about.

Now considering it, untethered context is sometimes unrelated to work.

Hanging out with a finance person from Cisco, a struggling artist, and research assistant studying diabetes at UCSF. Waiting on Jim Voll, one of the current WAFL architects.

Linkin Park headlines - listening to bands in the California sun.

Life is good.

Flexibility is Key

A technical paper describing the Flexible Volume architecture was presented at USENIX '08 in Boston in June. The paper describes FlexVols in ONTAP that support dynamic creation of right-sized data containers that can be resized (grown or shrunk), snapshotted, cloned (producing multiple writable shared unmodified data instances for efficient multi-use "copies" of data), and mirrored for disaster recovery. These features are part of our storage virtualization technology.

FlexVols evolved naturally in WAFL, from its original single volume, single RAID group design by Dave Hitz going back to the beginning of NetApp. Over the next few postings (I really do need to pick up the pace here on blogging) I want to explore some of the features and evolution of this core technology to our storage products.

From a technologist's viewpoint, what has struck me fundamentally about the design and implementation of WAFL has been its remarkable ability to evolve in an upward compatible way to take on new features and enable new capabilities since its inception. Prior to joining NetApp in January, 1994, I was at Sun Microsystems where I was a bit involved with discussions on file systems directions at Sun. I remember being struck by the impression that WAFL at that time had many of the things we were looking for in a new file system for Solaris.

Technology is interesting. You see the end result and sometimes marvel. But technology doesn't create itself. People create technology. Looking at the arc of WAFL technology over my time at NetApp I see three distinct phases from Dave's original design, to a multivolume, multiraid implementation led by Blake Lewis, to the FlexVol architecture led by John Edwards. After Dave's initial work, the development of WAFL grew to include many contributors.

The biggest reason I stay at NetApp is the pleasure of working with truly talented developers and shipping innovative product to our customers. It's a lot of fun!

We are going through some exciting times in the storage industry. The storage virtualization capabilities of ONTAP are proving to be a key enabler for re-architecting data centers around application server virtualization. The greatest gains in power efficiency and reducing data center footprint is by taking an end-to-end approach from the application to the storage holding the application and its data. As a technologist I find it interesting that WAFL and ONTAP continue to enable innovative and flexible data center storage architectures and enhance capabilities application provisioning and deployment.

Over the next few weeks I want to describe not only the what, but the why, of the architecture and design of WAFL and ONTAP and how it affected our product evolution.

How to Participate in the Linux Community

The Linux Foundation (of which NetApp is a member) has recently launched a resource for the Linux community - The Linux Developer Network. The site is currently in beta mode (like the perpetual Beta state of Google Mail? :-).

One very interesting resource is a small book entitled How to Participate in the Linux Community. A welcome practical guide for developers and engineering managers alike in how to participate in Linux kernel development.

Women of Vision 2008

The Anita Borg Institute has posted their Women of Vision 2008 videos and acceptance speeches, which I had the honor of seeing live at the awards dinner in San Jose in May, 2008. I had a great time - as did all the other NetApp attendees! A chance to catch up with some people from the CTO panel at the Grace Hopper Conference last October. Make new friends - enjoyed talking to Francine Berman, director of the San Diego Supercomputer Center (need to fly down for a visit!). Share some wine with my former colleague Whitfield Diffie. Listen to Diane Greene's keynote. And to listen to some wonderful acceptance speeches.

Susan Landau, Sun Microsystems Distinguished Engineer, was the winner of the 2008 Women of Vision Award for Social Impact. Her acceptance speech talked about the challenges of challenging conceptions of security and privacy in today's world.

Helen Greiner, Founder and Chairman of the Board, iRobot, was the 2008 Women of Vision Award Winner for Innovation. She spoke about challenging assumptions on roles throughout people's early education.

Justine Cassell, Professor Northwestern University, was the 2008 Women of Vision Award Winner for Leadership. Her acceptance speech was simply brilliant. I truly encourage you to watch and listen to the video.

Looking forward to Grace Hopper in Denver in October!

Supporting Open Source Is A Privilege

I was particularly happy to see NetApp well represented in a recent paper describing contributions to the Linux kernel. NetApp is in the top twenty companies contributing code to the Linux kernel - much of it through our support of the Linux NFS Client. I got involved personally in getting NetApp more involved in Linux kernel development as our customers in 1999 started to increasingly use Linux in their data center operations. What I love about the Linux community and the people that comprise it is their openness to contributions - both bug fixes and significant feature enhancements.

NetApp has a long history of using open source, and I firmly believe that the use of open source should be followed by investing in the communities developing it. Indirectly, our membership in The Linux Foundation is part of that support. In the past, we've directly funded separate programs for NFS testing and SAN attach scalability in Linux through The Linux Foundation.

We use parts of FreeBSD in our products, and we're trying to support the work of that community also. Again, direct code contributions from our developers is our greatest contribution in my mind. But donations to organizations is also important - with a lot of these contributions coming from my CTO Office - such as our support for the FreeBSD Foundation (gotta check into status of our 2008 donation I notice!). Being a storage company we've found that sometimes providing highly reliable storage for development activities is the best way to help out some projects.

It is a privilege to be able to give back to these communities - important not only to NetApp but more importantly to our customers.

And personally, I enjoy the time I spend with the open source developers. Always engaging. And while I'm definitely not trying to start an open source war, my personal experience is that the Linux community does really know how to celebrate. I'm looking forward to the next Linux Kernel Summit! (And where are we with the sponsorship on that this year? Another thing to check up on:-)

NetApp raises the bar in SAN performance reporting

On Tuesday, January 29, 2008, NetApp published a couple of SPC-1 benchmark results  for the mid-range FAS3040 storage system.  With our highly efficient Snapshot(TM) feature on the FAS3040 with RAID-6 (RAID-DP) enabled, and our thin provisioning FlexVol architecture, the results clearly show that enabling data protection comes at a very modest cost in performance. The  features enabled on the NetApp storage are recommended in our best practice deployments and represent a realistic customer configuration. By using the independently audited, industry standard benchmark SPC-1 to report performance effects of these features, we hope to raise the level of conversation around storage system performance measurement and reporting.

Characterizing the performance impact of NetApp's Snapshot technology is the significant result - there is only a 3% drop in throughput from a base of 30,985 ops/s.

By way of comparison, as part of this study we measured a competitive system from EMC, the  CX3-40 system. In sharp contrast to the NetApp result, the EMC CX3-40 system configured in a standard mirrored RAID configuration - following EMC best  practices - suffers more than a 60% decrease in performance when their snapshots are enabled.

You can find more information about the results here.

The team at NetApp that did this work (thanks everyone!), and the product development group, are justifiably proud of these results. Such information on the costs and tradeoffs around standard storage management technologies is needed to make informed decisions about  storage deployments.

Benchmarks by their nature  tend to be controversial. But they are simply tools - some better, some worse - used to measure performance and compare two or more systems. A benchmark result is a combination of several factors. First, all benchmarks define a specific workload - a model, sometimes actually more often synthetic - to subject the test system to. Better benchmarks reflect attention to calibrating the workload with actual application workload results. The next factor is the configuration of the system under test - here is where a lot of controversy lies in benchmark results. "Tuning a configuration" to achieve good results is expected - it's when the tuning gets increasingly unrealistic - or forbid, benchmark specific - that things can get confusing. Which brings me to uniform run and reporting rules. Good benchmarks strive for useful comparisons by having run and reporting rules sufficient to allow others to reproduce the result. Finally,  peer review or auditing of the results prior to publication is required to maintain the integrity of their respective benchmarks. The SPEC SFS and SPC-1 benchmarks represent two of the more useful and rigorously defined benchmarks for storage performance measurement.

What is very cool about NetApp results like this is that we have striven to configure the system in much the same way as a customer would (following our best practice recommendations).  These results use much of our core feature set (RAID-DP, Flexible Volumes, Snapshot feature).  They are run using an industry standard workload, and audited by an independent auditor.  They are real.

The FAS 3040 result is the only SPC-1 published result using a RAID-6 implementation at this time. In fact, it's one of a small handful of published results using anything other than (the more costly) traditional RAID 1+0 mirror. NetApp's RAID-DP is an extremely efficient RAID 6 implementation that is enabled by default on our storage systems - including our StoreVault S300 product for small and medium businesses. (In the interest of full disclosure, I must say I am not only an employee of Network Appliance, but a fervent customer of one of our resellers and proud owner of an S300 - with a two year support contract:-) Further, this result is on our industry leading storage software stack that provides a Unified Storage Architecture (with the ability to support NAS, iSCSI and FC SAN access), and sophisticated thin provisioning capabilities that allow you to right size your LUNs and data containers for your application deployment. Because we have a single unified storage stack, it is possible to extrapolate these results to other deployments (say iSCSI or NAS connectivity) for a similar workload.

The other aspect of the reported result is the characterization of performance overhead when snapshots are enabled.  Snapshots, for online backups and application recovery points, are a commonly supported feature today in many storage offerings - but is usually implemented as a copy out operation  of unmodified data to a separate volume as opposed to NetApp's Snapshot in-place implementation. Snapshots were done more frequently on the NetApp FAS 3040 (every 15 minutes) vs. the EMC array (every 1 hour). Further, the NetApp  storage retained three snapshots rolling the oldest one off as a new one was created, versus one snapshot retained in the EMC measurement. Given the performance cost of the copy out snapshot mechanism in the CX-3 40 it was felt this was a more realistic frequency and depth.

Our belief is that the NetApp performance under SPC-1 load (NetApp result dropped only 3% vs. the nearly 60% performance drop in the measurement in the EMC result) reflects  that the Snapshot features in  Data ONTAP are more efficient because they  were designed into the product from the start as a fundamental feature of our solution.

So, what does all this mean to you? Realize that your mileage will vary - your workload will undoubtedly have different characteristics than that defined by SPC-1. But broadly speaking, the results provide apple-to-apple comparisons of two  competitive mid-range storage arrays and show the viability of an efficient RAID 6 solution vs. a standard RAID mirror configuration, and the dramatic performance overhead difference that you are likely to encounter with different snapshot architectures and implementations.

So what happens next?  There are a few additional aspects of these results that I will discuss in this blog in the near future. Longer term, we'll continue to improve our products, to make them more effective at solving more customers problems. And we'll publish benchmarks from time to time, allowing customers to understand the competitive context. Other critical storage features, such as remote mirroring and disaster recovery, bear characterization. We will publish when we have interesting things to say - and work with standard benchmarks and the organizations that develop them to encourage measurement methodologies around realistic customer configurations and concerns.

A conversation with Jim Zemlin

What started out as a face to face conversation with Jim Zemlin, Executive Director of the Linux Foundation, morphed into an e-mail interview as weather and San Francisco parking tickets reared their ugly heads in the New Year. I am on the board of the Linux Foundation (NetApp is a member company). Jim and I were catching up - and he helped me write my next blog entry.

Brian: Good afternoon, Jim. Great weather we're having in San Francisco this New Year.

Jim: The weather here is terrible. I suspect it is due to global warming and in the spirit of relating everything to the Linux Foundation we have a Green Linux initiative to enhance power management of Linux and make the world a better place.

Brian: So, NetApp is a member of the Linux Foundation, before that, we were a member of OSDL [one of the predecessors of the Linux Foundation]. NetApp, as you know, is the most innovative storage vendor in the industry, and our product is based on proprietary software for the most part. I know why we are a member, but I'm curious from your perspective, where do we fit in to the Linux Foundation and open source?

Jim: NetApp, like many other companies in the world, depend on Linux for a variety of things. First, it is important for NetApp's customers to understand that their products will work effectively with Linux. Because Linux is the fastest growing platform on the planet, I suspect there are a lot of NetApp customers that fall into this camp. Second, NetApp uses Linux in some of their products and by doing so benefits from the collective R&D that goes on in key open source projects such as the Linux Kernel or other components of open source software. Keeping a close eye on these projects and building up relationships with the other participants in these projects is something that is important to NetApp. At the Foundation we provide a forum within which NetApp can work with their industry partners and the community to benefit all of those working on making Linux and open source better.

Brian: Yes. The other place we're playing is in direct support and contribution to the Linux code base. We employ Trond Myklebust (the Linux NFS client code maintainer), and have a couple projects underway in NFS and other storage technologies that are generating code for inclusion in Linux. And certainly Linux is used by many of our development engineers as a development platform. What's it like to be the Executive Director of the Linux Foundation? Your day to day?

Jim: Lots of different things. First, I make sure that the people that work at the Foundation, including Linus Torvalds, get paid on time and have the best possible work environment we can provide. Second, I work with our members in order to act as a spokesperson on behalf of the industry, whether that takes the form of debunking competitors FUD or promoting new aspects of growth for Linux. Third, I make sure our technical initiatives to support Linux through open standards provide benefit to ISV's looking to target the platform. Finally, I work with our legal team to make sure that Linux and the work of the key developers of the platform is unfettered from legal threats. There are lots of other things I do at the Foundation day to day, but with dozens of people on staff working on these initiatives it keeps me pretty busy.

Brian: Yes, by the way, the Legal Summit that the Linux Foundation held in Boston in October was pretty informative from what I gather from our lawyer who attended. Part of the benefits of being a member! The Collaboration Summit was a great success it seems - I plan on attending the next one. So Jim, besides hanging out with me, what do you enjoy most about your job?

Jim: I enjoy meeting smart people. I get to travel the world and see how Linux is used in innovative new ways, whether that is a cutting edge device manufactured in Taipei or the One Laptop Per Child initiative which uses Linux to help children in poverty.

Brian: Actually, I feel the same way - I've met some cool people through my involvement with Linux and open source. The conversations are thought provoking. A lot of talent there - and some of that talent is at the Linux Foundation itself. So tell me, what's your impression of Linus Torvalds?

Jim: Linus is one of the smartest and most humble people I know. I am truly impressed with how much he has accomplished with Linux. His ability to work with people to create consensus, to write phenomenal code, and do it all while raising a wonderful family is truly impressive. His analytical mind is without question, but what impresses me the most about Linus is his emotional intelligence and his true humility. It is inspiring.

Brian: I've mused on the management style around Linux development. It seems very unstructured - yet works as a collaboration in ways that many much smaller projects I've seen haven't. There's a tremendous amount of trust it seems amongst the development community, and an underlying common purpose - where contributions weigh most in the voice you have. So, Linux is fully accepted in the enterprise today, it is embedded in many of our customer environments, and it is embedded in many products. By all measures it has been an outstanding success for the Linux community. What challenges do you see facing Linux in the next couple years? What role will the Linux Foundation play?

Jim: I believe that as Linux grows and expands there will be several challenges. First will be one of continuing to allow more people to participate in the development process. Large parts of the world have not yet joined in the development process in China, India, Eastern Europe and other areas due to language barriers or low awareness of Linux. We hope to help that by providing forums where key developers can spread the word about Linux and open source in these regions. Another challenge for Linux as a platform will be consistency. The Unix wars are evidence of what can happen when an operating system fragments. Through our work at the Foundation with the Linux Standard Base we hope to provide a degree of consistency across the various versions of Linux so that developers can easily target the platform. Finally, I believe we will need to continue to respond to competitive FUD about legal risks around Linux or other ridiculous claims and I believe the Foundation is in a good role to make that response.

Brian: Well, I look forward to working with you in the coming year within the Linux Foundation! The weather hasn't improved much during this conversation... I'm wondering if I need to go out and get some sandbags? Have a Great New Year! Gotta run!

FAST '08 Conference - approaching ... fast!

The File and Storage Technologies conference is a pre-eminent conference for storage designers and researchers held once a year. The FAST '08 conference is nearly upon us! It will be held again in San Jose, CA, February 26–29, 2008.

I have to say I'm particularly excited about this year's conference - 6 of the 21 refereed papers to be presented are from Network Appliance and our university research partners. As a technologist at the most cutting edge storage company in the industry, this really puts a spring in my step.

Check it out:

The program:

Pergamum: Replacing Tape with Energy Efficient, Reliable, Disk-Based Archival Storage

AWOL: Adaptive Write Optimizations in Linux

Is Disk the Dominant Contributor for Storage Subsystem Failures? A Comprehensive Study of Failure Characteristics

Parity Lost and Parity Regained

An Analysis of Data Corruption in the Storage Stack

SWEEPER: An Efficienty Disaster Recovery Point Identification Mechanism

What's really great about these six papers is that the authors are comprised of NetApp developers, some very bright students from several universities who interned at NetApp and university researchers. We have a great intern program at NetApp - and the students always bring a sense of enthusiasm and vibrancy to the Advanced Technology Group reporting into me.

I really hope to see you at the conference - it's always a great time - with an opportunity to meet with storage engineers and discuss where we are - and more importantly - where we are going.

See you there!

P.S. In response to availability of papers. As far as I understand it, full access to the papers on the Usenix site requires a Usenix membership for one year following the conference - which can be as low as $45 per year for a student membership. After one year the papers are available for free to one and all (for example - all the FAST 2007 papers seem to be online now). Usenix needs membership money to run, and I assume the one year incentive to become a member helps them make ends meet.

So, expect the same for the FAST 2008 conference. That said, I would note that there are often related papers on the university sites for the papers submitted to FAST. And there are other conferences more aggressive about posting their papers quickly such as SOSP 2007 - and if you follow the links off the technical program page - you can eventually access such gems as this paper on Dynamo.

Finally, the ATG Group is organizing NetApp submitted papers to conferences over the next couple months to post as quickly as possible (per the various agreements with refereed conferences and journals) on our web site - I'll update this information when we go live with that publication effort.

P.P.S. FAST is this coming Monday - and I would note that we just had John Strunk start at NetApp, co-author of a seventh FAST paper written while at CMU - "Using Utility to Provision Storage Systems".