Extensible NetApp: Thinking outside of the box

VTL Deduplication, NetApp Style

I believe that architectural style is a natural outgrowth of your organizational DNA.

If you spend all of your time thinking about how to modify a software system, then you tend to modify the software system.

If you spend all of your time building different hardware platforms, then you tend to build lots of hardware systems.

And this fundamental philosophical approach to system design is reflected in how you think about systems. For example, for a NetApp engineer the notion that a unified storage platform is a combination of a set of distinct hardware pieces running different software is an odd statement. For EMC, apparently, the combination of different hardware systems into an aggregate hardware system is unified storage. And I guess from their point of view, namely that there is a single aggregate hardware device that can do everything, they are right.

And I've explained why the two approaches offer fundamentally different value propositions.

Why bring this up, again?

Because the NetApp VTL team announced the availability of deduplication on our VTL systems.

And because, the NetApp VTL team has our engineering DNA in them rather than apply bolted-on technology, like EMC, they integrated deduplication into the core system design.

The net effect is that the VTL team can deliver a simpler system, a more reliable system and I believe, ultimately, a faster system. The system is simpler because it has fewer moving parts. More reliable because it does not have multiple hardware components that can fail. And a faster system because integrating deduplication with the code that manages the disk ultimately can give you faster performance because you don't have multiple layers of software mucking with the disk layout.

I believe those attributes of simplicity and reliability and speed have real value to customers. 

So I am very excited with this most recent announcement, not only because we address a key customer demand, but also because of the way we did it.

Why WAFL is at the core of unified storage

A while ago, in part a reaction to the baffling claims made by EMC about Celerra NX4, I explored what is unified storage. And in particular, I tried to expose the fact that EMC had redefined he value proposition of unified storage from:

A storage manager who wants a single device to support all protocols should not have to buy or setup more than one device, should not have to learn different tools to perform the same task, and should not have to understand different replication mechanisms to do DR or backup should buy a unified storage device.

to:

So if you are a storage manager who wants a storage device that requires multiple distinct devices to be configured, that management for common tasks involves multiple distinct tools, that has multiple protocol and device specific replication strategies, then the Celerra NX4 is that storage device.

Because of this EMC distortion, NetApp has decided to take back the definition to mean what we know it means.

When I wrote up that series, I concluded there were two definitions of unified storage, a unified storage device and a unified storage architecture.

And to remind folks a unified storage device has the following attributes:

1. A piece of hardware that has CPU, Memory and disk
2. That supports FC, iSCSI, CIFS and NFS
3. That has a common management console for all storage functions that are not protocol specific
4. Has a single replication mechanism that is independent of protocol

And a unified architecture has the followed attributes:

• Each device is a unified storage device
• Each device may have different amounts and kind of CPU and Memory and Disk but components must be interchangeable.
• The management model for each device is the same.
• Any device can replicate storage to any other device

And I drew this picture for what a unified storage device should look like:

But I never really explained what enables NetApp to deliver this value.

If you read my series on WAFL, you'll find this picture:

And the answer is fairly obvious: WAFL.

Because of the way we layered the protocols on top of WAFL, and implemented RAID as a software layer below WAFL we offer a single hardware and software stack for all kinds of data access enabling us to have a single piece of hardware (1) that supports all protocols (2).

Because all space management is done in WAFL ,we have a consistent way to manage space across every protocol. You provision and manage storage space in the same way. Because of the way RAID is layered below WAFL all of your disk management is done in the same way. We can, therefore, offer a common management console for all storage functions (3).

Because all reads and writes go through WAFL and because all space is managed through WAFL we can have a single replication mechanism for all protocols (4).

And finally because all of our hardware platforms run the same software, we offer a unified storage architecture for our FAS systems.

If there is a point to this blog post, it is that a unified storage device and architecture are not bolted on features, or marketing spin, but fundamental architectural decisions about how storage systems are built.

Why WAFL is Not a File System.

In response to a set of comments, I wrote up a series on WAFL to address the common misconception about WAFL being a file system and in particular, how SAN is layered within Data ONTAP.

This post will act as a launch point to that series.

In my first post, I explained why the notion that WAFL is a file system did not match my experiences using WAFL.

Next I explored what is a file system and offered a reasonable set of attributes that defined a file system.

Then I compared WAFL to those attributes. I used that comparison to argue WAFL was not a file system but something new.

I then turned to the question of how WAFL and SAN are layered.

Finally I offered some thoughts on why there is so much confusion about WAFL.

What is WAFL part V: Confusion and Conclusion

So why the confusion?

The optimist in me believes that the reason we have so much confusion about WAFL and SAN is that NetApp has not done a good job explaining the layering. And that without an explanation, it was reasonable for folks to assuming the following, very incorrect layering:

Or using my object diagram

 

that a File IS-A inode and a LUN IS-A File.

And some concluding thoughts and an apology.

This all began because of a comment I made in a response to a comment by Steven Schwartz. And I said later on, in a rather harsh tone of voice:

Comments, such as yours, that WAFL == FS, remains one of the more entertaining, misleading and ill-informed comments that our competitors continue to make on a regular basis.

In retrospect, if Steven was ill-informed that was my fault, and if he was ill-informed then he could not be misleading, and shame on me for being entertained by his ignorance.

So I apologize for my response, because it was rude.

I hope, in future, this discussion about whether WAFL is or is not a file will not be necessary.

WAFL is not a file system. WAFL provides mechanisms that enable a variety of file systems and technologies that want to access disk blocks. WAFL mechanisms deliver industry leading performance, while offering customers the flexibility to use any and all of our data management primitives with no compromise to that performance.

What is WAFL part IV: SAN and WAFL

Thus far, I've been making an argument that WAFL is an underlying infrastructure for building file systems by observing that it serves the basis for two distinct distributed file system technologies, and has served as the basis for a streaming media file system.

But the problem with that argument is that it asks the reader to buy into the notion that there is a smallest unit of software that is a file system and anything less than that is not.

The fact that WAFL supports SAN storage and the way in which it supports SAN storage is, in my mind, a more compelling argument.

But a brief digression on What is a LUN?

The key intuition is that a LUN, fundamentally, is a logical organization of disk blocks. It turns out that a logical organization of disk blocks requires some on-disk data structure. And it turns out that the on disk data structures that represent a file can be used to represent a LUN.

However, just because the on-disk data structure that tracks disk blocks happens to be the same as a file, does not mean that the LUN is accessed or manipulated in the same way a file is.

LUNs and WAFL

This picture, I hope, explains the layering of SAN on WAFL. SAN uses some WAFL primitives, in particular the ability to read and write disk blocks but does not use all of the primitives.

Furthermore, not illustrated in this diagram is that there are specific optimizations within the read and write paths to get the kind of performance that SAN customers expect.

Now because the underlying data structures required to read from disk are the same as the ones NFS uses to manipulate files, an interesting side effect emerges, namely an NFS client can access see the LUN as a file. But remember, when NFS is accessing the LUN it’s going through a different path through the system. More to the point, the on-disk object is only a file when you go through the NFS stack, not when you go through the LUN path.

If you have any kind of object oriented programming language training, you could almost say the following:

That an inode is the on-disk object, and a LUN IS-A inode, and a FILE IS-A inode, but a LUN is not a FILE.

Okay, so, what’s the point?

The point is that WAFL is the part of the code that provides the “read or write from-disk” mechanisms to both NFS and CIFS and SAN. The semantics of a how the blocks are accessed are provided by higher level code not by WAFL, which means WAFL is not a file system.

What is WAFL part III: An Answer

Let’s agree that a file system has the following elements:

1. Methods to allow operations on files
2. Methods to allow operations on directories
3. Methods for storing data on disk
4. Methods for retrieving data on data.
5. Data structures for representing data on disk
6. Rules about how files are found
7. Method for organizing disks into large physical pools
8. Methods of partitioning physical pools into smaller logical pools

Let’s focus a little bit on item #1 and #2. and #6

In the particular case of WAFL, what is a file or what is a directory is defined by the distributed file system WAFL is a part of. And very early on, and before I joined, and before I even knew what Data ONTAP was, smarter people than me figured out the following interesting layering:

What they did was create a common set of primitives on top of which the NFS and CIFS semantics could be commonly layered. The NFS and CIFS semantics, were part of the protocol stack, the shared primitives were part of WAFL. So even at this early stage, WAFL provided a mechanism for storing files and directories, without assuming the semantics of those operations.

In fact the rules about how  files are looked up is defined by how NFS clients choose to lookup files rather than how Data ONTAP works.

In this GX paper there is a discussion in section 5 of this separation in the context of the Spin NP protocol. Although the discussion is about GX, it supports much of what I am saying here.

So is just providing mechanisms sufficient to be called a file system? I’m not so sure.

At this point, my seems closed but ....

Now let’s focus on part 3 and 4.

It turns out that the layering of file-systems has evolved over the last 20 years. When I was in college, a file system was responsible for the disk layout, because the OS knew where the data blocks resided. Today, with LUN’s, many file systems persist in the illusion, but the reality is that the layout of blocks is a mystery. The genius of Hitz and Lau was that the designed WAFL around the explicit assumption that WAFL did not control exactly where disk blocks were allocated, but instead, a lower layer, RAID did that.

Ironically, ONTAP RAID is particularly tuned to work with WAFL. The two are very much tied together. So although today almost every file system lives on top of some RAID layer, only WAFL and ONTAP RAID know enough about each other to take advantage of each other.

But to the question of whether WAFL is a file system, well it does some of 3 and some of 4, but it very much relies on something outside of WAFL to do things that traditionally were the domain of the file-system.

And what about parts 7 and 8?

Today most file system vendors consider the volume manager (8) and the logical volume manager (7) to be external to the file-system. In fact, Symmantec clearly distinguishes between the volume manager, vxvm and the file system vxfs.

But at NetApp, WAFL is both the volume manager and logical volume manager.

Which leaves 5?

And yes, the on-disk data structures are all WAFL data structures. And this is an important point. Because WAFL has a single on-disk format, NFS and CIFS file operations can be performed on the same underlying on-disk object.

Well then what is WAFL?

Well it provides mechanisms for building file-system semantics, it manages the on-disk format, it manages the free and allocated space, and provides a logical and physical volume manager.

So in my mind, that’s not enough to call it a file system. WAFL is a necessary part of a file-system, but not sufficient.

In my next post I will explain how SAN fits into the picture and provides further evidence that WAFL is not a file system.

Death to agents that add little value

Paul Turner wrote Death to Agents. And that got me thinking about agents in general.

There is a common and reasonable perception that everyone hates agents. They are a pain to deploy, they are pain to manage, they are a pain to upgrade. And yet we want the functionality they bring to the table. And so we install them, grouchily.

Paul's post really speaks to the following question:

If the agent functionality can be done without an agent do we need the agent?

And the answer is a resounding: No!

And I believe that for Storage Resource Management, the delta what can be done without an agent, and with an agent is so small, that the value of agents is negligible.

The problem is that for a lot of other very interesting functionality we need agents. For example, to create an application consistent image we need to interact with the application to create that image. And a secure, robust, and reliable way to interact with an application is to use agents.

Of course, as environments add network APIs that provide some if not all of the functionality of an agent, we need to continue to evaluate whether an agent makes any sense.

So for example, when we built SMVI, we specifically did not install an agent either in the Console OS or inside the guest, because VMware provides all of the necessary APIs.

On the other hand, when we integrated OSSV with VMware we chose to deploy an agent in the Console OS because we could deliver a significant performance boost over agentless or VCB approaches.

What is WAFL part II: What is a file system?

In my last post, I gave the motivation to my proposition that WAFL is not a file system.

According to the zeitgeist definition in Wikipedia:

A file system is a special-purpose database for the storage, hierarchical organization, manipulation, navigation, access, and retrieval of data.

This is actually a very good definition. The problem is that it can be too expansive.

I believe that a file system definition should also encompass how we experience file systems.

We expect a file system to have operations like open, read, write, and close that can be applied to files. We expect files to have specific kinds of meta-data like names, access control lists and time.We expect that before we can open a file we have to traverse a directory structure. We expect that the directory structure to be a rooted graph.

And on top of that each file system has additional rules and semantics that enforce how files get accessed and how the directory hierarchy can get manipulated.

My first, cheesy, observation is that for a user of WAFL, the file system is not WAFL, but NFS or CIFS.  The semantics of how data gets accessed, organized, and manipulated are defined by the NFS protocol. NFS is a distributed file system, that has as a component a thing it calls the file system server, but that there are additional components like authentication services, that are not part of any reasonable file server. This holds true even for CIFS.

So case closed, right? That is too cheesy for me to be satisfactory. 

I wanted to go a little bit deeper than that.

Another way of probing the question is the following…

Let’s agree that a file system has the following elements:

1. Methods to allow operations on files
2. Methods to allow operations on directories
3. Methods for storing data on disk
4. Methods for retrieving data on data.
5. Data structures for representing data on disk
6. Rules about how files are found.

Let’s further agree that a modern storage stack also has the following elements

     7.     Method for organizing disks into large physical pools

     8.     Methods of partitioning physical pools into smaller logical pools

:

My next post is to explore how WAFL and Data ONTAP match up against those elements.

What is WAFL part I: An introduction

Over the years, potential NetApp customers have been mystified about the relationship between WAFL and SAN. In particular, are NetApp LUN’s a file on top of a file-system?

For someone who has worked at NetApp for many years, the question always struck me as odd, because my one experience using WAFL was that it provided mechanisms that allowed you to build a file system, but was not itself a file system.

My first project at NetApp was to build a streaming media cache using WAFL. And for us WAFL was a way to move data to and from disk and a way to store our on-disk format, but was most certainly not a file system. For a LEGAL description you can check out our patent.

Consider that a file system has rules about how you access and find objects the file system manages. Conventionally, you first have to do a directory look up, and then open the file, and then do the read or write operations on a file handle returned by the file system. WAFL imposed no such constraints on our design. We could directly write to or read from an object. We were free to use what ever name lookup scheme we wanted.

As a concrete example, when we were serving a piece of a media stream we would use a clever hash to find the object on disk without doing a name look up, effectively bypassing the entire traditional directory lookup.

What WAFL provided  was a mechanism for tracking all of the disk blocks. WAFL also managed the free space. WAFL was responsible for ensuring all of the data got written to disk reliably.

In fact, when we wanted to use WAFL as a standard file-system we discovered that WAFL did not provide the necessary interfaces. For example, WAFL had no mechanism for handling open and close operations. More annoyingly, WAFL was optimized, circa 2000, for 32kB and 64kB read and write operations not 12 Byte writes.

To build our product we ended up using a whole bunch of WAFL mechanism to build our own specialized data store optimized for streaming media. During the entire experience, WAFL never felt like a file system. A set of primitives that made it possible for us to build our own file-system like layer sure, but most definitely not a file-system.

So my personal experience was that building a high performance file system on top of WAFL was pretty easy, but WAFL was no file system.  

However, given the fact that our founder, Dave Hitz, called WAFL a file-system, and our original patents and a lot of our historical literature call it a file system, I realize that my point of view may not be shared.

So over the next couple of days I will dig into this topic.

Just Use RAID-DP

When NetApp introduced RAID-DP oh-so-many-years ago, a bunch of folks wondered if it would make sense to release a SAN product that was built directly on-top of RAID-DP.

The concern was that getting NetApp's virtualization layer to perform for SAN workloads would be such a herculean task, that perhaps we could short-circuit our efforts.

After all RAID-DP, offers better reliability and no-compromise performance than RAID mirroring.

In the end, we decided that the path forward was to complete the herculean task ahead of us.

And we did.

And maybe our competition believed their own FUD, that WAFL or NetApp could never perform or scale, so they never bothered to figure out how to make RAID-6 perform in  a traditional array.

So here we are.

We offer equivalent performance, more features, and lower cost with better reliability for all of your storage workloads.

So if you have the time to carefully segregate your workloads, and you have the time to deal with the mistakes you will inevitably make, and you have the time to think through every provisioning request, because doing that is fun, maybe you'll find this advice on RAID-6 from StorageZilla to be useful:

• Don't use RAID 6 in random small WRITE intensive situations where you're pushing the metal to the wall.
• For systems using 1TB SATA or other such capacity dense drives RAID 6 is recommended
• For VMware RAID 6 performs great and works fine.
• For Virtual Provisioning RAID 6 performs great and works fine.

If you're a NetApp customer:

For all of your storage workloads, just use RAID-DP.

Spend more time thinking about how to add more value, like what your DR policy should be.