NetApp - Dave's Blog

To NetApp Employees and Customers on Sun’s Lawsuit

[Note: This is an e-mail that I sent internally to our employees, with the expectation that they might also share it with customers. Some of it repeats previous posts, but other parts are different. In the spirit of openness, I decided to post here as well.]

To: everyone-at-netapp
Subject: Sun's Lawsuit Against NetApp

This morning, Sun filed suit seeking a “permanent injunction against NetApp” to remove almost all of our products from the market place. That’s some pretty scary language! It seems designed to make NetApp employees wonder, Do I still have a job? And customers to wonder, Is it safe to buy NetApp products?

I’d like to reassure you. Your job is safe. Our products are all still for sale.

Can you ever remember a Fortune 1000 company being shut down by patents? It just doesn’t happen! Even for the RIM/Blackberry case, which is the closest I can think of to a big company being shut down, it took years and years to get to that point, and was still averted in the end. I think it’s safe to say the odds of Sun fulfilling their threat are near zero.

If you are a customer, you can be confident buying NetApp products.

If you are an employee, just keep doing your job! Even if your job is to partner with Sun, keep doing your job. Here’s a ironic story. When James and I received the IEEE Storage Systems Award for our work in WAFL and appliances “which has revolutionized storage”, it was a Sun employee who organized the session where the award was presented. He was friendly, we were friendly, and we didn’t talk about the lawsuit. You can do it too. The first minute or two might feel odd, but then you’ll get over it. We have many joint customers to take care of.

If you are a salesman with concerned customers, feel free to show them this note, and maybe also point them at a couple of my blog posts (here and here). But don’t waste too much time before getting back to how we can work with them to solve their storage and data management problems.

I am frustrated with Sun and with Jonathan, as I describe in this blog entry. We have tried to be very open, detailed and specific about how Sun is infringing our intellectual property. We’ve tried to set a higher standard in how companies conduct patent litigation. It’s frustrating that Sun would just do a two-barreled blast, threatening to shut down our company. Frustrating and silly, to be honest, because it’s just so unlikely for a patent case to shut down a major corporation.

The other thing that’s frustrating is the way Jonathan wraps himself in the open source flag. We aren’t against open source, and we aren’t even against non-commercial use of ZFS. The number one rule of open source is that you should only give away stuff that belongs to you. That is what this suit is about, and everything else is just fluff.

Sun Sues NetApp: Says “You Cannot Unfree What Is Free”

Sun is seeking a “permanent injunction against NetApp” to remove almost all of our products from the market place. This is exactly the sort of broad but vague threat that gets people so frustrated with patent litigation. (The timing is no surprise: their deadline for responding to our complaint was this Friday.)

I have tried very hard in my blog to be unusually open – very detailed and specific – about how Sun is infringing our intellectual property. I’ve been trying to set a higher standard in how companies conduct patent litigation, and I’m disappointed in Jonathan for not doing better than this. This sounds like Sun’s broad threats when they sued Azul, but in the end, Sun didn’t put Azul out of business or even stop them from shipping products. I’m quite confident that two years from now – or however long it takes this suit to reach court – NetApp will be doing just fine. (For details on how this whole mess started, see my blog post, Jonathan’s response, and my response to him. I won’t revisit those arguments, so if you have comments about who is evil or not, please put them in those posts.)

But from a philosophical perspective, I found one part of Jonathan’s post especially interesting:

NetApp’s objectives were clear - they'd like us to unfree ZFS, to retract it from the free software community. Which reflects a common misconception among proprietary companies - that you can unfree, free. You cannot.

Jonathan seems to be arguing that once something has been put into open source, it is beyond the law. I disagree completely! To get us away from the details of Sun and NetApp’s particular case, let me make an analogy.

Suppose that I steal and then open-source Jonathan’s patented recipe for chocolate chip cookies. The recipe will probably live forever in the web. There is no getting those bytes back, and if it’s a good recipe, there is no stopping individuals from baking those cookies.

On the other hand, if I start a company to sell Jonathan’s Patented Cookies™, then it’s perfectly reasonable for him to ask me to stop. (Let’s just assume, for the sake of the analogy, that he’s got a valid patent.) Or if Nabisco starts selling Jonathan’s Patented Cookies™, it’s perfectly reasonable for Jonathan to ask them to stop. It isn’t a question of trying to unfree what’s free, or retracting the recipe from the free recipe community. It’s a question of whether corporations must obey the law.

Jonathan’s claim that “you cannot unfree what is free” sets a very dangerous precedent. It says that you can steal anything, as long as you open source it afterwards. That can’t be right! I do understand that many open source proponents argue there should be no legal protection at all for information. “Information wants to be free.” But even if Jonathan believes that, he ought to wait until the law changes before taking Sun down that path.

One of the most important rules of open source is that you must only give away things that belong to you. If protected information does leak into open source, it will probably live forever in the web, but that isn't the issue. To me, the issue is that large corporations should stop making a profit on protected information that doesn't belong to them. That's what we're asking here.

Defining Consensus: What Does It Mean To “Own” a Decision?

Cooperation, collaboration and consensus are part of NetApp’s culture. Partly we feel that it makes for a better work environment, but also that it leads to better decisions and better alignment. However, consensus can lead to paralysis if carried too far. Simply giving one person ownership for a decision – they can do whatever they want – would certainly be much faster. How do you balance between these two extremes? What does it mean to be the “owner” of a decision? 

Dan, our CEO, has very strong views on culture, so a few years ago, when I was struggling with these issues as a manager, I asked his advice on what it means to delegate the ownership of a decision to someone. He said:

When I delegate a decision, I want the owner to identify the key stakeholders, and bring them to a consensus on the right plan, if possible. If they fail, then bring the decision back to me, probably at my staff meeting. (I am sometimes a key stakeholder, so don’t forget about me when driving to consensus!)

The trickiest issue here defining consensus. Even when people have rough agreement, they may differ slightly on details. Does consensus mean 100% agreement? If not, who determines whether consensus has been reached? What does it mean to “own” a decision if you still have to reach consensus?

I asked Dan all of these questions, and he said:

If the owner and key stakeholders cannot reach agreement, then I’d like them to bring the decision back to me, generally at my staff meeting, and preferably in the form of different recommendations to choose from.

But everyone should be aware that I have a strong bias to accept the owner’s recommendation. This bias helps to define consensus.

Consensus doesn’t mean that you agree 100% with the owner. It means that you believe the owners plan is fundamentally broken, and you want to try to convince me of that, despite my bias to accept the owner’s recommendation. I am not interested in arguments that “the owner’s plan is good, but mine is better.” Go convince the owner.

On the other hand, decision owners better be careful about ignoring input from key stakeholders, because if a stakeholder thinks a plan is really busted, then I do want to hear about that!

I love this definition, because it describes consensus operationally, instead of getting bogged down in theoretical philosophy. If you are a stakeholder trying to decide whether to withhold consensus, consider whether your case is so strong that you want to argue it in front of Dan’s staff. (And remember, Dan’s bias is with the owner!) But if you are an owner, and a key stakeholder is objecting, consider whether your case is so strong that you want to argue it in front of Dan. Maybe you should work harder to accommodate the stakeholder’s concern and reach consensus. I bet your plan will be stronger for it.

As near as I can tell, this is Dan's ownership model for pretty much everything. I can't think of many decisions where Dan has told the owner, “You can do whatever you want, and I’ll just support it.” On the other hand, for many areas it is very clear who the owner is, and unless people think there is a serious, serious issue with the owner’s plan, they unlikely to withhold consent.

I’m using Dan and his staff as an example, but obviously, managers who own a decision can apply this model recursively to their own staff, as they delegate further down.

Is NFS a Form of SAN? (NFS for Enterprise Apps)

I recently wrote about how both Oracle and VMware support NFS for situations that would have traditionally used block-based storage, like SAN or iSCSI.

Here is my favorite reaction to Oracle's Direct NFS Client:

This is so wrong, but I guess they're so far down the path it no longer matters.
-– Wes Felter 

I disagree that it’s “wrong”, but it’s important to ask why Oracle over NFS gives some people the heebie-jeebies. Perhaps people worry that the NFS protocol might be “chatty” – somehow less efficient at transporting data across the network. It’s true that NFS has lots of fancy requests that iSCSI and Fibre Channel SAN do not, to create directories, set permissions, move files, and so on, but none of that matters when all you do is read and write blocks of data.

At the protocol level, block-reads over NFS and block-reads over iSCSI are almost identical. The main difference is that NFS asks for a certain number of bytes, starting at a given byte offset, and iSCSI asks for a certain number of blocks, starting at a given block offset. With NFS you must divide by 512 to convert bytes to blocks. So what! The other difference is that iSCSI and FC-SAN use a LUN to identify the container holding the blocks, and NFS uses a file handle. I’m over-simplifying, but you get the point. At the protocol level, for block traffic, there is almost no difference between NFS and iSCSI or SAN.

A LUN and a file are both just containers that hold blocks of data, so get over it. It was this similarity that originally convinced me NetApp could “unify” SAN, NAS and iSCSI into a single storage appliance. (This is the key idea behind unified storage.)

There are significant differences in how the protocols connect to the host operating system. FC-SAN and iSCSI both plug in at the block layer, while NFS plugs in at the filesystem layer. Those have very different paths through the OS. Historically, bugs in the filesystem layer slowed down NFS. For instance, NFS in Solaris used to be single-threaded, so multiprocessor systems only got one CPU worth of NFS performance. Those bugs have been fixed, but in most operating systems, the block layer is still a slightly lighter-weight interface.

This is why it is so significant that both VMware and Oracle have now built NFS directly into their software as if it were a block interface. When Oracle generates NFS packets directly, it completely bypasses the OS filesystem layer, and it reduces the difference between NFS and iSCSI to the minimal protocol differences that I described above. VMware takes block level requests out of virtual machines and converts them into NFS requests, again blurring the distinction between NFS-as-NAS and NFS-as-blocks (or SAN).

Why bother converting blocks to NFS? After all, I just argued that NFS and iSCSI are almost identical for block traffic, so why not just use iSCSI if you want Ethernet, and FC-SAN if you prefer Fibre Channel?

Those extra NFS operations link to filesystem capabilities that are valuable for managing large numbers of containers. If you only had one LUN, or just a handful, it probably wouldn’t make any difference, but if you have hundreds or thousands of LUNs, it’s very convenient to give them meaningful names, group them into directories, and back them up with filesystem tools designed to handle thousands (or even millions) of separate objects.

NFS also lets you virtualize the data path. NFS provides an abstraction of the path from Application Server to Storage that customers really like and leverage. A whole industry of Fabric virtualization products is coming to market for Fibre Channel (think IBM SVC) to solve a problem already solved by NFS and TCP/IP. Path virtualization is why NFS does so well in grids. It’s the key to simple application mobility, and enables all kinds of resilience techniques.

To be clear, I’m not arguing that NFS is always the best solution for traditional block-based applications. We have many happy customers running Fibre Channel SAN and iSCSI, and we are investing aggressively in those protocols. My point is that many people simply reject NFS, mostly based on old data and misunderstandings. NFS definitely deserves consideration. Sometimes – not always – NFS is the best solution for mission critical enterprise applications.