NetApp - Dave's Blog

Insanity Is What Divides CIOs And CTOs

I got this lesson from a customer:

The difference between a CIO and a CTO is a certain level of insanity. A CIO would never take a working infrastructure, turn it inside out, and redesign it. But a CTO will.

Another customer I talked with – a CTO – described his job this way:

There are five stages to technology adoption: monitor, evaluate, standardize, proliferate, and retire. I’m responsible for all but the fourth. It’s not my job to proliferate technology through our infrastructure, but it is my job to figure out when to proliferate, exactly what to proliferate, and – eventually – when to stop proliferating.

I asked for details about these different stages. He described monitor as a casual awareness of a technology, maybe from reading occasional articles or white papers, and keeping an eye on it to determine when to investigate more deeply. Evaluate is when you seriously consider whether a technology is valuable enough and mature enough for you to deploy. To make a technology manageable in a large enterprise environment, you standardize on a particular set of configurations and management processes. At that point, you can proliferate the technology broadly, until a replacement technology is ready, at which point you retire this one.

I’ve noticed that the CTO title is used very differently, depending on the company.

In tech startups, the CTO is a smart development engineer, often a founder, who has the technology vision for the company’s product. These technology-CTOs are deeply involved in technical details – the bits, bytes, atoms and electrons of the new technology.

In very large corporations, the CTO is often a person who reports to the CIO, and whose job is to define the technology vision for the company’s IT infrastructure. These IT-CTOs generally care more about the big picture – how all the different technologies fit together to solve a business problem – than they do about the bits and bytes.

Companies that sell enterprise IT products often have one or more CTOs who work closely with customers. These customer-facing-CTOs work especially closely with IT-CTOs, so they tend to think more about the big picture than about the bits and bytes.

Using the same title for different jobs can cause confusion. Sometimes when a customer asks to “talk with your CTO”, they want the nitty-gritty details about product technology, in which case the technology-CTO is the right person. But sometimes they want advice on enterprise data center issues – on large-scale deployments and optimal procedures and best practices – in which case a technology-CTO may be a bad fit, and a customer-facing-CTO would be better.

My opening quote was about CTOs who report to CIOs, but I recommend against asking for “the insane one” to distinguish between the different types.

To NetApp Employees and Customers on Sun’s Lawsuit

[Note: This is an e-mail that I sent internally to our employees, with the expectation that they might also share it with customers. Some of it repeats previous posts, but other parts are different. In the spirit of openness, I decided to post here as well.]

To: everyone-at-netapp
Subject: Sun's Lawsuit Against NetApp

This morning, Sun filed suit seeking a “permanent injunction against NetApp” to remove almost all of our products from the market place. That’s some pretty scary language! It seems designed to make NetApp employees wonder, Do I still have a job? And customers to wonder, Is it safe to buy NetApp products?

I’d like to reassure you. Your job is safe. Our products are all still for sale.

Can you ever remember a Fortune 1000 company being shut down by patents? It just doesn’t happen! Even for the RIM/Blackberry case, which is the closest I can think of to a big company being shut down, it took years and years to get to that point, and was still averted in the end. I think it’s safe to say the odds of Sun fulfilling their threat are near zero.

If you are a customer, you can be confident buying NetApp products.

If you are an employee, just keep doing your job! Even if your job is to partner with Sun, keep doing your job. Here’s a ironic story. When James and I received the IEEE Storage Systems Award for our work in WAFL and appliances “which has revolutionized storage”, it was a Sun employee who organized the session where the award was presented. He was friendly, we were friendly, and we didn’t talk about the lawsuit. You can do it too. The first minute or two might feel odd, but then you’ll get over it. We have many joint customers to take care of.

If you are a salesman with concerned customers, feel free to show them this note, and maybe also point them at a couple of my blog posts (here and here). But don’t waste too much time before getting back to how we can work with them to solve their storage and data management problems.

I am frustrated with Sun and with Jonathan, as I describe in this blog entry. We have tried to be very open, detailed and specific about how Sun is infringing our intellectual property. We’ve tried to set a higher standard in how companies conduct patent litigation. It’s frustrating that Sun would just do a two-barreled blast, threatening to shut down our company. Frustrating and silly, to be honest, because it’s just so unlikely for a patent case to shut down a major corporation.

The other thing that’s frustrating is the way Jonathan wraps himself in the open source flag. We aren’t against open source, and we aren’t even against non-commercial use of ZFS. The number one rule of open source is that you should only give away stuff that belongs to you. That is what this suit is about, and everything else is just fluff.

Sun Sues NetApp: Says “You Cannot Unfree What Is Free”

Sun is seeking a “permanent injunction against NetApp” to remove almost all of our products from the market place. This is exactly the sort of broad but vague threat that gets people so frustrated with patent litigation. (The timing is no surprise: their deadline for responding to our complaint was this Friday.)

I have tried very hard in my blog to be unusually open – very detailed and specific – about how Sun is infringing our intellectual property. I’ve been trying to set a higher standard in how companies conduct patent litigation, and I’m disappointed in Jonathan for not doing better than this. This sounds like Sun’s broad threats when they sued Azul, but in the end, Sun didn’t put Azul out of business or even stop them from shipping products. I’m quite confident that two years from now – or however long it takes this suit to reach court – NetApp will be doing just fine. (For details on how this whole mess started, see my blog post, Jonathan’s response, and my response to him. I won’t revisit those arguments, so if you have comments about who is evil or not, please put them in those posts.)

But from a philosophical perspective, I found one part of Jonathan’s post especially interesting:

NetApp’s objectives were clear - they'd like us to unfree ZFS, to retract it from the free software community. Which reflects a common misconception among proprietary companies - that you can unfree, free. You cannot.

Jonathan seems to be arguing that once something has been put into open source, it is beyond the law. I disagree completely! To get us away from the details of Sun and NetApp’s particular case, let me make an analogy.

Suppose that I steal and then open-source Jonathan’s patented recipe for chocolate chip cookies. The recipe will probably live forever in the web. There is no getting those bytes back, and if it’s a good recipe, there is no stopping individuals from baking those cookies.

On the other hand, if I start a company to sell Jonathan’s Patented Cookies™, then it’s perfectly reasonable for him to ask me to stop. (Let’s just assume, for the sake of the analogy, that he’s got a valid patent.) Or if Nabisco starts selling Jonathan’s Patented Cookies™, it’s perfectly reasonable for Jonathan to ask them to stop. It isn’t a question of trying to unfree what’s free, or retracting the recipe from the free recipe community. It’s a question of whether corporations must obey the law.

Jonathan’s claim that “you cannot unfree what is free” sets a very dangerous precedent. It says that you can steal anything, as long as you open source it afterwards. That can’t be right! I do understand that many open source proponents argue there should be no legal protection at all for information. “Information wants to be free.” But even if Jonathan believes that, he ought to wait until the law changes before taking Sun down that path.

One of the most important rules of open source is that you must only give away things that belong to you. If protected information does leak into open source, it will probably live forever in the web, but that isn't the issue. To me, the issue is that large corporations should stop making a profit on protected information that doesn't belong to them. That's what we're asking here.

Sun Patent Team Demanded $36 Million From NetApp

Part of Jonathan’s response to our lawsuit stunned me. He simply denied our account: “First, Sun did not approach NetApps about licensing any of Sun's patents and never filed complaints against NetApps or demanded anything.” (Our name is actually NetApp, without the ‘s’, but I didn’t want to edit his quote.) 

He says Sun never “demanded anything”, but here is an e-mail from Sun’s lawyers doing exactly that. Legal language can be tricky to understand, but the translation of Sun’s letter is “You infringe our patents, so pay us $36 million.”

The part about “mapping claims onto NetApp products” is the legal way of saying “you infringe our patents”. The statement that NetApp apparently believes “the best defense is a good offense” indicates that we had something to defend against. Indeed. The phrase, “over a year and a half ago,” shows that the attack had been going on for a long time, both from StorageTek prior to the acquisition, and continuing at Sun long after the acquisition. 

It’s true that Sun lowered its price as the implications of our WAFL patents began to sink in, but it amazes me that Jonathan would simply deny that Sun ever demanded anything, and would imply (in the name of his URL) that NetApp is somehow the patent troll here.

[Note: Sun has now countersued. See Jonathan’s blog post announcing that, and my response.]

Litigoperation: Litigating While Still Cooperating (was “NetApp Sues Sun”)

Since NetApp sued Sun, our employees have been curious how this lawsuit will affect the way they do their jobs. Here is my message to them:

If your job is to cooperate with people from Sun, then keep on cooperating with them! Be nice to them. Aside from the small team whose job is to work on this lawsuit, I hope that employees can mostly ignore it.

People have asked how I can take this attitude. A lawsuit does mean that we have an important disagreement, but it does not mean we must be “at war”. Even though we disagree, Sun and NetApp can keep working together as major IT vendors to effectively support our shared customers.

Ray Noorda, once the CEO of Novell, coined the term coopetition to describe the relationship between companies that compete, but also cooperate. A great example is NetApp’s relationship with EMC. We compete with their storage products, but we partner very effectively with VMware, which EMC owns. (Diane Greene, the CEO of VMware, even recorded a video for our annual sales conference earlier this year.) We do compete with Sun, but many, many Sun servers use NetApp storage, and we have cooperated in areas like fixing customer problems and improving NFS. I remember the early days of NFS when Sun sponsored the Connectathon conference. Engineers from all the competing computer companies would come together to test their implementations of NFS. (I always thought that management would go crazy if they saw how closely the engineers were working together to find each other’s problems and help fix each other’s bugs.)

In this same spirit, companies that are litigating can still cooperate. (Litigoperation. You heard it here first.)

Part of Jonathan’s response to our lawsuit stunned me. He simply denied our account: “First, Sun did not approach NetApps about licensing any of Sun's patents and never filed complaints against NetApps or demanded anything.” Our name is really NetApp, without the ‘s’, but I didn’t want to edit the quote.

Here is an e-mail from Sun’s lawyers that refers to “claim charts mapping StorageTek/Sun patent claims onto NetApp products.” It says they had originally sent these to us “over a year and a half ago.” Do the math, and you can see that this started in the StorageTek days, but Sun continued it after the acquisition. Their proposed settlement was “a disk storage cross license in exchange for payment from NetApp to Sun of $36.548M.” At other times, Sun said it was entitled to hundreds of millions of dollars, so we are talking about very large sums of money.

On the other hand, it was no surprise for Jonathan to focus on open source, but I just don’t get it. For me, one of the most important rules of open source is that you must only give away things that belong to you. This is a lawsuit about defending our intellectual property against Sun, and against what Sun is paying their own engineers to do. It does complicate things that Sun has open sourced ZFS, but this is not a lawsuit about open source.

NetApp Sues Sun for ZFS Patent Infringement

This morning, NetApp filed an IP (intellectual property) lawsuit against Sun. It has two parts. The first is a “declaratory judgment”, asking the court to decide whether we infringe a set of patents that Sun claims we do. The second says that Sun infringes several of our patents with its ZFS technology.

How did we get here?

Like many large technology companies, Sun has been using its patent portfolio as a profit center. About 18 months ago, Sun’s lawyers contacted NetApp with a list of patents they say we infringe, and requested that we pay them lots of money. We responded in two ways. First, we closely examined their list of patents. Second, we identified the patents in our portfolio that we believe Sun infringes.

With respect to Sun’s patent claims, our lawsuit explains that we do not infringe, and – in fact – that they are not even valid. As a result, we don’t think we should be paying Sun millions of dollars.

On the flip side, our suit points out that Sun’s ZFS appears to infringe several of NetApp’s WAFL patents. It looks like ZFS was a conscious reimplementation of our WAFL filesystem, with little regard to intellectual property rights. Here’s what creators of ZFS have to say: “The file system that has come closest to our design principles, other than ZFS itself, is WAFL … the first commercial file system to use the copy-on-write tree of blocks approach to file system consistency.” One of the first patents I filed at NetApp describes this “copy-on-write tree of blocks” technique in detail.

We filed suit against Sun because after we pointed out the WAFL patents, their lawyers stopped getting back to us. The first part of our suit is a declaratory judgment. It’s complicated, but the basic idea is that Sun claims we infringe their patents, so we are requesting a trial to show that’s not true. In essence, a declaratory judgment calls their bluff. It allows us to force a legal conclusion, rather than leaving this threat hanging over our heads. The second part is a complaint against Sun for infringing several WAFL patents with ZFS.

As we file this case, I’m painfully aware of the bad feelings that many people have about IP lawsuits. Business people sometimes view them as the last gasp of a dying company, while some technical folks view them as evil under any circumstances. In this case, I doubt that business people will be too concerned, given our growth rate, profitability, and the fact that we are responding to Sun’s claims against us. I expect skeptical technical people to be harder to comfort.

This case is especially sensitive, because Sun has released ZFS as open source. It is admirable to contribute to open source. I have done it personally, although it was a long time ago that I was writing code, and NetApp has also contributed as a company. But it doesn’t help the open source movement to give away code that is encumbered with someone else’s patent rights. The sooner we determine the true status of ZFS, the better it will be for everyone. NetApp certainly doesn’t believe that we can somehow erase every copy of ZFS that has been downloaded. (Impossible!) This lawsuit isn’t about downloads for personal or non-commercial use; it is about what Sun is doing.

We could have a long debate about the merits of the patent system. (I expressed my own qualms here.) But like it or not, it’s the law of the land. Here’s an analogy. Suppose you own a pro football team, and you really believe that touch football would be safer and more humane than tackle. You can lobby all you want to try to change rules, but until you are successful, I recommend that your team keep wearing pads and helmets. NetApp is participating in attempts to reform the patent system, but meanwhile we will play by the rules as they exist.

In closing, let me say that the legal system is a method of resolving disputes between companies, but it does not mean we are “at war”. During this process, we will continue to support all Sun products, including ZFS filesystems that use NetApp storage. It is not in anyone’s interest – Sun’s or ours – to leave a bunch of customers in the lurch. I hope that Sun will respond in kind.

[Note: Sun has now countersued. See Jonathan’s blog post announcing that, and my response.]

Non-technical people can stop reading now.

It is important to me that technical readers not confuse NetApp with SCO, so in our lawsuit, we provided a starting point for people who want to dig deeper. This is not an exhaustive analysis of our case. We simply highlight one particular patent and one particular aspect of ZFS to help people see that this case of infringement is real.

Here’s how the ZFS designers describe filesystem consistency:

The best way to avoid file system corruption due to system panic or power loss is to keep the data on the disk self-consistent at all times, as WAFL does. To do so, the file system needs a simple way to transition from one consistent on-disk state to another without any window of time when the system could crash and leave the on-disk data in an inconsistent state.

In the ZFS paper, search for uberblock, and compare its role in filesystem consistency with the role of the root inode and file system information structure in our patent 5,819,292. Read claim 4 and its descendents, which describe our tree-of-blocks consistency technique. Claim 8 and its descendents describe efficient snapshot creation based on the tree-of-blocks. Some more useful references are here (see pages 7 and 8), here, and here.

We hope that this level of openness will help raise the bar on how people pursue intellectual property suits.

Let me insert the usual legal disclaimer: I’ve quoted our complaint, but beyond that, I won’t engage in any public comment on the technical details of our case while it is pending.

More Links

• Sun's CEO discusses the lawsuit.
• Sun Patent Team Demanded $36 Million From NetApp

What Killed The Storage Service Providers?

Storage Service Providers (SSPs) were common back in the dot-com days. The idea was that instead of buying your own disk drives, you could buy disk space from an SSP and access it over the internet, or over some kind of metropolitan area network.

Many people believe that the SSPs died out because corporations want to keep their data nearby and will never allow it to be stored offsite. (I followed Jon Rath’s blog to two articles about the possible resurgence of SSPs, here and here.)

I don’t think offsite data was the main problem. I believe that SSPs died because their business model evaporated out from under them.

To understand the original SSP business model, you have to think back to the crazy days of the dot-com boom, when the SSPs were popular. Companies were growing so fast that they just couldn’t keep up with the hiring. Money was plentiful, but IT staff was scarce. Everyone believed that there was a limited time to make a land grab, so they couldn’t afford to slow down.

In that environment, the SSP business model was: We charge more, but you’ll pay it anyway, because you can’t hire IT people yourself. Part of the reason that startups couldn’t hire IT staff themselves was that SSPs and other IT-centric startups were luring them in with pre-IPO stock options. Why be in a supporting role at a product-centric startup when you could run the show at an IT-centric startup?

The model fell apart after the dot-com crash, because dollars became scarce, potential customers no longer needed to grow fast, and there were plenty of unemployed IT people around to hire. “We charge more so you can grow fast” lost its appeal.

In response, SSPs tried to change their business model. They claimed, “We can save you money because of economies of scale.” The problem is, they never actually demonstrated that. Helping your customers grow very quickly, but at a high cost, is very different from helping your customers save money. The SSPs just weren’t able to make the change.

Part of the cost problem is that the bandwidth required to feed a big disk array is expensive. It’s true that bandwidth keeps getting cheaper, but then again, disk drives keep getting bigger. Some SSPs focused on outsourcing storage within co-location facilities where customers would also host their servers. This solved the bandwidth problem, but left SSPs vulnerable when the co-lo model collapsed. The final blow was that many of the SSPs’ biggest customers were the failing dot-com companies.

In summary, their business model evaporated, their technique for dealing with bandwidth cost collapsed, and their best customers went bankrupt. (“Other than that, Mrs. Lincoln, how was the play?”) So it’s true that the SSPs flamed out, but that doesn’t prove that the storage outsourcing model can never work.

I don’t buy the part about corporations never allowing data offsite. Ask people at many corporations where their voice mail is stored; they have no idea, and it doesn’t worry them. (Answer: Offsite in a phone company data center.) Ask them where super-important legal documents are? (Answer: Offsite in an Iron Mountain warehouse.)

Of course, you only let critical data go offsite if you really trust the provider, which makes this a difficult business for startups.

One way to get around the bandwidth problem is to outsource the whole application, instead of just the storage. There is usually less bandwidth between the application and the user’s eyeball than there is between the application and the storage. This is exactly what Oracle does with its On Demand business, which has been quite successful, and likewise SAP with their Managed Services offering. Another approach is to store less critical data, like photos at Shutterfly, or personal e-mail at Yahoo! and Google. For disaster recovery copies, the data must be offsite anyway, so it takes just as much bandwidth to mirror to your own remote storage as it does to mirror to outsourced storage. Iron Mountain is getting into this business.

I doubt it will ever be cost effective (cheap enough bandwidth, fast enough bandwidth) for SSPs to outsource all disk drives, and I’m sure there is some data that big corporations will want to keep close to home, but there are already many situations where it makes sense to outsource the management of important corporate data. I predict this trend will keep growing.

 

Think of a Will as a Program You Can Only Test By Dying

Both legal documents and computer programs are written in a language that looks somewhat like English, but isn’t. You may recognize many words, but you are sadly mistaken if you think that fluency in English translates into LEGAL or COBOL.

The smallest “useful” computer program simply prints “Hello World!”. It does almost nothing, so most of the program is overhead. In C, it takes 53 characters of program to print 12 bytes of text – an overhead factor of 4.4.

I’ve been reading LEGAL this week, because some friends of mine are writing their will. I agreed to be the trustee in case both parents die while the kids are still young. It occurred to me that the “hello world” of wills is this:

Leave everything to my spouse. If s/he is dead, then split it evenly among my kids.

This is pretty much what my friends’ will said, but to express these 83 bytes of idea took 18,700 bytes of LEGAL, for an overhead factor of 225. That is, LEGAL is 51 times less efficient than C.

Why is LEGAL such a shitty language?

For starters, it doesn’t use modern techniques like subroutines or standard libraries. Consider this phrase from my friends’ will:

any and all household goods, furniture, furnishings, utensils and supplies, paintings, pictures, glass, silver, papers, rugs, china, books, linens, objects of art and other, similar articles of tangible personal property which I may own at the time of my death, any wearing apparel, jewelry and personal effects which I may then own and any interest which I may then possess in any automobiles

Wouldn’t it be easier to define ALL_MY_STUFF as a macro in a standard library? Instead, lawyers cut-and-paste big chunks of text from other legal documents. Part of the problem is that LEGAL was invented thousands of years ago, long before there were computers to help with mundane issues, like formatting and making sure that parenthesis match up properly. LEGAL would be so much easier to read if it used indentation and parens to make the structure clear, rather than subtle rules of comma placement.

In their defense, lawyers are legitimately afraid to make changes, because there is no way to debug or test a legal document. Think of a will as a program that you can only test by dying. If the program is wrong, your heirs could lose their inheritance, or they could be tied up for years in court.

When writing in a language that is impossible to test, it is only prudent to make the smallest change possible. When you see a complex, bizarre phrase in a legal document, there’s a good chance that it was added after somebody lost a legal battle. “Oops – we listed glass, china, silver and linens but we forgot to specify utensils. Also, let’s add ‘similar articles of tangible personal property’ just in case we missed anything else.” Imagine the outrage of the children cheated of their utensils! Once a phrase has been tested in court, no lawyer dare change it. As a result, the ancient heritage of LEGAL shows through. Chunks of text may have been copied thousands of times, over hundreds of years. You can even see bits of Latin sprinkled throughout.

Of course, there is one additional reason that legal documents are so long: Many lawyers are paid by the hour.

Extreme and Surprising Events Happen More Often Than You Think

I just read The Black Swan, by Nassim Taleb. To summarize the whole book in 10 words: Extreme and surprising events happen more often than you think.

Many psychology studies have shown that humans are inherently bad at dealing with improbable events. (See my recent blog on Shark Island.) Taleb talks about this, but to a mathematically minded person like me, his real point is much scarier. He argues that bell curves and standard deviations—tools that number people use to understand probability—often fail in the real world. With Gaussian bell curves, the probability of extreme events goes down exponentially as you get further from the average. But in many real-world situations, the probability goes down much slower for extreme events. The tail is fatter. If you trust standard statistics, you could end up in big trouble.

Taleb uses concrete examples to build intuition. Peoples' height is a normal bell curve, but wealth is not. Suppose you randomly select 10 people out of the entire world, and check their height. The average will be six feet, or whatever it is. Now take the world's tallest person and add him to the mix. The average only goes up three inches. Increase your random sample to a hundred, and throwing in the tallest person changes things by less than an inch.

Now try the same thing with wealth. Take ten random people worldwide, and their average income is $10,000 or whatever—remember I said worldwide. But add Bill Gates into the mix, and the average goes up many thousand-fold. Even if you increase the sample size to 1000, adding Bill makes the average several hundred times higher—from ten thousand dollars to millions.

What if height were distributed the way wealth is? Six feet might be the most common height, but there would be many 10 feet people wandering around, and even some hundred footers. Mathematically speaking, this is the difference between a normal bell curve and a power curve, but an example can show the difference clearly even if you don't care about the math. Consider a town with a million people, and compare how many tall ones there are with a bell curve versus a power curve. (Check the endnote for math-nerd details.)

Count of all people People over 6' People over 6'3" People over 6'6" People over 7' People over 8' People over 10' People over 100'

Bell Curve 1,000,000 500,000 158,655 22,750 32 0 0 0

Power Curve 1,000,000 500,000 158,655 81,067 34,790 13,143 4,584 27

At first the two curves look similar: half the people are over six feet, 158,665 are over six-three—exactly the same. But at extreme heights, things are so different. With a bell curve, you will never see a ten-foot person. There is some theoretical probability that it might happen, but it's so rare that you can count on never seeing it in your life. With a power curve, there are not only 4,000 ten-footers, but 27 one-hundred-footers! For the planet as a whole, the power curve predicts several dozen people over ten thousand feet tall.

With power curves, extreme and surprising events happen more often than you think.

It is obviously very important when you are managing probability (or risk), to understand which curve applies. How would you design hospital beds, for your town of a million, if you knew that hundreds of citizens were over thirty feet tall? What if you thought height was a bell curve, and built eight-foot beds, but it turned out later to be a power curve?

Power curves are very common where big guys can grow at the expense of little guys. Tall people can't take height from short people, but large companies (e.g. Microsoft) can take business from small ones. Popular websites (e.g. Google) take web-hits from small ones. As a result, power curves are very common in business statistics. Power curves can also result when there are many interactions between elements. I suspect that failure events in interconnected infrastructures, like the nation's electric grid or enterprise data centers, follow power curve rules.

Taleb mostly worries about the implications for investors, but I see lessons for companies as well. Don't trust your plans. You still have to make plans, but they will change more than you think. Don't trust statistics on small or medium sized samples unless you know it's a bell curve. If you suspect a sample might be a power curve, don't trust bell curve statistics at all. Expect surprises.

[Math-nerd note: In my example, the average for the bell curve is 6 feet, and the standard deviation is 3 inches. For the power curve, I set 6 feet as the starting point in a half million person population, made 6'3" the first doubling point, and selected the exponent to make the probability match the bell curve at 6'3" for easy comparison. The exponent was 1.656. I ignored the population below 6 feet because power curves don't handle the left side of a peaked distribution well. You obviously don't have bazillions of people six inches tall, or people negative a thousand feet tall, so you have to cut off the left side somehow. I'm only half-good with math, so I probably made some mistakes, but I think it's accurate enough to make the point.]

How the Patent System Really Works

Many people think the patent system protects the little guy who has a great idea and is trying to get started. Perhaps it was supposed to do that, but today – at least in the computer industry – it has evolved into a taxation system that lets companies with more intellectual property impose a tax on companies with less.

Here's what it feels like if you are a small company. Lawyers from a really big computer company – let's call it BigCo – show up and say, "We think you are violating these five patents. You should license our patent portfolio." The patents don't impress you. You write a document criticizing each patent – either it doesn't apply to your design, or you found some prior art that invalidates it. Hard work, but worth it to avoid that licensing fee.

A week later, BigCo's lawyers show up again and say, "Here are five more patents we think you are violating. Did you know we have sixty-seven thousand patents? We really think you should license our patent portfolio." You can spend the rest of your life rebutting BigCo patents, or you can pay. Plus, you have a nagging fear that BigCo's patents may cover something you do, given how many they have. (I could discuss whether too many patents are being issued, but I am describing how the patent system does work, not how it should work.)

BigCo wants your money, but also access to your patents, so the cross-licensing negotiation begins. You try to identify patents that cover their products, and they try to identify patents that cover yours. In the end, you balance how much of your revenue their patents cover, and how much of their revenue your patents cover. The little guy usually loses. Ironically, patents on your super-special technology often don't cover any of BigCo's products, so "miscellaneous" patents may be more important than the ones you thought were so valuable. BigCo has so many patents that it's hard to examine them all, so they will do a statistical analysis showing how many patents they have that might apply to you. Why not save the lawyers fees and just assume that some percentage does apply? (NetApp was in a cross-licensing negotiation with a large semiconductor company, and since we don't design or manufacture chips, we got them to remove those patents from their analysis.)

Fortunately for NetApp, our CTO Steve Kleiman explained the system to us early on, based on his experience at Sun. We invested a lot in our patent portfolio, and it paid off when the negotiations began. (I hope this note will help other companies the same way.) The big reason to file patents is not so much to defend your ideas as to save money in cross-license negotiations.

Don't patents ever protect your good ideas?! In theory, they should, but in practice it doesn't usually work that way. Suppose your small company wants to protect its ideas against a big company. Filing suit will accelerate the cross-licensing negotiation, and you'll probably end up paying. Better to let sleeping dogs lie. Patent battles between small companies work poorly because they are so expensive and take so long. Better to fight it out in the marketplace. Big companies suing small companies have a harder time than you'd imagine, because the courts recognize that an incorrect decision, especially against a startup, can cause irreversible damage. Courts are reluctant to impose injunctions, even if the patents really do apply. In the case of two big companies, both almost always violate each other's patents, so they end up cross licensing. (I'm not saying that patents never help to protect good ideas. If someone steals your patented idea, it's perfectly reasonable to go after them. I'm just saying that it seldom works out as well as you might hope.)

I know that some people are so frustrated with the patent system that they want nothing to do with it. The problem is, there's no good way to opt-out. If you don't file any patents, then the inevitable cross-licensing negotiations will be much more painful. What's worse, if you keep your ideas secret, instead of patenting them, then someone else might patent them and file suit against you for infringement. (Sounds crazy, but – in some countries at least – that's how the system works.)

In the end, my conclusion is that the patent system really is a lot like the tax system. You may not like it, but you are better off participating. Small companies should patent their "secret sauce", to protect against copycats, but more general patents will actually be most useful in cross-licensing negotiations. Larger companies have invested so much in their patent portfolio – given how the system works they really have no alternative – it only makes sense for them to collect the cross-licensing "tax" like everyone else. It helps pay for the cost of doing the patents.

My goal here is not to defend the patent system. It helps big guys more than little guys, and I don't think it encourages innovation very well. My goal is to describe as best I can how it seems to work. You may not like a system, but if you understand it you'll do better than if you don't.

My Philosophy of Language (Why Google is My Word Usage Guide)

Linguists have two rival views of human language: prescriptive and descriptive.

 

Descriptive says: Never mind what's "right" or "wrong", here is how people actually speak. Language is defined by people, so let's describe that.

Prescriptive says: Here is how people should speak. People who speak differently are simply wrong. They are damaging the language and should be corrected.

Prescriptive is fine for grammar school teachers, but in my opinion, if you want to communicate effectively in the real world of grownups, you should figure out how other people use words and use them the same way. If people are hopelessly confused about what a word means, then it's best to avoid it unless you are prepared to lead a campaign to educate the masses. (People who read my blog on whether iSCSI is a form of NAS (here and here) won't be surprised to hear that I am a descriptivist.)

I love the American Heritage Dictionary (see my book list) because it is descriptive and often backs me up when people try to correct my grammar. For instance, I use data as a singular word, but some presciptivists insist that it is the plural of the Latin word datum. American Heritage, on the other hand, reports that 77% of their usage panel now accept data as a singular word. Language changes. If it didn't, we'd all still be speaking Latin, Sanskrit, or Proto-Indo-European. (Proto-Indo-European is the ancestor to both Latin and Sanskrit, and the American Heritage has a dictionary of Proto-Indo-European word roots. My favorite is deru or drue, which is the root of words like tree, true, durable, druid, and even tryst - presumably because trysts happen under trees.)

Google is great for probing today's usage. In my recent blog on data deduplication, I had to decide whether to use dedup, de-dup, dedupe, or de-dupe. I checked with Google and got these results:

dedup:	35,400 matches
de-dup:	75,200 matches
dedupe:	116,000 matches
de-dupe:	789,000 matches

De-dupe is the winner by a landslide, so that's what I used. To get accurate results, you have to use the exact phrase feature of advance search, because otherwise Google treats the dash as a space.

You don't always have to go with the majority. Perhaps you want to invent a new phrase, or make a point. For instance, NetApp officially refers to iSCSI as IP SAN, even though Google shows iSCSI winning by 4,940,000 to 637,000. We want to make the point that people can use iSCSI in place of Fibre Channel SAN for surprisingly high-end applications. On the other hand, we always use iSCSI nearby so that people will know what we are talking about.

Sometimes you want a unique name. Last week my wife and I had a baby girl. Her name is Mira Hitz, which has no matches now but will as soon as I post this!

Tom Mendoza's Lessons on Public Speaking (Feelings, Action, Data)

This week I worked on a big presentation, so I thought I’d share some public speaking techniques I learned from Tom Mendoza.

Many people – especially technical folks like me – focus way too much on the data that they want to present to their audience. Tom always asks me how I want the audience to feel after my talk. Before I learned his technique, my answer would be something like, “I want them to feel that they understand the design goals and architectural details of ?”

At this point Tom would interrupt: “Feelings are one word. Angry. Proud. You know – emotions.” You can have a small phrase describing what the feeling is about. “Disappointed in our performance.” “Proud of our new release.”

Next, Tom would ask what I wanted people to do differently after my presentation. He argued: “If you don’t want them to do anything different, why are you wasting your time talking with them?” If you’ve reached an important milestone in a project, like a key code-freeze date, you might want people to feel proud of what they’ve accomplished so far, but to keep working hard until they’re done. If a project is way off track, the feeling of disappointment in the progress so far could motivate people to accept and engage a new approach. If a competitor is beating you, perhaps anger will help drive action. Part of the trick is to choose actions and emotions that naturally reinforce each other.

When you are clear on the feelings and actions that you hope to inspire with your presentation, then, and only then, should you start to worry about the content – about what data to share to inspire those feelings. You can say, “I want you to feel excited about what you did,” but it might work better to show the sales figures or benchmark results that prove people did a good job. Then the audience will naturally be excited. Or if the results are bad, naturally disappointed.

When I start with feelings and actions, it makes my presentations much better. Good content is important, but it’s only a tool. Feelings and actions are the goal.

At first, I struggled with Tom’s method because I wanted to share too much information. Now, I’ve learned to appreciate the elegance of finding the smallest amount of data required to drive the feelings and actions I want. For exhaustive detail, a web site or white paper is a much better communication tool. Sometimes go read the white paper is the action I want to inspire. Even in a classroom setting, lectures don’t replace textbooks.

I used to worry that removing details would result in over-simplified presentations. Now I believe that my job as a public speaker is to over-simplify.

But I still try not to over-over-simplify.

Does Helping Customers Use Less Disk Hurt NetApp's Business?

We got an interesting question in the Q&A session at our Analyst Day conference a couple of weeks ago:

NetApp focuses so much on technologies to save disk space—like Thin Provisioning and Clones—could that hurt your business by reducing the amount of disk that customers need to buy?

I don't think so. Here are three reasons.

First, we have an opportunity to gain share. NetApp has less than 10% market share for SAN, NAS and iSCSI, so if we can help customers save money by using less disk, then we have a great opportunity to grow. (We've roughly tripled our revenue in three years, so it seems to be working.)

Second, storage purchases are elastic. When storage gets cheaper, customers just seem to buy more. Remember, disk prices have been dropping by 40% per year forever, yet the storage industry keeps growing.

Third, we charge for the software that helps people use less storage. I would much rather make money by selling software that helps customers rather than by marking up commodity disk drives. (To put it another way, would you rather be Michael Dell or Bill Gates?)

These are the reasons that we gave on stage at the analysts meeting, but there is also a deeper, more philosophical reason. I strongly believe that innovating to help customers is always the right thing for our business. If there is a clever way to help customers, someone is going to do it, and I want it to be NetApp. One of the reasons that we've been taking share from EMC and HP is that we have been very creative about how to help customers. If we hold back, it will make us vulnerable to the next generation of emerging competitors.

Summary: If you don't eat your own lunch, somebody else will.

Tom Mendoza's "Three Step Management Algorithm" for Solving Any Problem

I became a manager in November of 1999. Before that I'd been a programmer, an architect, a visionary and an evangelist, but I'd never managed people. After that, I became VP of Engineering at NetApp with 250 people reporting to me. We hired another 500 engineers in the next couple years, so I had to learn fast.

There were always lots of problems to deal with, and I sometimes went to Tom Mendoza for advice, since he'd been managing for years. He always asked about the people working on the problem. Who was involved? What other things were they working on? What were they good at?

It was clear that Tom thought about things very differently than me. I would dig in on the problem itself. I'd learn about the details, explore the options, and worry about the right answer. Tom didn't focus on the problem; he focused on the people whose job was to solve the problem. After talking with Tom, I seldom understood the problem any better, but I had lots of ideas about how to move forward in solving it. Occasionally I was the right person to dig in and come up with a solution, but especially as the organization grew, it became obvious to me that Tom's approach was much more powerful and scalable.

Since I'm an engineer at heart, I reverse-engineered the process that Tom seemed to be using as he questioned me. I concluded that Tom applied a simple three-step algorithm to every problem:

1. Who owns the problem?
2. Do I trust them?
3. How do I find an owner I trust?

If you can't find an owner, that may be the problem right there. Skip to step (3). Sometimes it's obvious what person or group owns the problem, and they just need to be reminded that it's theirs and that you are watching.

When you find the owner, the next step is to figure out whether you trust them. I don't mean trust in some abstract sense; I mean trust them to fix this particular problem. In the abstract, I trust Tom Mendoza completely. For a problem involving spreadsheets or programming languages, I don't trust him at all. Even if you trust someone's skills, they may be too busy to do more. Do you trust that they have the skills, the time and the passion to solve this problem?

If steps (1) and (2) fail, then you must find an owner you trust. Sometimes there's someone nearby who can take it on. Other times you may need to reassign someone or hire a new person. Sometimes the answer is "Do it yourself," but the larger the organization, the less often this will be true.

Tom's background is sales, and he knows absolutely nothing about programming, so imagine my surprise when I watched him apply his algorithm recursively. By following steps (1) and (2) he concluded that he needed to hire someone to own a particular problem. "Hire someone" became the new problem, and by following steps (1) and (2) he concluded that he wanted to use an external headhunter. "Find a headhunter" became the new problem, and—here we reach the root of the recursion—Tom identified the owner as the VP of Human Resources, who he trusted.

NetApp Future History from 2003

This is an unusual blog entry because instead of telling a short story in 700 words, I'm releasing a twenty-page strategy document that I wrote in late 2003.

I call it a "future history", because I wrote it in the past tense, as if it were a history of NetApp focusing on the period from 2003-2007. My goal was to describe as accurately as possible what we needed to accomplish over the next few years, and I thought that writing in the past tense, as if it had already happened, would help make the vision more real for people.

This paper started as a whine. I was confused about NetApp's vision and strategy, and I figured if I was confused, then many others probably were as well. Late 2003 was a confusing time. We were just reaching $1 billion in revenue for the second time. The first time had been about 3 years earlier, in 2001, at the end of the dot-com boom. And then the tech crash pummeled us back down to $800 million. It didn't feel like a 20% drop. We had been doubling annually for many years, so our whole planning process was focused on reaching $2 billion. Going from the $2 billion we expected down to $800 million was traumatic.

For the three years after the crash, we had a clear mission: Get back to where we had been—reach a billion again, with a reasonable profit margin. But when we succeeded, it wasn't clear where to go next. Before the crash, our goal had always been to double, but after a three-year struggle to return to a billion, doubling annually didn't seem reasonable. This is what triggered my whining, and I started asking lots of questions throughout Dan's staff. What could we hope to accomplish? Given the market situation, what did we need to accomplish?

This paper is the result of my questions, and we shared it with every employee at NetApp. Some people were concerned about being so open with our strategy, but I am convinced that a culture of openness and inclusiveness has helped fuel our success. Three years later, I'm proud to say that we have achieved most of the goals that we set. The paper set out a $2.5B goal for this fiscal year, which ends in April 2007, and most analysts predict that we will exceed that. The run-rate of our last quarter is well over that, so we have declared victory and moved on.

I am sharing this Future History publicly because I think it is an interesting case study for people to see how we put this vision together, and what we thought we were trying to accomplish. Please keep in mind that this is three-year-old strategy, so don't assume that everything in here came true! On the other hand, a surprising amount did, which shows the power of being clear about your vision as a company, and sharing it as broadly as possible with your employees.

NetApp Meets Monopoly and SimCity

Every year we get our top 100-200 people together for a three day Senior Leaders Offsite. The specific goals are different each year, but the general idea is to help people understand our high-level vision and strategy so that they can run their piece of the company more effectively.

Lately Dan’s staff has been working on our strategy to reach $6 billion and beyond, so this year’s goal was to help everyone understand the key success factors and pitfalls that we foresee over the next several years. We could have spent hours on stage with PowerPoint slides explaining our beliefs in gory detail, but we thought that letting people play a computerized business simulation of NetApp would give them a much more visceral understanding of the strategy. Think of it as NetApp meets Monopoly and SimCity.

Each team consisted of five people making the same sorts of decisions as our executive staff. Should they cut investment in new products to fund marketing? Should they increase the direct sales force or sign up more resellers? There were also confounding events like product delays and competition from disruptive startups. (We didn’t build the model ourselves – we worked with a company that specializes in business simulations.)

I didn’t get to play, since I helped design the game, but it was fascinating to watch and coach. I worked with one team that was trying to understand why their revenue had fallen behind, despite strong investments in new products. They had slashed their direct sales force, hoping to replace it with resellers, but they didn’t take into account the time it takes for new resellers to get up to speed. Because of the product investments, the few customers they did have were extremely satisfied. We could have PowerPointed till we were blue in the face about the tricky balance between direct and indirect channels, but nothing brings a lesson home like watching your market share plummet as a result of a bad decision.

Part of what made the simulation so interesting is that the teams weren’t competing against the computer, but against each other. There are no “right answers”, because what works best depends on what the other teams do. If you invest heavily in brand awareness, that might give you an advantage, unless everyone else does the same.

Early in the game, I noticed that engineers argued for more product investment, and sales people pushed for channel programs and marketing, but that changed as the game wore on. One sales VP told me, “My table didn’t have any engineers, so we completely underinvested in product development. Lots of sales people but nothing to sell.”

By the end, the discussions were much more balanced. Given that our goal was to help people understand the big picture, what made me happiest was hearing how much the arguments at all of the tables sounded like the arguments we have in Dan’s executive staff meetings.

NetApp Values

After my blog on company values, one reader asked for a link to NetApp's values statement. There you go.

When Dan joined NetApp, the first value he focused on - long before we had an explicit, written list - was Trust and Integrity. At the end of his first staff offsite meeting, Dan said, "I want everyone to rank our candor. You know each other better than I do. Did people say what they really believe? Did you? I won't ask you to explain your score, but I'm going to go around the room, and I want everyone to give a grade from one to five - five is good - on how candid you think we were with each other during this meeting."

NetApp had gotten to be a pretty political place before Dan joined, and that was something he wanted to quash. He didn't mind at all if people disagreed with each other - that is a healthy part of finding the best path forward - but he wanted us to do it in the open, to each other's faces.

We went around the room, and the average score was two, maybe two-and-a-half. Dan didn't beat us up, and he didn't ask for details; he just said, "I see we have some work to do. This is something that's important to me."

The value Create a Model Company represents our explicit choice to build a company that would have a long-term presence in the market. Many entrepreneurs plan to "flip" their startup, to invent some intellectual property and quickly sell it to a large corporation. There's nothing wrong with that, but you should be clear about what you are doing. If you are building a startup to flip, you would probably focus 100% on the technology, and on getting that first product out the door. Little need to spend time on corporate values or long-term relationships with customers.

I remember Tom Mendoza's reaction to the first draft of our values. Dan had convinced his staff to spend time at an offsite to discuss our values and write them down, and Tom said, "These are all so touchy-feely. Trust, Customers, Integrity, Teamwork - I certainly can't disagree with any of them - but where does it say that we spend our time doing stuff that matters, and not just sitting around feeling good about each other?"

That's when we added Go Beyond and Get Things Done! to the list. Tom always drives people to action and I love that our written values reflect his passion about this.

In the end, a company's true values are what employees actually believe and how they actually behave, not some list on a piece of paper. Written values can be destructive and lead to cynicism if they don't align with belief and behavior, but if they do, then they can help reinforce the values, especially with new employees.

Company Values are a Tool for Employees to Beat Up Management

Corporate culture is one of the first things Dan focused on when he joined NetApp in 1994. (Profitability was another.) He wanted to create a formal set of company values, so he asked everyone on his staff to hold “values meetings” to collect input.

James and I met with Engineering, and it didn’t go well at all. In fact, we absolutely pegged everyone’s Dilbert-o-meter. Lots of cynicism. At the time, Engineering was maybe 20 people, so we all fit into a medium-sized conference room. I remember a woman named Florence asking, “How will these values be used against us?” I found out later that she’d had a bad experience with values at a previous company, along with many others. I confess that company values made me uncomfortable as well. To me, values seem like something personal, not something that a company can tell people to have.

Dan got so much negative feedback that he postponed the project, but he didn’t give up. A year later, at another staff offsite, we spent several hours talking about values. Mostly they were what you’d expect in anybody’s corporate values – integrity and honesty and so on. At first this bothered me. What’s the point, if it’s just the same old stuff? But then I thought about it differently. If you started a new country and needed a constitution, you’d probably include freedom of speech and habeas corpus and a bunch of other stuff just like everybody else. The point isn’t to be original; the point is to capture what’s important to the culture you hope to have.

Telling people what their values should be still makes me uncomfortable, but I have gotten comfortable talking about NetApp’s values. Two things helped me.

The first is that I personally do believe the values we came up with. I can’t tell people what their values should be, but I can tell them what my values are. Not all of my values of course, like about religion or marriage or children, but the ones that apply in a work context. In essence, I’m saying: “These are things that we – as a management team – believe, and since you are an employee here, it seems only fair to let you know.”

Florence’s question haunted me: “How will these values be used against me?” The more I thought about it, the more I realized that values aren’t that useful for beating up employees. I mean, if managers want to beat up on employees they have so many tools at their disposal. They can give them bad reviews, no raises, bad assignments – even fire them if it comes to that. Realizing that values are a tool that employees can use to beat up management is the second thing that got me comfortable talking about them.

Here’s an example. Early on at NetApp, we were competing against Auspex (now dead). Auspex had promised the customer support for a new protocol within 12 months, but we knew we couldn’t deliver it in less than 18. The thing is, we knew enough about Auspex to know it would take them 18 months as well. We told the customer that, but they said we should focus on our promises and let Auspex focus on theirs. They made it clear that they would choose the vendor who could meet this requirement. We were having a meeting about what to do, and the question was whether we should also lie. The argument was that neither company could deliver it sooner than 18 months, so it wasn’t like this lie would hurt the customer any; they’d get the protocol in 18 months either way. In fact, we suspected that we could actually get it done faster than Auspex, so if we lied, we would actually be helping the customer, wouldn’t we?

We went on like this for a while until someone – I can’t remember who it was but it wasn’t the boss – said, “I can’t believe that are discussing whether or not to lie to a customer. Don’t we have Honesty as a value? How can we even be having this conversation?” I’m not proud to admit that we had the discussion, but I am proud to say that this comment shut it down immediately.

We were honest, and we lost the business. We were a small company and that deal really would have helped us. But here’s the ironic twist: Auspex never did deliver the new protocol – it’s not clear they ever intended to – and the customer was furious. Two years later, Auspex was out the door and they became one of our largest customers. Over ten years later, they still are.

I think that values are like that. Adhering to values can painful in the short run. Sometimes lying or cheating is the best way to make a quick buck. But if your goal is long-term success, then values are in your own self-interest. I can’t prove that “Cheaters never prosper”, but I do believe this: In the long run, cheaters tend not to prosper, especially if they need to keep dealing with the same people. Tom Mendoza once summed it up like this: “Life is a long game. People remember.”

I’m sure that NetApp doesn’t always live up to our values, but it is something that I aspire to, and it is something that I’m happy to be beat up about.

The Story of Grandma at the Bottom of the Dot-Com Crash

The dot-com crash was hard on a lot of people. At NetApp (NTAP), the stock dropped from a peak of about $150 down to $6. Hard for investors and also for many employees.

I remember the first shareholders meeting after the stock had hit its $6 low. It had "recovered" to about $12, so it wasn't quite at its worst ever, but the mood in the room was not at all cheery. Still, Dan gave a good presentation, and then opened up the floor for questions.

A small, old woman got up, clutching her walker, and slowly made her way to the microphone. She had a quavery voice. "I was watching the stock ticker a couple months ago. When I saw my beloved Network Appliance hit $6, I almost fell out of my wheelchair rushing to the phone to call my broker..."

What flashed through my head was, "Oh my God, Grandma sold it all at the bottom." As if I didn't feel bad enough already.

She continued, "...and I told him to buy as much as we could get our hands on. Mr. Warmenhoven, I've doubled my money since then. Here is my question. Whatever you are doing, are you going keep doing it?"

After a short silence, as her question sank in, the whole audience started clapping, and then laughing. One simple question completely changed the mood of the room. I've always wondered whether she did that on purpose, just to stop people from being so grumpy.

Sometimes the Conclusion Belongs at the End

My blog on how to fail in executive staff presentations argued that you should make your request—whatever it is that you are trying to get approved—right up front.

Several people complained that this blog didn't follow its own advice. Linda Henry put it best:

Okay, had to laugh. Your concluding statement on your blog "Always start your presentation with the conclusion" was in the final paragraph. Master of Irony!

The real irony is that I didn't do this on purpose—didn't even notice what I had done!

When I first started blogging, I did worry about the rules for conclusions in short essays. In most papers and presentations, I try to "Tell them what you are going to tell them, tell them, and then tell them what you told them." That argues for putting the conclusion in front.

For blogs, this format didn't work. The entries were so short that "conclusion, meat, conclusion" left no room for meat; simply repeating the conclusion twice didn't seem right. To confirm my intuition, I read several opinion/editorial pieces in the newspaper, because those are about the same length as my blog entries, 600-700 words. Sure enough, they generally make a short argument with the conclusion at the end.

Why? Why should the rules be different for an op/ed or a blog than for an executive staff presentation?

The key is whether you require a decision at the end of a limited time. If you have 60 minutes, and you hope to leave the room with a decision, then you better tell your audience right up front what you hope to accomplish. I have seen so many meetings fail because the goal doesn't appear till time is almost out, and then it's too late for the presenter to accomplish what he had hoped. Sometimes, if I want to help the presenter out, I'll interrupt early in the meeting and ask, "Before we go too far, can you tell me what you actually want?" If I suspect that I'd rather say no, I might choose not to help; I'll let the presenter dig their own grave.

In a blog I'm not requesting a decision and there is no time limit, so neither rule applies.

A written essay is also different than a presentation because you can easily go back and re-read the arguments after you get to the conclusion. One of the big problems with leaving the goal till the end in a presentation is that it's hard to evaluate arguments when you don't know where you are going. But in a short essay, it's easy to go back and re-evaluate the arguments in light of the conclusion.

A short essay is different from a long paper. Reading a long paper takes time, and at the end of the paper I'm going to be upset if the conclusion catches me by surprise and I need to re-read the whole thing. E-mails are also different because many people read only the first few lines before hitting delete, so if there's something you need people to know, you had better put it right up front.

Conclusion: For presentations that end in a request, or for long papers, it's best to start with the conclusion. E-mail too. For short essays, or for presentations without a request, it's okay to save the conclusion till the end.

The "Year of iSCSI"

I just read a blog by Chuck Hollis, from EMC, musing about why iSCSI hasn't been that successful. He says we keep waiting for the "Year of iSCSI", but it never comes.

I respectfully beg to differ!

For years, iSCSI really did stand still. The spec was coming, almost finished, almost-almost finished, shipping any year now... I do admit that's how it felt at first. But when I look at the iSCSI numbers for the past few years, what I see is an explosion of success.

Here's a little thought experiment. Let's compare the first four years of iSCSI revenue with NetApp's first four years of revenue. (I think that most people would acknowledge NetApp as a high-growth successful startup. :-)

	Year	iSCSI	NetApp
	1	$6.7m	$14m
	2	$80.9m	$43m      (The year NetApp went public!)
	3	$228.3m	$93m
	4	$457.5m	$157m

[Note: The iSCSI numbers are for the four-quarter periods ending in Q3, to align with the most recent IDC data. The NetApp revenue numbers are my fuzzy recollection and could be off by +/-$5m in the last two years.]

Just to put things in perspective, iSCSI is now ten times as big as NetApp was when we went public. I can tell you that it sure felt like the "Year of NetApp" when we reached that milestone! iSCSI's growth rate is way outstripping NAS in the early days.

I think that Chuck is thinking about it wrong because he is thinking from the perspective of a big VP at a multi-billion dollar company. Of course, I'm also a big VP at a multi-billion dollar company, but I remember NetApp's early days well enough to know how hard it is to create the first $100m in a new market.

My theory is that $100m qualifies any new technology for "the year of". (Back where I come from, a hundred million is a lot of money.) This means that "The Year of iSCSI" was 2004, with $109.6 million in revenue. The most telling statistic is that iSCSI has doubled in the last 12 months and shows no sign of slowing. The quarter-over-quarter growth rate was 15.6% three quarters ago, 19.1% two quarters ago, and 18.5% last quarter. Let it double two or three more times and iSCSI will definitely be a force to be reckoned with.

The other place I disagree with Chuck is his expectation that, "You'd think it'd be pervasive in large enterprise shops these days." A year ago I argued that, iSCSI Sucks, but that's missing the point. It's cheap and it's easy. My point was that you should not expect iSCSI to compete against Fibre Channel in enterprise data centers. Instead you should expect it to compete at the low end, helping small windows servers enter the modern world of networked storage. Over time, iSCSI might become disruptive and compete against high-end Fibre Channel, but that's not where you would expect the first billion in revenue.

In closing, however, I have to admit that I also agreed with many of Chuck's comments. In particular:

"Customers could save big money by avoiding the purchase of fibre-channel HBAs."

"They could use lower-cost ethernet switches rather than their more pricey FC counterparts."

"As we matched up iSCSI performance against our rather extensive knowledge of real-world workloads, it became pretty clear that a significant majority of applications could run comfortably on iSCSI with no negative performance impact whatsoever."

"First, I think you'll see more of the same: iSCSI in smaller, greenfield SAN builds where FC isn't entrenched yet. But from small acorns mighty oak trees grow."

In the end, Chuck and I may agree more about iSCSI's future than about its past. I'm arguing that the "Year of iSCSI" is well behind us, back in 2004, but as for the future of iSCSI, Chuck may be more right than wrong.

[The iSCSI numbers are from IDC's "Worldwide Quarterly Disk Storage Systems Tracker, Q3 2006 Release", November 2006.]

I'm Turning on Comments

After a year and a half of blogging, I'm finally turning on comments. The entry below on why lawyers aren't evil is your first opportunity to disagree (or agree) with me in public. I just couldn't handle the pressure from folks like Hu Yoshida saying that vendors without comments are basically wimps. Thanks for the nudge, Hu! :-)

My plan is to leave new blog entries open for comments for two weeks, and then lock them down. Why? The reason is simple laziness. I intend to read all of your comments, even if I don't reply to them all, and I don't want to have to deal with comments from 18 months worth of entries all at once.

I will sometimes filter comments, but my goal is to remove spam and porn, not to stifle discussion.

Have fun!

Lawyers Aren't Evil—Fairness and Morality Aren't Their Job

When I was young and naive I used to think the justice system was about fairness and morality.

My first exposure to the opposing view was when several of my college friends became lawyers. They were taught that laws were more like rules of a game. Four of a kind beats a full house, but there's no morality to it. I remember one lawyer arguing that morality and fairness are irrelevant; all that matters is passing some kind of judgment so that people can get on with their lives. This view just felt wrong to me, and it helped me understand the reputation that lawyers have for being slimy and evil.

My time at NetApp has resulted in an extensive legal education. Here's a blog entry about the time I violated SEC insider trading rules. Another time I was sued for sexual harassment, violation of the California constitution, and intentional infliction of emotional distress. (I wasn't alone: the suit was against all of NetApp's founders, officers, VPs and board members. Settled out of court. The alleged harasser no longer works at the company. The lawsuit included the rest of us just to get our attention. It worked!) At least two other times I've been accused of stealing another company's intellectual property, again along with a long list of people. (We didn't do it.)

My new slogan: Success breeds litigation.

I have a new respect for the lawyer's point of view: Sometimes what matters most is getting to a conclusion so that everyone can move on.

Suppose you accidentally bought a stolen TV. Let's say you bought it at a reputable store, so it's not like you could have known, like if you'd gone to a bad part of town and gotten 80% off from a guy selling TVs out the back of his van. A year later, the original owner somehow finds you, and he wants his TV. You didn't do anything wrong, so you think the old owner should leave you alone and go hunt down whoever stole it. But he argues that he didn't do anything wrong either, and since the TV was his first, he wants it back. For completeness, let's just say the store where you got it has closed down, no forwarding address.

Two people are in a dispute, and as a society we need a process to determine who gets the TV so that everyone can move on with their life. Laws are not about right and wrong. Laws are simply the operating system of the country.

That's all fine for this example, where neither person did anything wrong, but what if our moral intuition says one person was good and the other person evil. Then shouldn't we worry about right and wrong? The lawyer response is, "That's not my job. It's Congress's job to pass the laws. As a lawyer, all I care about is what the law actually says."

To put it in computer terms, laws are the program, Congress is the programmer, and lawyers, judges and juries are the CPU whose job is to execute the program. If there are bugs, blame the programmer, not the CPU. That blue screen isn't Intel's fault. Based on this analogy, I have come to agree with lawyers that their job is not to worry about fairness and morality.

Still, I think it's wrong if nobody worries about "moral bugs" in the legal code. It may not be the lawyer's job, but if the legal system veers too far from people's moral intuition, then I think that society runs into trouble. Perhaps one problem is that we have too many lawyers in Congress. As lawyers they were able to legitimately ignore the moral implications of laws, but that left them ill prepared for the moral debugging that should be part of their job in Congress.

I think there's also an important lesson for companies: The law sometimes matters less than the judgment of the public. Even if a company follows the strict letter of the law, it will run into trouble if it violates the moral intuition of its customers, shareholders and employees. Saying "what I did was legal" is not always a good defense.

The Genius Detective Game: How to Fail in Executive Staff Presentations

Imagine this scenario.

An old friend rushes up to you and says, "I've been looking all over for you. You wouldn't believe the amazing car I saw yesterday. Twenty-five years old, but the paint is perfect, and it runs like a charm."

You ask him what his point is, but he just talks more about the car. "The current owner just replaced the tires. It's not expensive now, but a classic like this is sure to go up in value." All well and good, but you still wonder what this has to do with you.

You need to leave soon, so you stand up and look at your watch, but your friend suddenly says, "Wait! Here's the deal. This car is for sale on eBay, and I'd like you to buy it and lease it to me. Seriously, this is going to be a great investment for you."

There you are, two minutes from your next meeting. How do you react? When you say that you need time to think about it, he says, "The auction ends today. I just gave you all the data you need. My facts prove that this is a great deal. I need you to decide right now."

I hate this feeling, and I've discovered that presenters sometimes trigger it in executive staff meetings. The meeting agenda has 45 minutes on a vague but interesting topic (e.g. Classic Cars). For 42 minutes the presenter shares lots of facts and figures, and then—with three minutes left—the presenter puts up a complex proposal (e.g. Automotive/Leasing Investment) and asks for approval. You should know that presentations like this almost never go well.

The way I describe this, it sounds completely idiotic, maybe even sneaky and underhanded, but I think that people make these presentations with the best of intentions. They believe that they have an airtight case, and they want to make sure that they present all of the evidence so that when they get to the conclusion, everyone will immediately see that the plan is perfect. Kind of like in detective shows where the hero reveals the evidence piece by piece and then stands up dramatically and identifies the killer. The genius detective game is great in movies, but in presentations it leaves the audience wanting to go back and re-examine the evidence.

The problem is lack of context. As a listener, it is very difficult to evaluate an argument if you don't know where the presenter is taking you. The presentation may be informative, and it may be interesting, but if you don't know what the conclusion is, it is almost impossible to tell whether the evidence supports it or not.

To return to my simple analogy, if you knew that the conversation was about buying and leasing a used car as an investment, there are all sorts of questions you might have asked. What is the blue book value of the car? Has it been inspected? How many miles does it have? You can't ask the right questions unless you know what the presenter is going to propose.

I think sometimes people defer their actual proposal to the end because they are afraid of getting shot down, and they want to postpone the pain. Better to get shot at right away, because then you have the rest of the meeting to try to recover. You can focus very specifically on the concerns that caused objections. Maybe you'll change people's minds. Or else, you may get input on alternate approaches that will work better. Either outcome is better than getting shot down at the very end when there's no time to recover or to prepare for the next attempt.

Here is my advice: Always start your presentation with the conclusion. Somewhere in your PowerPoint slides—it may not be the last slide but it often is—there is a conclusion slide that summarizes the proposal that you hope to get approved. Put that slide first. If you need some background information, so that people can understand your proposal, then maybe you can get away with one slide before the conclusion slide, but if you push the conclusion any further back than that, then you are probably playing the genius detective game. As I said, those presentations almost never go well.

Why NetApp's Earnings Results Last Quarter Frustrated Me

There's an old saying in computer science:

Fast, cheap, reliable—choose any two.

The point is that these goals are in conflict. Improving one tends to hurt the others. Sometimes a new technology paradigm lets you improve all three at once, but within the new paradigm there will still be a conflict between the three.

There is also a conflict between different goals when you design a business model. In this blog, I'll describe the key conflict in NetApp's business model, and how we've resolved it.

If you round everything to the nearest 5%, here is what NetApp's business model generally looks like. For every dollar of revenue we receive from customers, we spend 40 cents manufacturing the products we ship and another 45 cents running the company, which leaves about 15 cents in operating profit. To put this into business terminology, the "operating stack" looks very roughly like this:

40% COGS (Cost of Goods Sold: components, labor, overhead, etc.)
45% Operating Expenses (sales, marketing, engineering, salaries, etc.)
15% Operating Income (profit from running the business, before taxes)

(For the real numbers, see the transcript of our Q2 earnings call.)

The key conflict is long-term growth versus profits this quarter. To maintain growth, you have to invest for the future. If you want to sell more next year than you are selling today, then you'd better hire more sales people. If you expect to have more customers next year, then you'd better hire more customer service engineers to support them, more manufacturing people to build products, and so on.

But investments increase operating expenses and drive down profits.

Let's focus on hiring new people. Employees usually aren't very productive for their first few months, so at first, hiring new people raises expenses without improving revenue. The faster you grow, the more people you have who aren't yet pulling their own weight. I call this the growth tax. At high growth rates, the penalty can be significant. Suppose it takes people 3 months to get up to speed, and suppose that all operating expenses are proportional to head count. At a growth rate of 35%, our growth rate last quarter, the growth tax is about 3.5% of the overall stack. (Here's my math: (135%^(1/12)^3-1)*45% = 3.5%.)

That might not seem like a big percentage, but I can assure you that stock market analysts focus very closely on profitability. That 3.5% growth tax would reduce an 18.5% profit to 15% which is definitely significant. At 50% growth, based on these same dramatically oversimplified assumptions, the tax would be almost 5%.

[Aside: I enjoy applying engineering-style thinking to the operation of the company as a whole. What are we optimizing? Can we model the result? And so on.]

Summary: To grow you must hire, but hiring drives down profits.

The management team at NetApp has decided to optimize for long-term growth, as opposed to optimizing for short-term profits. We believe that optimizing for profit would actually be damaging to the company, because it would doom us to being the perpetual underdog.

That's why I'm frustrated with our earnings results last quarter (FY2007 Q2). By many measures, Q2 was wonderful. It was our largest quarter ever, with $652 million in revenue. That was a 35% increase over Q2 a year ago, which is an awesome growth rate for a company our size. The profit level was also unusually high. We had a non-GAAP operating profit of 18.2%.

It's that operating profit that frustrated me. We believe that the optimal trade-off between growth and profitability occurs at roughly 16% operating profit. (The range we target is 15.8% to 16.4%.) Generating higher profit meant that we invested less. At $652 million in revenue, that 2.2% difference comes to almost $15 million that we could have invested in future growth but did not.

At our annual analyst day conference a couple of years ago, we were describing our business model to investors, and we got an interesting question. Laura Conigliaro, an analyst at Goldman Sachs, said, "If you are targeting a 16% profit model so that you can invest in the future, does that mean it would be bad news if your profit level were to rise? Would that mean that you had run out of new ideas to invest in, and that growth was going to slow?"

We're certainly not out of ideas. In the earnings call, Dan summarized it like this: "We missed an opportunity to get more aggressive this past quarter, and I'd like to see that not happen again."

Data and Ethics (Who Owns My Medical Records?)

Yesterday I testified to the FTC (Federal Trade Commission) as part of a panel on the effect of technology on consumers. My talk was basically a summary of this editorial that I wrote for the Financial Times.

An interesting topic during the open discussion was this: Who owns data about you? Perhaps more importantly, who should own that data? Should Amazon own the record of all the books you have ever bought, or should you? Medical records are even more personal. Should your doctor own your medical records, or should you?

Professor Deirdre Mulligan (at U.C Berkeley, Boalt Hall School of Law) was also on the panel, and I thought she had the deepest insights on the legal and policy issues around personal data. She argued that ownership is the wrong mental model. I certainly have a strong interest in my own medical records, but my doctor has an equally strong interest in the collection of medical records that he has created over the years. From his point of view, it is the record of his career. In the case of a malpractice suit, the records may be required to demonstrate his competence. In some cases, a medical center as a whole comes under regulatory scrutiny, and the records of all doctors and patients at the center may be required to understand the patterns of care.

Mulligan argued that instead of ownership, it is better to think in terms of rights and responsibilities associated with the data. As a patient, I have many rights with respect to my own medical records. I can get access to them, transfer them to a new doctor, and so on. My doctor and the medical center have rights as well, but also responsibilities. They are required to keep the records safe and private. If they are electronic, then HIPAA regulations require them to keep that data for the rest of my life. (You can be sure that the storage implications of these regulations are not lost on us.)

I used medical records as an example, but financial records have the same issues. I think they are my records, but my bank and my broker have lots of reasonable reasons—including legal reasons—to think they are their records.

In summary, if I heard Mulligan's arguments correctly, she was saying that ownership just isn't the right model for thinking about this fuzzy, blended combination of rights and responsibilities that are shared between my doctor, my medical center, my bank and me.

On the other hand, Mulligan used quite a different line of thinking for data that I create that nobody else has any rights to—like my personal calendar or my diary. She argued that it is a very odd artifact of our legal system that my rights are dramatically different depending on whether I store my calendar on my own PC or on a remote server at Yahoo! or Google. If the data is on my own PC, in my own possessi