NetApp - Dave's Blog

Management Lessons from Hernando Cortez

In 1519, Hernando Cortez began his conquest of Mexico. He “motivated” his soldiers to fight by burning all his ships, making retreat impossible.

Here’s the thing: It’s tough being a big boss. You can try ordering people around, but it’s hard to make hundreds or thousands of people do what you want. Sometimes they don’t pay attention or don’t understand. Sometimes they hear you clearly but decide to do something different anyway. Cortez’s management technique left no escape: fight or die.

Several years ago, our CEO Dan Warmenhoven applied this technique to force NetApp managers to hire more people outside of Sunnyvale. We had development centers in Research Triangle Park in North Carolina, in Pittsburgh, in Bangalore India, but most of our employees were in Sunnyvale and their natural inclination was to hire more people in Sunnyvale. I remember the executive staff meeting where our VP of facilities was explaining why we needed to start construction on a new building in Sunnyvale: “It will take 18 to 24 months to complete construction, and at the rate we’re growing, we must start now to be ready.” Dan replied that he didn’t want to continue the same growth rate in Sunnyvale, but the head of facilities pointed out that Dan had been saying this for years, and the hiring rate in Sunnyvale was faster than ever. “We really need to start those buildings now.”

Dan has this little vein in his forehead that pops up when he is frustrated: it pulsed pinkly. “If we build more buildings, the hiring will get faster still. I don’t want to hear another proposal for buildings in Sunnyvale for at least 12 months. If managers hire more people in Sunnyvale, they’ll have to work standing in hallways.” Cortez management in action. We eventually did do more construction in Sunnyvale, years later, but not until the hiring trajectory had changed dramatically.

In many companies, the tightest control systems are for large capital purchases. Perhaps bosses can’t control what workers do on a day-to-day basis, but they can control what gets purchased.

This was a common tool in the transition from Mainframes to UNIX servers in the eighties and nineties. CIOs said, “It’s not that I don’t like mainframes, but they’re too expensive, and I won’t buy another one.” The boss didn’t need to specify details about how many UNIX systems to purchase or what applications to move over. One top-level decision forced the transition, and the IT staff handled the details on their own.

A popular use of Cortez management these days is for CIOs to announce that they aren’t going to build another data center: “I know the current data center is almost full and has no more power, but I’m not building another one. Figure it out.” This is the best way I know for a CIO to drive faster adoption of Server Virtualization.

Encouraging the Kindle Phenomenon

The physical version of my book is now available for pre-order on Amazon, but they don’t have the Kindle edition. Being the Kindle-loving geek/nerd that I am, I’m trying to get the Kindle edition up as well, so I’d love it if people would help me by going to the Amazon page for How To Castrate A Bull: Unexpected Lessons on Risk, Growth, and Success in Business and clicking on the link that says, “Please tell the publisher I’d like to read this book on Kindle.” It’s on the left side, just below the picture of the front cover.

There’s even a link to read a random page. I wonder if anyone has written a script to click a bazillion times and download whole books for free? You can also do searches. (I recommend “thud” or “testicles”.)

Is WAFL a Filesystem?

Many people think WAFL is a filesystem. I certainly thought so fifteen years ago when I wrote it, but folks like Kostadis Roussos are now claiming that I was wrong. (A NetApp employee no less!)

To understand why, you have to understand how WAFL is structured.

WAFL has a “top-half” that deals with files and folders—“My Documents” and “QuarterlyEarnings.ppt”. It keeps track of who created the file, when they created it, who can look at the file, who can modify it, and so on. This certainly sounds like what a filesystem would do. The top-half actually supports multiple filesystem protocols. We started with NFS for UNIX, but right from the beginning we knew we’d be adding more. Originally we expected that Novell Netware would be next, but Windows CIFS gained momentum so quickly that we did that instead and never got around to Netware.

WAFL also has a “bottom-half” that manages the physical disks in the system. It organizes the disks into separately managed pools, it keeps track of which disks are part of which RAID array, and it arranges data on the disk to maximize read and write performance. The “bottom half” also handles data management functions like snapshots, remote mirrors, cloning, de-duplication, thin provisioning, and so on. These capabilities would traditionally be part of a volume manager or a block virtualization layer.

One of the things that was unique about WAFL when we shipped it was the way we integrated the two layers. There are many opportunities for new features and clever optimization if you design the layers to work well together.

When we decided to support iSCSI and Fibre Channel SAN, WAFL’s bottom-half data management capabilities were a perfect fit. In fact, one of the things that helped convince me that NetApp should support block-based storage was realizing how valuable our data management features would be in that environment.

This top-half/bottom-half structure explains the confusion about WAFL. My current view is that WAFL contains a filesystem, multiple filesystems actually, but that’s different from being a filesystem.

Legal Question about Amber Road

Someone asked me the other day whether our patent infringement claims against Sun (see here, here, and here) include Sun’s “Amber Road” storage system. I said "Of course!" Commercial products that use ZFS are exactly the point of our lawsuit. Sun likes to portray this as a dispute about open source, and ZFS being open source does complicate things, but the reality is that this lawsuit is about Sun building and shipping products based on our intellectual property. At this point, we have sixteen patents that—we believe—ZFS infringes on. So naturally, the case we are making applies to any product from Sun (or any other company) that is based on ZFS, including Amber Road.