NetApp - Dave's Blog

Capacity versus I/O Throughput and Latency

After my blog entry about why "RAID-DP doesn't waste extra disk space", I got a comment from a reader saying that he thought performance was usually the limiting factor in storage systems rather than capacity:

In my experience, I/O throughput/latency is much more important than storage capacity and is the main driver for the number of spindles. If RAID-DP does not decrease I/O throughput, I don't care if it decreases my available disk space by 15%. Modern disk drives have a completely unbalanced capacity to throughput ratio, in any case, as ever-lengthening backup windows show (hence the value of snapshots, of course). —Fazal Majid

It's true that disk capacity is going up much, much faster than seek time. As a result, for many applications, the number of spindles required is determined by disk performance and not by capacity. One of the Exchange folks at Microsoft told me that one of their focus areas in development is to reduce disk traffic, for exactly this reason. Since people buy more capacity than they need in order to support the disk I/O, driving down disk traffic is a great way for Microsoft to help lower customers' costs.

I think this is one of the reasons that snapshots have become practical. We've got all of this disk space without enough performance attached to it, so we might as well fill it up with something that we don't look at very often, like snapshots.

On the other hand, there certainly are applications—especially archival apps—where capacity is the limiting factor and not performance. In order to correctly size a storage array, you really have to understand the performance requirements of your application, and not just the amount of storage used.

Why "Double Protecting RAID" (RAID-DP) Doesn't Waste Extra Disk Space

Mike Linett raised a concern in his blog that using NetApp's RAID-DP "wastes storage space" and requires customers to "buy more protection against disk failure".

Actually, using RAID-DP doesn't waste any extra space, and since it's bundled with the base system, there's nothing extra to buy. (For folks who don't follow our technology closely, RAID-DP is a version of RAID that protects against double disk failures by using two parity disks instead of one. Regular RAID only protects against single disk failures. The generic name for double protecting RAIDs is "RAID-6". RAID-DP is our own particular implementation.)

Here's why RAID-DP doesn't waste space. With regular RAID, we recommend that users create 7 disk RAID arrays with one parity disk. With RAID-DP, we recommend that users create 14 disk RAID arrays with 2 parity disks. Either way, it's 2 parity disks for each 14 disk shelf. The math works out that RAID-DP on 14 disks is much, much safer than regular RAID on 7.

The really elegant thing about RAID-DP is that it is backward and forward compatible with our regular RAID. That is, you can take a regular RAID array, add an extra parity drive, and presto you are now double protected. Or if you have a RAID-DP array, and you want the extra parity drive back to increase data capacity, you can downgrade to regular RAID.

Of course, if you convert a 7 disk regular RAID array to RAID-DP, you'll now have an 8 disk array with two parity disks, so your overhead goes from one-out-of-seven to two-out-of-eight, that is from 14% to 25%. To avoid wasting space, you have to create it as a RAID-DP array to start. Either that or convert a 7 disk raid group to RAID-DP and then add 6 more data drives. We thought it would be really cool to be able to take two regular RAID arrays and combine them into a single RAID-DP array, but it was too much work. Oh—and there is essentially no performance penalty. We run all of our benchmarks now with RAID-DP turned on.

Bottom line, I think everyone should use RAID-DP whenever possible.

(For gory technical details on RAID-DP, see here or here. At the conference where they presented their work, the authors Peter Corbett, Bob English, Atul Goel, Tomislav Grcanac, Steven Kleiman, James Leong and Sunitha Sankar won "Best Paper".)

Oops—Single Mailbox Restore is a Separate Product

In my previous blog entry I made it sound like restoring a single mailbox in exchange was a feature of our SnapManager for Exchange product. In fact, you have to buy a separate product called Single Mailbox Recovery for Microsoft Exchange.

I was confused, but fortunately there are enough people reading my blog that when I screw up, they can set me straight. I wanted to correct the record so that nobody would end up unhappy due to my misinformation.

Data Management and Automatic Teller Machines

Last week's launch included a major overhaul of our Data Manageability Software Family. I'd like to dig into our strategy there. (The launch also included a new high-end storage system, the FAS6000 series, but instead of covering that myself, I'll refer you to Tony Asaro's ComputerWorld blog. Summary: He likes it.)

My favorite way of thinking about data management is to focus on how storage administrators spend their time. If you are a storage admin, then all day long you respond to requests from people who want more storage, or want you to make some change to their storage. Sometimes they telephone and sometimes they e-mail, but the requests are always the same: "I'm installing a new Exchange server and I need a one terabyte LUN." "I've got a one terabyte LUN, and I need you to grow it to two terabytes." "I need a copy of my primary database so that I can experiment with a new app that I'm developing." What a bunch of busy work! (Some people call this the "storage help desk".)

Think about this from the application administrator's point of view. They can't do their job until the storage admin gets back to them. It reminds me of how people used to get cash at the bank, before Automatic Teller Machines were invented. You would wait in a long line and eventually talk with a live person who handed you your money. What a pain! ATM machines are better for everyone. Less work for the bank, but also better for me as a customer. I don't have to drive to the bank, and I don't have to wait for the bank to open. I can get my money at 3am in a liquor store.

NetApp's goal is to do the same for data management—to let application administrators manage data themselves, without any help from the storage administrator. Of course, storage administrators will never allow this unless they retain control. Let's return to the ATM analogy. When you walk up to the ATM, you should be under no illusion about who is in control of how much cash you can take. The bank is in control, it's just that they control your withdrawals through policies that are programmed into the system. Likewise, the storage administrator will never let application admins directly manage their data, unless there are policies to let the storage administrator maintain control.

This isn't just future vision. As of today, server administrators can do self-service storage provisioning without contacting the storage admin. In addition, we have products that allow application administrators to control snapshots, trigger remote copies for disaster recovery, recover lost data from snapshots or backups, and even create clones for test and development.

Here's an example that shows the extent of our application integration. Suppose an Exchange database gets corrupted. That happens sometimes. With most storage systems, the Exchange administrator would need to work closely with the backup admin, who in turn would need to work with the storage admin to keep restoring old versions of the database until they find one that's valid, then restore the valid one, replay the logs, and restart Exchange. SnapManager for Exchange lets the Exchange admin do it all himself, all automatically. [Note: Single mailbox restore is actually a separate product from SnapManager for Exchange.] Or if you just want to restore a single mailbox, it can do that. The SnapManager tools for Oracle and SQL Server have similar capabilities.

Never mind the details. Here is the big picture.

We are giving application administrators self-service access to a whole variety of data management services, and we're doing it in a way that gives storage administrators enough control to keep them happy as well. Data administrators can get what they need right away, without hunting down and waiting for a storage administrator. Storage administrators can stop wasting time on a steady stream of individual requests, and focus on the stuff that matters—stuff like storage optimization, business continuance, and performance management. Who knows, maybe they can even introduce themselves to their children.

NetApp in the Data Center

Most product launches are primarily about the cool new products a company is just shipping. Our launch earlier this week was different—it is mostly about a strategy that we've been working towards since the late nineties.

Don't get me wrong. There are plenty of cool new products that we are announcing: a new high-end storage system, broader switch and HBA support for 4 Gb Fibre Channel SAN, a restructuring and update of our manageability software family, and new service offerings to help customers deploy storage more quickly.

But never mind today's new products, this launch is more about strategy than products. If you look at NetApp in the mid and late nineties, we sold mostly to engineering organizations at technical companies. Cisco and Yahoo! became customers during that time. We reached $1 billion in revenue selling to customers like that, but we knew that to get past a billion we'd need to broaden our market. Tech is a great niche, but if you want to get really big, you have to crack the data centers of non-technical companies. You have to change your focus from Cisco to Nabisco.

I got lessons in enterprise selling from an IT manager at Georgia-Pacific. This was back in the late nineties when we had just begun targeting non-technical companies, and we didn't yet know much about it. I was giving this IT manager a presentation about all of our cool features—Snapshots and SnapMirrors and the advantages of our write anywhere flexible layout on disk and so on. Before I finished, he interrupted me and said, "Slow down son. What you don't seem to understand is, here at Georgia Pacific, we take logs and we turn them into paper. How's this box of yours gonna help us do that better?" I was dead in the water. I had no clue about his business, and I had no clue about whether or not we could help him turn logs into paper.

Rob Salmon (our executive VP of field operations) got a similar lesson, again in the late nineties, from a bank in North Carolina. He was visiting the CIO, and before he went out, he checked the customer support records to find out what their experience was like. They hadn't had any outages or any serious problems—just a couple of calls about how this or that feature works—so he figured that it would be a friendly meeting. The first thing the CIO said to him was, "Rob, I just want you to know that your salesmen really piss me off."

Not at all the meeting Rob was expecting! He asked the customer for more details, and the CIO said, "Your sales guys come in and they tell me about your cool new box, and your cool new features, and then they lean back and smile, like they are done. I pay you guys enough money that I expect you to come in here and understand my business problems and tell me how you are going to work with my people to help me solve them."

Over the years, our attempts to sell to enterprise data centers have driven change through pretty much every part of NetApp. Our sales and market strategy has changed, we've made large investments in customer support and professional services, we've retooled our engineering release model and development processes. Even organizations like finance and legal have had to change as we've gotten into leasing and volume purchase agreements. And of course, the products themselves have changed. Our new high-end storage system (FAS6000) scales beyond a thousand disks, over double the new CLARiiONs that EMC just launched. In capacity, performance and reliability, it's well beyond the modular mid-tier of the market and up into the range of frame arrays like Symmetrix and TagmaStore. (In addition to Fibre Channel SAN, it also supports NAS and iSCSI.)

For a strategy like serving enterprise data centers, you never reach a point where you can say, "Yesterday we weren't enterprise-ready, but today we are." It's only after the fact, when you've got thousands of commercial customers running business critical apps on your equipment that you can look back and say, "Eight years ago we weren't enterprise-ready, but today we are." Never mind all the new products, that's the real message of the "Data Center Proven" launch.

Replacement CEOs

There is a theory among boards of directors about how to choose new CEOs. The idea is that you should choose an internal replacement if you are pleased with the performance of a company and want to minimize shifts in direction and culture.

I thought this was interesting in the context of Jonathan Schwartz's recent promotion.

I am a Lobbyist. Am I Evil?

I spent the last two days lobbying with folks on Capitol Hill about legislation to protect private information—credit card numbers, financial records and the like—that big companies store about customers. The proposed laws focus on what should happen when bad guys access large databases of sensitive consumer information. Should the company notify the press? Notify the consumer? Pay a fine? Should someone go to jail? And what should companies do to protect against theft? There's been a lot of progress since the last time I visited, six months ago.

It was fascinating to hear so many perspectives. I talked with staff people from both Senate and Congress, both Republicans and Democrats, and I also talked with two lobbying groups that focus on digital privacy and security.

There are currently at least 23 states with protective laws and another 17 or so have legislation in progress. As a result, consumer protection advocates feel pretty good. They see some benefit to a nation-wide law, but they are uninterested in a compromise that weakens existing state laws. They feel that if a federal law is going to preempt the states, it ought to be stronger than any existing state laws, to make up for the fact that states will be losing the ability to regulate further.

On the other hand, companies are horrified at the idea of having to deal with 23 (never mind 50) different sets of regulations. The last thing a company wants during a crisis is to have fifty parallel sets of laws to obey, each with a different set of procedures they have to follow. In my mind, this is probably the strongest argument in favor of a federal law that preempts state laws. Talking with CIOs and VPs of Storage at large financial institutions, I get the sense that they would be thrilled to have a single federal law, even if it were stronger than any of the existing state laws.

However, lobbyists from these companies mostly argue that the current state laws are too strong, and they don't want a federal law unless it's weaker. Also, financial institutions also worry about a new regulator watching over them. They have so many regulators already, so even if there are new rules, they hope they can be enforced by the same regulators that they have now, even if different regulators would be better for other companies.

The House and Senate Judiciary Committees want to make sure that notification requirements don't screw up law enforcement. They worry that if the press is notified too quickly, it could tip off the bad guys and give them a chance to run away before the FBI or police can catch them.

NetApp is in a subtle situation. From a purely self-interested perspective, the stronger the laws, the more encryption equipment we are likely to sell. That tends to align us with the consumer. On the other hand, many of our largest customers are lobbying for weaker laws, and it seems like a bad plan for us to lobby against our own best customers. So why is NetApp visiting the Hill at all? We want to be a technical resource with information about encryption and the role it can play in protecting consumer data. This is a perfect job for me, because I enjoy describing technical issues in ways that non-technical people can understand.

One Senate staffer said, "I am skeptical of encryption. I've got this IT consultant working for me—he's really, really good—and he told me that he can break any code, even the strongest encryption in the world, within eight minutes. So I'm not sure it's safe."

I told him that, no offense, but if this was true then this guy should have a job at the NSA, not wiring networks in a Senate office building. I tried to give him a sense of the ways that the Military and Intelligence communities use encryption, the things they trust it for and the kinds of testing and regulations they have around it. If military-grade encryption is strong enough to satisfy the paranoid people (and I mean that in a good way) at the NSA, then it ought to be good enough to protect credit cards numbers.

Through all of this, it was interesting to track my own feelings. How do I feel about all of this lobbying from all these special interests (including NetApp)? Never mind what's good for NetApp or our customers, what makes sense to me—as a person and not as a corporate officer? Am I evil for lobbying without having answered this question?

Here's something can say: Almost all of the different positions I heard had at least some merit. I definitely support strong protection for private data, but it also seems fair for companies to share their view as well. I do see a problem making companies deal with 50 separate sets of laws every time data is lost. In addition, I was impressed with how much time and effort the congressional staffers were putting in to understanding the issues and figuring out the right answer. I talked with some seriously smart people. It was especially encouraging to see how much better informed they were now than at my last visit six months ago. Since these guys are working hard to balance all of these different issues, maybe it's fair for me to talk only about the areas that I know about, and leave the other issues to experts in those areas.

Perhaps I'm in denial, but I think I might not be evil. At least I didn't pay anybody off.