NetApp - Dave's Blog

Management Tip: To Defend a Decision, Highlight Its Flaws

Sometimes the best way to defend a decision is to point out its flaws.

Let’s say you have decided to pursue “Plan A”. As a manager, it is part of your job to defend and explain that decision to folks who work for you. So when someone marches into your office to explain that Plan A sucks, and that Plan Z would be much better, what do you do?

My old instinct was to listen to Plan Z, say what I don’t like about it, and to describe as best as I can why Plan A is better. Of course, the person has already seen these same arguments in the e-mail I sent announcing the decision, but since they don’t agree, they must not have heard me clearly, so I’d better repeat them again, right? I can report that this seldom works very well.

It works much better if I start out by agreeing: “Yep. Plan Z is a reasonable plan. Not only for the reasons you mentioned, but here are two more advantages. And Plan A – the plan that we choose – not only has the flaws that you mentioned, but here are three more flaws.” The effect of this technique is amazing. It seems completely counter-intuitive, but even if you don’t convince people that Plan A is better, hearing you explain its flaws, and the benefits of alternate plans, makes people much more comfortable.

Here’s what I think is going on. When a decision comes down from on high, it can be scary, because people wonder whether management understands the consequences of their choice. If you understand the pitfalls of your plan, it reassures them that you didn’t make the decision blindly. If you knew all that stuff, and still chose Plan A, perhaps it’s not as they thought. Ideally, being open about pros and cons leads to a conversation in which you can convince them that Plan A really is better. But whether that happens or not, they’ll definitely leave feeling better than if you simply pretended that your plan was perfect and that the alternative was completely stupid. (Of course, if the person raises flaws or alternatives that you hadn’t considered, then you may need to reopen the decision. When to reopen decisions is a whole nother topic.)

This is part of a larger philosophy. I want an environment where it’s okay to openly discuss the pros and cons of ideas and plans. Every idea has advantages and disadvantages. I shouldn’t take it as a personal attack when someone points out a flaw. When you choose a plan, you choose it flaws and all. The best way to succeed is to be open about the flaws, and work to avoid them.

It helps to avoid a strong link between ideas and people. If I think of an idea as mine, then when you insult it, you are insulting me. If I think of an idea as a hypothesis that we can investigate together, and maybe modify together, then we can discuss flaws and alternatives without making it personal. When ideas are a joint project with multiple people contributing, you end up with something better than anyone would thought up on their own.

Controversy: NetApp Outperforms EMC in SAN Database Benchmark

We just released benchmark results showing that our FAS storage systems outperform EMC’s CLARiiON on SAN database workloads. For details, see Brian Pawlowski’s blog. The quick summary is that a NetApp FAS3040 beat an EMC CX3 Model 40 on SPC-1, which is an industry standard benchmark that measures OLTP (Online Transaction Processing) performance. Our system cost less, had fewer disks, and beat the EMC by 24%. With snapshots enabled on both systems, NetApp was three times faster. Here’s the chart:

Why the controversy? EMC has never posted any SPC-1 results, so we had to run the benchmark ourselves. We did follow EMC’s best practice document for the CLARiiON, and we did have the Storage Performance Council independently audit the results. But still, the fact that we ran the tests ourselves caused concern. For instance, Chuck Hollis, the VP of Technology Alliances at EMC, raised questions about the integrity of the auditor, calling him “an infamous part-time ‘administrator’ for the Storage Performance Council” and saying that “It seems to be a part time gig for him to make a few bucks.” Chuck also raised questions about why we would do this: “The only reason you'd spend the money to buy the equipment and run the tests is to put your competitor in a bad light. I think most reasonable customers would figure that part out.”

It is fair to ask why we did this, so let me share our thinking:

First, to continue improving our SAN and database credibility. Given our NAS origins in the early 1990s, many customers and analysts have been skeptical of NetApp’s SAN capabilities. Results like this show today’s reality. Steve Duplessie at the Enterprise Strategy Group commented that “Netapp appears to have legit block performance, and shouldn't be dismissed because people (like me) presume it can't be true.” Chuck Hollis is a vocal skeptic of NetApp’s ability to play in SAN and database environments, so it shouldn’t surprise him that we want to refute his claims. (See here for a brief history of our benchmarking efforts over time.)

Second, to showcase our snapshot performance. Snapshots help customers improve backups and archive old data, and writable snapshots (FlexClones) let customers completely rethink their database test and development strategy. Unfortunately, snapshots in most storage systems are unusably slow. With NetApp, performance dropped only 3% with snapshots on. With EMC, performance dropped by a factor of three. I freely admit that our goal in focusing on snapshot performance was to “put our competitor in a bad light.” I think that’s fair because EMC’s snapshots really are painfully slow in real-world use. On the other hand, we expected our non-snapshot performance to be about the same, or maybe even a bit lower, given that our system is less expensive, has fewer disks, and uses RAID-6 instead of mirroring. Winning there was a pleasant surprise.

Third, because Chuck asked us to. In his blog entry on SPC, Chuck said: “We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.” I don’t promise always to follow Chuck’s advice, but I think it’s important to recognize good ideas no matter where they come from!

One key take away from this result is that turning on a simple feature like snapshots can radically change performance. Don’t let a bad experience with EMC’s snapshots scare you away from NetApp’s.

Let me close with a final word on benchmarks. Any honest vendor will agree that benchmark results are sometimes misleading, and that you should examine the details carefully. We believe that SPC-1 effectively simulates OLTP workloads, and we used real-world configurations based on each company’s own best practices documentation. But despite our best efforts, I stand by my argument in a previous blog entry that you should admire and respect great benchmark results, but also be careful.

 

 

The following SPC-1 results have been posted at www.storageperformance.org:

NetApp FAS3040 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00057

NetApp FAS3040 (with Snapshots):
http://www.storageperformance.org/results/benchmark_results_spc1#a00058

EMC CLARiiON CX3 Model 40 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00059

EMC CLARiiON CX3 Model 40 (with SnapView):
http://www.storageperformance.org/results/benchmark_results_spc1#a00060

All comparisons are current as of January 29, 2008.