Extensible NetApp: Thinking outside of the box

We’re #1, dang

When I was in college, I would always check the US News and World Report college rankings to see where Brown ranked (we’re 16th). The annual ritual was that me and my friends would sit around  and bemoan that XYZ was ranked higher than us but it was only because they had a bigger endowment or because the ranking system was broken. After all we were cooler or you know better. 

And we would read the annoying interview about XYZ’s students who paid no attention to those rankings, because they knew they were at the best school. We knew they paid close attention, but I guess being the top dog let’s you pretend you don’t care.

Maybe this was all a form of buyer’s remorse.

It turns out that the same ritual exists with the Fortune survey of the 100 best companies to work at. Ratings come out, and then you spend half a day trying to figure out how they could be so flawed to pick you as anything other than #1. If you love your job, you want to know why you’re not the best place to work at, because how could things be any better?And then you begin to rationalize that this survey is flawed and broken, because how could ABC be any better than you?

But today we’re #1, and so I am in uncomfortable territory. I don’t know how to act indifferent. But I do know how to say that it doesn’t surprise me that we were voted #1.

And if you ask me why it’s about our culture. What brought that home to me was reading Dave’s book. What was surprising was that the book contained no net new information about the history of NetApp. Oh sure there were details, but there was nothing substantive. I didn’t read the book and think: wow, they lied to me about why they did this or wow, this isn’t what they told us about why this happened. Think about it, a book about 9 years of history written by a key executive had little new information for an employee who had been here for 9 years.

That says something about the kind of culture we have, and perhaps goes to some length to explain our current ranking.

p.s  XYZ was Harvard, and dangnabit, was Harvard this year, again! And I am sure it’s the endowment, it’s always the endowment, cause Brown is just sooooo much better.

How to embarrass your co-workers with a funny book title

There,  How to Embarrass Your Co-Workers With a Funny Book Title, that’s the title of my book. And I have Dave to thank for that.

Someone lent me a copy of his book titled How to Castrate a Bull: Unexpected Lessons on Risk, Growth and Success in Business. Unfortunately they left it on my desk, and unfortunately the part of the title that says How to Castrate a Bull is in black on light orange, and the part that says Unexpected Lessons on Risk Growth and Success in Business  is in a different shade of orange on a orange background. And of course, the book was left on my desk on Tuesday and I only picked it up on Thursday… So for two days people walked by my desk and wondered why I wanted to know about how to castrate a bull. And some of my co-workers were wondering if there was some marital issues …

And now that I am blogging about his book, I wonder if that makes me a brown noser (see page 162). I suppose it does if the opinion of his readers interests Dave.

So what about the book?

Read a book about a place that you’ve been at for 9 years, and the first thing you want to do is see if you made it into the book, and the answer, for me at least, was no. More depressingly my first project, NetCache, didn’t get mentioned either.

But it is about NetApp so some of my favorite moments made it.

My all time NetApp moment is when Tom Mendoza told the sales force:

Your job is to find people who have money and sell them something.

When at the height of the bubble that very much did not seem to be the plan for many of our competitors in the IT space.

Ask me why NetApp is still in business, I really believe it’s because we never lost sight of that simple fact.

So is the book just a collection of cute anecdotes? Is that the point of the book?

No.

So then what is? Maybe it’s a chance to hang with Dave.

At NetApp I’ve had the chance to talk to Dave Hitz at best a hundred times. That is remarkable given the company size and our relative positions. And reading the book reminded me of why I enjoy listening to Dave talk about NetApp.

Reading his book is like getting 192 pages of 1 on 1 time with Dave. You’ll get exposed to things that you’d never expect a geek to know about, like how to actually castrate a bull , and things you would expect a geek to know about, Dungeons and Dragons, and passion about all things related to technology. And you’ll get a few chuckles from some of his jokes. My personal favorite is on page 192.

But even hanging with Dave is not the point about the book.

No, the real point is you get to learn about a remarkable company and how that remarkable company succeeded from a remarkably eloquent man, and the lessons Dave learned along the way.

But what if you are not a card carrying member of the Dave Hitz fan club?

Here’s why you should read it. What makes the book unique in its genre is that Dave takes his engineering mind and applies that mind to understanding the how and the why of NetApp. Not just from grand strategy, but about culture and about management. And what makes that fascinating is that engineering is an art and a discipline. And Dave understands the limits of both. And sometimes he is able to explain the why, and sometimes he admits that it’s just magic.

But even if that’s not good enough, read it because of NetApp.

There are very few companies the size of NetApp (about 1000). And there are very few companies that grew as fast as NetApp to become that big. And the story of why NetApp grew to be that big should be interesting. And if it’s told by someone as fun to listen to as Dave, you’ll find it that much more interesting.

twittering

So I am available on twitter.

My twitter id is kostadis_netapp

If you don't need another thing to learn, you can always just read my tweets at http://www.twitter.com/kostadis_netapp

DR testing

What's the worst time to test your DR strategy?

Why in the middle of a DR event.

Why do I ask?

Well my laptop has a piece of hardware that failed. That hardware failure exhibited itself as a blue-screen-of-death. My experience with software in the past has been that blue-screen-of-death has more to do with the OS being corrupted than the hardware failing (and sometimes the two events are related).  So my strategy has been to recover from an image on a different hard-drive and roll my backups forward.

As I started the re-image process, I observed that I didn't actually know if the re-image would succeed. I had never actually tested the DR process to make sure I had not screwed something up.

Part of the reason I never did the test is that I would need a third hard-drive. If you're testing the DR image is good, you don't want to copy the DR-image over the original image. And another hard-drive costs money, and hardware reconfiguration and copying data takes time.

And then that made me appreciate FlexClone even more as part regular DR management.

Using FlexClone I can create very quickly and at very low-cost a third copy. That third copy can be mounted, validated, and if it works, deleted.

As I sat there watching the data get copied, I wonder whether I was in a fantastical world of pain.

Good news is that I am not.

There is still the small problem of rolling my backups forward, but that's a manageable bit of pain.

But there is the small issue of proper DR procedures.

But just having a DR copy is not enough. The other piece is to have a DR strategy that is written down. In the middle of a DR event you can't think straight. All you want to do is get back to running as fast as possible. You're stressing out that you'll lose all your data. You're worried about lost productivity etc.

So you end up doing whatever cooky idea comes first rather than methodically figuring things out.

Well it turns out that my problem could have been solved by:

1. Examining the BSOD details to see if there is a specific object that is failing
2. Search on google for that object
3. See what corrections are recommended

If I had done that rather than jumping to re-imaging, I would have realized a simple enough fix would have been to extract my hard-drive, put it in a USB case, delete a single offending file, and then continue.

I can assure you my new DR strategy is written down, and one of the key items is don't start until after you've drunk some coffee.

Memories of a different life

In 1996 I graduated from Brown University to go work at SGI on what was a rather interesting technical problem: how to make hybrid scheduling work.

Just to recap, in the early 90’s the UNIX vendors were adding threading to their kernels and there was intense debate over what the right model was:

In the 1 on 1 model, each user thread had a kernel  thread. In the N on 1, all of the user threads were multiplexed on one kernel thread. Finally in the M on N model, many user level threads were multiplexed on a smaller number of kernel threads.

The N on 1 model was the “simplest” to implement, but did not allow the application to exploit multiprocessors. One property of the N on1 model was that synchronization between different threads did not require a context switch.

Although 1 on1 was simple enough to implement it had, what at the time seemed serious, issues. The two most discussed were the memory resources necessary to maintain the kernel data-structures, and the second was the performance implications of making every synchronization operation a kernel operation.

So the M on N model was promoted as a compromise between the N on 1 model and the resource intensive 1 on 1 model. To implement the M on N model, two schedulers were required : a kernel and user level thread scheduler. The central notion was that the kernel scheduler would schedule processors and the user level thread scheduler would assign threads to the available processors represented by the kernel threads. The nice property of this model is that if synchronization was required between user-level threads, the performance would be as good as the N on 1 model.

The challenge of this model was that the thing scheduling the physical resources, the kernel scheduler, had incomplete information and was prone to making mistakes. And the cost of mistakes was idle processors and therefore worse performance.

Ultimately a variety of attempts were made to fix the problem including scheduler activations and scheduler aware synchronization and nano-threads but none good enough to stop the ultimate triumph of the 1 on 1 model.

It was the need to co-ordinate between the two schedulers that eliminated the advantages of the M on N model.

One could almost argue that this was a computer science boondoggle.

So what does this have to do with storage?

At the end of the day the lesson I learned from the whole experience is that having multiple layered resource schedulers scheduling the same resources is difficult to make work.

If you look at the storage stack, what you’ll see is multiple distinct storage layers each managing virtualized pools of storage that they have no visibility into. And what’s worse the thing that actually does the physical resource management has no visibility into what the resources are being used for.

So to me it looks that the storage stack has evolved over the last 20 or so years to look a lot like the hybrid thread scheduling model.

And that is enough to give me reason to pause and think.