Extensible NetApp: Thinking outside of the box

Go NetApp!

After a decade at NetApp, I am moving on to a different adventure.  Which means that my blog and my twitter feed (kostadis_netapp) will go permanently dark…

I believe NetApp has the most compelling product, vision and technology out there. And our continued success in the market, in spite of a very focused program of unrelenting FUD by our competitors is proof of that. Customers buy high value product, and they reward vendors who build high value product, and they continue to reward NetApp.

Why leave?

My decision to leave is about my need to do different things, and I am very excited about those different things.

Over the last decade I learned that if you have a great team that is focused on adding value to customers, regardless of what the market does, you have a great chance of winning. And in the past decade I have had the chance of working with one of the greatest teams out there.

If I could borrow a phrase, Dave Hitz said in his book that he believes in Magic. And seeing what the teams at NetApp have done over the years, I too believe in Magic. I had and have an unshakeable belief in this company’s ability to create incredible things.

One last word of thanks…

Thanks to everyone who helped create a model company.

10 years at NetApp!

Between writing the requiem of the 15K RPM drive and dealing with a broken sewage pipe at home, I forgot that June 1st, was my 10 year anniversary at NetApp.

The coolest part of my time here has been the people. Listing them all here would be problematic. After all I don't want to encourage recruiters.

The most surprising part of my tenure here has been two things:

1. Everybody keeps saying we're going to go out of business because we're a niche NAS player, and we don't.
2. EMC is still our most ruthless competitor. I mean c'mon guys, just keel over and die already :-)

I've said this before, but the executive team at NetApp continues to be the single most important reason I've stayed here over the years. When I joined in 1999, I wasn't sold on NetApp for the long haul. And after the thespian troupe (acting CEO and acting CFO and acting COO) that mismanaged SGI, I wasn't sold on executives adding any value. But after a day long training session on NetApp where Tom Mendoza and Dan Warmenhoven spoke, I was sold on them, and the company. And nearly 10 years later, the team they've built and continue to build is probably the best retention policy they could have.

Some random fun facts:

1. I spent 9 years working on anything but Data ONTAP.
2. When I joined NetApp I thought SCSI was a cable not a protocol
3. There were fewer than 500 employees when I joined.
4. I remember when we thought it was Network Appliance (s) not Appliance
5. I've been through bubble 1.0, recession 1.0 and now bubble 2.0 followed by recession 2.0... and I am praying for bubble 3.0
6. When I joined 90.8 million in a quarter seemed like a lot of money.
7. Dave's book is still the best and most accurate history of NetApp
8. I managed to serve in the Greek military and debug code at the same time.
9. It took me only 9 years to figure out what Tom Mendoza meant when he said that attitude was everything.
10. I've changed jobs in NetApp 3 times going from Streaming Media, to Storage Management and now data protection, demonstrating that I lack any ability to focus.
11. I've had way too Dr. Evil moments: well let's say we have 100 MBs (crowd laughs), okay 100 GB (crowd laughs), fine folks, throw me a bone, how about 100 PB (crowd stops laughing)
12. I've been here long enough to see disk replace tape (oops not yet)
13. There was a point in time when an outrageous opinion was that Flash would replace disk. Which is especially funny, because now it's outrageous to argue that Flash won't replace disk.

After 10 years, it's tempting to want to jump off the train, but man, if I can forget that this was my tenth anniversary, who knows, maybe there's another ten in me...

Almost famous

NetApp recently released a corporate video.

And at the very end of the video, you can see me wearing my red t-shirt and hat. I am part of a larger group greeting a customer who thanks to NetApp has navigated through the realm of data complexity.

Image vs Logical Backup

What I love about mismanaging my own IT shop (my house), is that I learn first hand what not to do.

So for example, a logical backup scheme (backup incremental files according to some schedule) is great if you want to do a logical restore (restore a single) but sucks wind if you want to do a physical restore.

And yes, I knew that.

So for my laptop I rely on a image based backup utility because I don't really care about the data on the laptop as I keep all of my important data on a Filer.

This does make single file restore painful, but do-able.

Unfortunately, the other day, my wife really needed an image to do a full image restore of her laptop. And unfortunately, the clever logical scheme she was relying on got FUBAR'ed making it impossible to do a restore.

So now we have two backup strategies for her laptop: a logical scheme so she can recover individual files, and a physical schemes so she can recover her entire hard drive quickly. The theory being that any file she needs can be recovered, and she can be back on her feet working without having to wait for some logical scheme to try and piece together all of the files that need to be restored.

The problem with this approach is that it doubles the cost of our storage, never mind the complexity of having two distinct approaches to data protection.

Which reminds me again, why I love snapshots. A snapshot is an image based backup, but it allows you to browse the backup as if it were a logical image, providing logical restore level granularity. But thanks to the magic of SnapRestore, provides a constant time technology that allows you to restore all of the data all at once.

We’re #1, dang

When I was in college, I would always check the US News and World Report college rankings to see where Brown ranked (we’re 16th). The annual ritual was that me and my friends would sit around  and bemoan that XYZ was ranked higher than us but it was only because they had a bigger endowment or because the ranking system was broken. After all we were cooler or you know better. 

And we would read the annoying interview about XYZ’s students who paid no attention to those rankings, because they knew they were at the best school. We knew they paid close attention, but I guess being the top dog let’s you pretend you don’t care.

Maybe this was all a form of buyer’s remorse.

It turns out that the same ritual exists with the Fortune survey of the 100 best companies to work at. Ratings come out, and then you spend half a day trying to figure out how they could be so flawed to pick you as anything other than #1. If you love your job, you want to know why you’re not the best place to work at, because how could things be any better?And then you begin to rationalize that this survey is flawed and broken, because how could ABC be any better than you?

But today we’re #1, and so I am in uncomfortable territory. I don’t know how to act indifferent. But I do know how to say that it doesn’t surprise me that we were voted #1.

And if you ask me why it’s about our culture. What brought that home to me was reading Dave’s book. What was surprising was that the book contained no net new information about the history of NetApp. Oh sure there were details, but there was nothing substantive. I didn’t read the book and think: wow, they lied to me about why they did this or wow, this isn’t what they told us about why this happened. Think about it, a book about 9 years of history written by a key executive had little new information for an employee who had been here for 9 years.

That says something about the kind of culture we have, and perhaps goes to some length to explain our current ranking.

p.s  XYZ was Harvard, and dangnabit, was Harvard this year, again! And I am sure it’s the endowment, it’s always the endowment, cause Brown is just sooooo much better.

How to embarrass your co-workers with a funny book title

There,  How to Embarrass Your Co-Workers With a Funny Book Title, that’s the title of my book. And I have Dave to thank for that.

Someone lent me a copy of his book titled How to Castrate a Bull: Unexpected Lessons on Risk, Growth and Success in Business. Unfortunately they left it on my desk, and unfortunately the part of the title that says How to Castrate a Bull is in black on light orange, and the part that says Unexpected Lessons on Risk Growth and Success in Business  is in a different shade of orange on a orange background. And of course, the book was left on my desk on Tuesday and I only picked it up on Thursday… So for two days people walked by my desk and wondered why I wanted to know about how to castrate a bull. And some of my co-workers were wondering if there was some marital issues …

And now that I am blogging about his book, I wonder if that makes me a brown noser (see page 162). I suppose it does if the opinion of his readers interests Dave.

So what about the book?

Read a book about a place that you’ve been at for 9 years, and the first thing you want to do is see if you made it into the book, and the answer, for me at least, was no. More depressingly my first project, NetCache, didn’t get mentioned either.

But it is about NetApp so some of my favorite moments made it.

My all time NetApp moment is when Tom Mendoza told the sales force:

Your job is to find people who have money and sell them something.

When at the height of the bubble that very much did not seem to be the plan for many of our competitors in the IT space.

Ask me why NetApp is still in business, I really believe it’s because we never lost sight of that simple fact.

So is the book just a collection of cute anecdotes? Is that the point of the book?

No.

So then what is? Maybe it’s a chance to hang with Dave.

At NetApp I’ve had the chance to talk to Dave Hitz at best a hundred times. That is remarkable given the company size and our relative positions. And reading the book reminded me of why I enjoy listening to Dave talk about NetApp.

Reading his book is like getting 192 pages of 1 on 1 time with Dave. You’ll get exposed to things that you’d never expect a geek to know about, like how to actually castrate a bull , and things you would expect a geek to know about, Dungeons and Dragons, and passion about all things related to technology. And you’ll get a few chuckles from some of his jokes. My personal favorite is on page 192.

But even hanging with Dave is not the point about the book.

No, the real point is you get to learn about a remarkable company and how that remarkable company succeeded from a remarkably eloquent man, and the lessons Dave learned along the way.

But what if you are not a card carrying member of the Dave Hitz fan club?

Here’s why you should read it. What makes the book unique in its genre is that Dave takes his engineering mind and applies that mind to understanding the how and the why of NetApp. Not just from grand strategy, but about culture and about management. And what makes that fascinating is that engineering is an art and a discipline. And Dave understands the limits of both. And sometimes he is able to explain the why, and sometimes he admits that it’s just magic.

But even if that’s not good enough, read it because of NetApp.

There are very few companies the size of NetApp (about 1000). And there are very few companies that grew as fast as NetApp to become that big. And the story of why NetApp grew to be that big should be interesting. And if it’s told by someone as fun to listen to as Dave, you’ll find it that much more interesting.

twittering

So I am available on twitter.

My twitter id is kostadis_netapp

If you don't need another thing to learn, you can always just read my tweets at http://www.twitter.com/kostadis_netapp

DR testing

What's the worst time to test your DR strategy?

Why in the middle of a DR event.

Why do I ask?

Well my laptop has a piece of hardware that failed. That hardware failure exhibited itself as a blue-screen-of-death. My experience with software in the past has been that blue-screen-of-death has more to do with the OS being corrupted than the hardware failing (and sometimes the two events are related).  So my strategy has been to recover from an image on a different hard-drive and roll my backups forward.

As I started the re-image process, I observed that I didn't actually know if the re-image would succeed. I had never actually tested the DR process to make sure I had not screwed something up.

Part of the reason I never did the test is that I would need a third hard-drive. If you're testing the DR image is good, you don't want to copy the DR-image over the original image. And another hard-drive costs money, and hardware reconfiguration and copying data takes time.

And then that made me appreciate FlexClone even more as part regular DR management.

Using FlexClone I can create very quickly and at very low-cost a third copy. That third copy can be mounted, validated, and if it works, deleted.

As I sat there watching the data get copied, I wonder whether I was in a fantastical world of pain.

Good news is that I am not.

There is still the small problem of rolling my backups forward, but that's a manageable bit of pain.

But there is the small issue of proper DR procedures.

But just having a DR copy is not enough. The other piece is to have a DR strategy that is written down. In the middle of a DR event you can't think straight. All you want to do is get back to running as fast as possible. You're stressing out that you'll lose all your data. You're worried about lost productivity etc.

So you end up doing whatever cooky idea comes first rather than methodically figuring things out.

Well it turns out that my problem could have been solved by:

1. Examining the BSOD details to see if there is a specific object that is failing
2. Search on google for that object
3. See what corrections are recommended

If I had done that rather than jumping to re-imaging, I would have realized a simple enough fix would have been to extract my hard-drive, put it in a USB case, delete a single offending file, and then continue.

I can assure you my new DR strategy is written down, and one of the key items is don't start until after you've drunk some coffee.

Memories of a different life

In 1996 I graduated from Brown University to go work at SGI on what was a rather interesting technical problem: how to make hybrid scheduling work.

Just to recap, in the early 90’s the UNIX vendors were adding threading to their kernels and there was intense debate over what the right model was:

In the 1 on 1 model, each user thread had a kernel  thread. In the N on 1, all of the user threads were multiplexed on one kernel thread. Finally in the M on N model, many user level threads were multiplexed on a smaller number of kernel threads.

The N on 1 model was the “simplest” to implement, but did not allow the application to exploit multiprocessors. One property of the N on1 model was that synchronization between different threads did not require a context switch.

Although 1 on1 was simple enough to implement it had, what at the time seemed serious, issues. The two most discussed were the memory resources necessary to maintain the kernel data-structures, and the second was the performance implications of making every synchronization operation a kernel operation.

So the M on N model was promoted as a compromise between the N on 1 model and the resource intensive 1 on 1 model. To implement the M on N model, two schedulers were required : a kernel and user level thread scheduler. The central notion was that the kernel scheduler would schedule processors and the user level thread scheduler would assign threads to the available processors represented by the kernel threads. The nice property of this model is that if synchronization was required between user-level threads, the performance would be as good as the N on 1 model.

The challenge of this model was that the thing scheduling the physical resources, the kernel scheduler, had incomplete information and was prone to making mistakes. And the cost of mistakes was idle processors and therefore worse performance.

Ultimately a variety of attempts were made to fix the problem including scheduler activations and scheduler aware synchronization and nano-threads but none good enough to stop the ultimate triumph of the 1 on 1 model.

It was the need to co-ordinate between the two schedulers that eliminated the advantages of the M on N model.

One could almost argue that this was a computer science boondoggle.

So what does this have to do with storage?

At the end of the day the lesson I learned from the whole experience is that having multiple layered resource schedulers scheduling the same resources is difficult to make work.

If you look at the storage stack, what you’ll see is multiple distinct storage layers each managing virtualized pools of storage that they have no visibility into. And what’s worse the thing that actually does the physical resource management has no visibility into what the resources are being used for.

So to me it looks that the storage stack has evolved over the last 20 or so years to look a lot like the hybrid thread scheduling model.

And that is enough to give me reason to pause and think.

Shut Down for the Holidays

This is my last post for 2008.

To all of my readers, be happy and be safe.

And I can’t wait to get going in 2009.