Extensible NetApp: Thinking outside of the box

Go NetApp!

After a decade at NetApp, I am moving on to a different adventure.  Which means that my blog and my twitter feed (kostadis_netapp) will go permanently dark…

I believe NetApp has the most compelling product, vision and technology out there. And our continued success in the market, in spite of a very focused program of unrelenting FUD by our competitors is proof of that. Customers buy high value product, and they reward vendors who build high value product, and they continue to reward NetApp.

Why leave?

My decision to leave is about my need to do different things, and I am very excited about those different things.

Over the last decade I learned that if you have a great team that is focused on adding value to customers, regardless of what the market does, you have a great chance of winning. And in the past decade I have had the chance of working with one of the greatest teams out there.

If I could borrow a phrase, Dave Hitz said in his book that he believes in Magic. And seeing what the teams at NetApp have done over the years, I too believe in Magic. I had and have an unshakeable belief in this company’s ability to create incredible things.

One last word of thanks…

Thanks to everyone who helped create a model company.

Ex-chain smokers

Right after the acquisition of DDUP was announced, NetApp had it's quarterly all hands.

Dan Warmenhoven invited Frank Slootman, CEO of DDUP, to come and speak. His speech was recorded and is now available on youtube.

Frank Slootman talks about NetApp, Data Domain, the rationale for the acquisition, and culture.

And the rationale is still pretty obvious, Data Domain allows NetApp to penetrate into more accounts faster by picking a specific entry point where they have compelling technology.

But there is one funny moment when he describes ex-EMC employees as ex-chain-smokers that needed rehabilitation.

10 years at NetApp!

Between writing the requiem of the 15K RPM drive and dealing with a broken sewage pipe at home, I forgot that June 1st, was my 10 year anniversary at NetApp.

The coolest part of my time here has been the people. Listing them all here would be problematic. After all I don't want to encourage recruiters.

The most surprising part of my tenure here has been two things:

1. Everybody keeps saying we're going to go out of business because we're a niche NAS player, and we don't.
2. EMC is still our most ruthless competitor. I mean c'mon guys, just keel over and die already :-)

I've said this before, but the executive team at NetApp continues to be the single most important reason I've stayed here over the years. When I joined in 1999, I wasn't sold on NetApp for the long haul. And after the thespian troupe (acting CEO and acting CFO and acting COO) that mismanaged SGI, I wasn't sold on executives adding any value. But after a day long training session on NetApp where Tom Mendoza and Dan Warmenhoven spoke, I was sold on them, and the company. And nearly 10 years later, the team they've built and continue to build is probably the best retention policy they could have.

Some random fun facts:

1. I spent 9 years working on anything but Data ONTAP.
2. When I joined NetApp I thought SCSI was a cable not a protocol
3. There were fewer than 500 employees when I joined.
4. I remember when we thought it was Network Appliance (s) not Appliance
5. I've been through bubble 1.0, recession 1.0 and now bubble 2.0 followed by recession 2.0... and I am praying for bubble 3.0
6. When I joined 90.8 million in a quarter seemed like a lot of money.
7. Dave's book is still the best and most accurate history of NetApp
8. I managed to serve in the Greek military and debug code at the same time.
9. It took me only 9 years to figure out what Tom Mendoza meant when he said that attitude was everything.
10. I've changed jobs in NetApp 3 times going from Streaming Media, to Storage Management and now data protection, demonstrating that I lack any ability to focus.
11. I've had way too Dr. Evil moments: well let's say we have 100 MBs (crowd laughs), okay 100 GB (crowd laughs), fine folks, throw me a bone, how about 100 PB (crowd stops laughing)
12. I've been here long enough to see disk replace tape (oops not yet)
13. There was a point in time when an outrageous opinion was that Flash would replace disk. Which is especially funny, because now it's outrageous to argue that Flash won't replace disk.

After 10 years, it's tempting to want to jump off the train, but man, if I can forget that this was my tenth anniversary, who knows, maybe there's another ten in me...

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

Have and eat your cake.

One of my co-workers, Vikram Shukla, sent me this picture of the cake from our party to celebrate being voted #1.

One of those rare cases where you can have and eat your cake.

We’re #1, dang

When I was in college, I would always check the US News and World Report college rankings to see where Brown ranked (we’re 16th). The annual ritual was that me and my friends would sit around  and bemoan that XYZ was ranked higher than us but it was only because they had a bigger endowment or because the ranking system was broken. After all we were cooler or you know better. 

And we would read the annoying interview about XYZ’s students who paid no attention to those rankings, because they knew they were at the best school. We knew they paid close attention, but I guess being the top dog let’s you pretend you don’t care.

Maybe this was all a form of buyer’s remorse.

It turns out that the same ritual exists with the Fortune survey of the 100 best companies to work at. Ratings come out, and then you spend half a day trying to figure out how they could be so flawed to pick you as anything other than #1. If you love your job, you want to know why you’re not the best place to work at, because how could things be any better?And then you begin to rationalize that this survey is flawed and broken, because how could ABC be any better than you?

But today we’re #1, and so I am in uncomfortable territory. I don’t know how to act indifferent. But I do know how to say that it doesn’t surprise me that we were voted #1.

And if you ask me why it’s about our culture. What brought that home to me was reading Dave’s book. What was surprising was that the book contained no net new information about the history of NetApp. Oh sure there were details, but there was nothing substantive. I didn’t read the book and think: wow, they lied to me about why they did this or wow, this isn’t what they told us about why this happened. Think about it, a book about 9 years of history written by a key executive had little new information for an employee who had been here for 9 years.

That says something about the kind of culture we have, and perhaps goes to some length to explain our current ranking.

p.s  XYZ was Harvard, and dangnabit, was Harvard this year, again! And I am sure it’s the endowment, it’s always the endowment, cause Brown is just sooooo much better.

How to embarrass your co-workers with a funny book title

There,  How to Embarrass Your Co-Workers With a Funny Book Title, that’s the title of my book. And I have Dave to thank for that.

Someone lent me a copy of his book titled How to Castrate a Bull: Unexpected Lessons on Risk, Growth and Success in Business. Unfortunately they left it on my desk, and unfortunately the part of the title that says How to Castrate a Bull is in black on light orange, and the part that says Unexpected Lessons on Risk Growth and Success in Business  is in a different shade of orange on a orange background. And of course, the book was left on my desk on Tuesday and I only picked it up on Thursday… So for two days people walked by my desk and wondered why I wanted to know about how to castrate a bull. And some of my co-workers were wondering if there was some marital issues …

And now that I am blogging about his book, I wonder if that makes me a brown noser (see page 162). I suppose it does if the opinion of his readers interests Dave.

So what about the book?

Read a book about a place that you’ve been at for 9 years, and the first thing you want to do is see if you made it into the book, and the answer, for me at least, was no. More depressingly my first project, NetCache, didn’t get mentioned either.

But it is about NetApp so some of my favorite moments made it.

My all time NetApp moment is when Tom Mendoza told the sales force:

Your job is to find people who have money and sell them something.

When at the height of the bubble that very much did not seem to be the plan for many of our competitors in the IT space.

Ask me why NetApp is still in business, I really believe it’s because we never lost sight of that simple fact.

So is the book just a collection of cute anecdotes? Is that the point of the book?

No.

So then what is? Maybe it’s a chance to hang with Dave.

At NetApp I’ve had the chance to talk to Dave Hitz at best a hundred times. That is remarkable given the company size and our relative positions. And reading the book reminded me of why I enjoy listening to Dave talk about NetApp.

Reading his book is like getting 192 pages of 1 on 1 time with Dave. You’ll get exposed to things that you’d never expect a geek to know about, like how to actually castrate a bull , and things you would expect a geek to know about, Dungeons and Dragons, and passion about all things related to technology. And you’ll get a few chuckles from some of his jokes. My personal favorite is on page 192.

But even hanging with Dave is not the point about the book.

No, the real point is you get to learn about a remarkable company and how that remarkable company succeeded from a remarkably eloquent man, and the lessons Dave learned along the way.

But what if you are not a card carrying member of the Dave Hitz fan club?

Here’s why you should read it. What makes the book unique in its genre is that Dave takes his engineering mind and applies that mind to understanding the how and the why of NetApp. Not just from grand strategy, but about culture and about management. And what makes that fascinating is that engineering is an art and a discipline. And Dave understands the limits of both. And sometimes he is able to explain the why, and sometimes he admits that it’s just magic.

But even if that’s not good enough, read it because of NetApp.

There are very few companies the size of NetApp (about 1000). And there are very few companies that grew as fast as NetApp to become that big. And the story of why NetApp grew to be that big should be interesting. And if it’s told by someone as fun to listen to as Dave, you’ll find it that much more interesting.

Mea culpa or why I am doing modeling.

I've just spent the last week trying to come up with an analytical model for backup and restore.

At some point, other than the sheer intellectual curiosity, it's reasonable to ask why?

The value of primary storage deduplication surprised me. I thought, like everyone else, that this was a backup technology. 

In fact, in many ways, I thought deduplication was snake-oil. An over-hyped, over-promoted technology that would go away once sanity prevailed. Instead it got even more popular while I was trying to explain to people that this was being oversold.

But something was bothering me. Snapshots are a form of deduplication. NetApp snapshots on primary storage systems are widely popular. I didn't understand deduplication, and I thought I understood snapshots.

Am I insane or is the world insane?

After a while though, I could not ignore the popularity. Something very profound was going on. And what was more interesting, was that primary deduplication was having cascading effects on how people were thinking about backup. This told me that something really big was going on, something I did not understand.

One of my college professors ,Tom Dean, said that if the experimental evidence does not match our intuition we need more and better theory not less.

So out comes the "math" mire like basic arithmetic or just plain old common sense.

So why the modeling?

Because if something this big is happening, it should be amenable to analysis. And analysis will help us both understand why the new technology is so popular but perhaps allow us to predict new consequences of that technology.

That analysis should also help us understand snapshots, and it should also help  us understand what kinds of compression we need, and the nature of the compression. The analysis should also explain what tradeoffs need to be done, and where there are land mines.

So the modeling, and analysis was done to help me understand what is going on. And part of the reason this has been so slow and evolving, is that it takes time for me to figure this out.. I am more or less working this out in real time.

Recruiting at Brown

This is a bit of a puff piece. Full of nostalgia.

So here I am in Providence, Rhode Island at the Starbucks across the street from the CIT. In 1992 when I first came to Brown and sat at Ocean Roasters, I remember ordering my first coffee and being bewildered that there were choices: java, sumatra, etc.

Many years later and I am still bewildered.

But what I love about recruiting is talking to some of the best and the brightest.

And I once again got a chance to interact with some of the best and brightest about what I love about NetApp.

And after 12 years of recruiting at Brown it is still fun to do.

Because talking about your job is always easy when you love it. And I've been lucky to always love my job.

Self-Censorship and EMC Marketing

One of the things I was looking forward to in this blog was the opportunity to get down and dirty with some of the technology I am very familiar with.

I was hoping to discuss the limitations of what our system could do because that would be interesting to our customers, and I suspect the storage industry at large.

But discussing limitations is dangerous. Someone can misquote you and turn what is a nuanced argument into prima-facie evidence that our system behaves in a particular way. To be able to have these discussions you need to assume that your opponents in the market place will play fair. In particular, you have to believe that they will not take a sentence out of context and use it in a technical report that they will then distribute to your customers to spread misinformation.

Unfortunately for me, and for you, EMC's marketing seems to be specialists in the art of taking quotes out-of-context and using them to bludgeon competitors. Their latest assault on usable capacity is yet another example of that. And if that was an isolated example, maybe I could ignore what happened. But I can not. Over the past several years I have been personally involved in correcting disinformation spread by EMC using out-of-context quotes, out-of-date information and plain old fashioned half-truths.

I don't want to be drafting explanations, handling customer calls, and wasting a lot of my and sales and customer time to deal with EMC Marketing generated FUD.

And if you think it's paranoia, when I wrote Rethinking Deduplication, within a couple of days, I had NetApp field person complaining that EMC was using my blog to argue that NetApp was arguing that Dedup was not intended for primary storage. I was forced to write a new blog entry, Deduplication, it's not just about backup just to clarify what I was saying.

Look I don't know if Chuck's right about NetApp's marketing. He goes to great length to ridicule whatever we do. And he is a VP in marketing, and I am an engineer, so that makes him the expert, not me. Maybe EMC's marketing is the right way to go.

But I do know that this style of marketing has forced a considerable amount of self-censorship on this particular blog writer, and that sucks.