Mea culpa or why I am doing modeling.

I've just spent the last week trying to come up with an analytical model for backup and restore.

At some point, other than the sheer intellectual curiosity, it's reasonable to ask why?

The value of primary storage deduplication surprised me. I thought, like everyone else, that this was a backup technology. 

In fact, in many ways, I thought deduplication was snake-oil. An over-hyped, over-promoted technology that would go away once sanity prevailed. Instead it got even more popular while I was trying to explain to people that this was being oversold.

But something was bothering me. Snapshots are a form of deduplication. NetApp snapshots on primary storage systems are widely popular. I didn't understand deduplication, and I thought I understood snapshots.

Am I insane or is the world insane?

After a while though, I could not ignore the popularity. Something very profound was going on. And what was more interesting, was that primary deduplication was having cascading effects on how people were thinking about backup. This told me that something really big was going on, something I did not understand.

One of my college professors ,Tom Dean, said that if the experimental evidence does not match our intuition we need more and better theory not less.

So out comes the "math" mire like basic arithmetic or just plain old common sense.

So why the modeling?

Because if something this big is happening, it should be amenable to analysis. And analysis will help us both understand why the new technology is so popular but perhaps allow us to predict new consequences of that technology.

That analysis should also help us understand snapshots, and it should also help  us understand what kinds of compression we need, and the nature of the compression. The analysis should also explain what tradeoffs need to be done, and where there are land mines.

So the modeling, and analysis was done to help me understand what is going on. And part of the reason this has been so slow and evolving, is that it takes time for me to figure this out.. I am more or less working this out in real time.