Extensible NetApp: Thinking outside of the box

Unified Storage: The Professional Movie

After my award winning amateur performance, NetApp's marketing team decided I need to get my story done professionally.

I like to think of my first video as my screening test, this is the real thing.

In my last post there was a sense of confusion about data protection.

I said NetApp had a single mechanism for doing data protection and touted the value of that.

Some folks wondered: isn't flexiblity a good thing? And I agree. What I meant, was that  for DR, there was a single mechanism for doing data protection regardless of data access protocol (CIFS, NFS, FC, iSCSI).

If you want to hear more about our data protection, I recommend this video

Versatility of SnapMirror -- NetApp -- Storage Efficiency

A video of unified storage

After blogging about the value of unified storage, I thought a video would be nice…

SANscreen Lives!

Chris Evans, The Storage Architect, declared victory over SANScreen with EMC’s latest offering.

And it’s the usual set of claims about how EMC, the company that brought us ECC, is better at delivering management tools and products.

Huh?

Management software is about delivering simplicity to the customer, and EMC has a lot of value in their infrastructure and their offerings, but the value of simplicity is not what comes to mind.

EMC infrastructure continues to suffer from a proliferation of incompatible architectures that provide point value in the data center.

Their management tools seems to be following that trend. They have multiple management consoles for management products. Even their unified storage architecture has no consistent way to manage replication or storage provisioning.

Their answer to the management complexity they create with their infrastructure is to create another management infrastructure.

But enough about His Excellency Mr. Complexity

What really surprised me was the outrageous claim that SANscreen has made very little progress in the years since the acquisition. He derisively cited the progress as meaningless reporting.

Earlier this year I wrote about the recent SANscreen 5.0 release.

Just to remind folks, SANscreen 5.0 added

1. iSCSI support
2. Storage tiering

And what I didn’t talk about then was the support we added for VMware, called VMInsight.

So if managing your SAN and iSAN and managing your space adds no value, well then I don’t know what does.

And yes, I know there is a collection of EMC management tools….

I have no doubt that through a combination of ControlCenter and SAN Configuration Advisor, and who knows what else, there is an overlapping value proposition that you could construct using EMC products.

But that’s the wrong way to go.

In management software, fewer products is better than more products. The fewer products, the fewer workflows, the less training, the simpler operations are.

And SANscreen not only delivers compelling  value but it also delivers the value for heterogeneous infrastructure and in one place.

The increasingly meaningless question of storage utilization

I've been recently engaged in a series of internal discussions about the question of what is storage efficiency.

As a company, NetApp has continued to deliver amazing amount of value by allowing our customers to store huge amounts of data on a relatively small storage foot print and yet gets routinely criticized for our inefficiency.

In point of fact, we create a marketing program to put our money where our mouth is, hell we even run a local marketing campaign, and still we get asked: well how efficient are you really?

Which makes me wonder if we're just barking up the wrong tree.

Storage Efficiency is so last century

SAN storage architectures built around Real Fibre Channel principles are what created the current insanity that frames the storage efficiency discussion.

They declared that the storage problem could be best be solved by creating a storage architecture that first required you to combine lots of storage into a single disk pool, and then statically parceled out the resources into smaller chunks that were handed out to servers. Storage utilization was a measure of how effective you were in using your storage pool. And then the question was of the storage you had allocated how efficient you were in using that.

Great.

So the entire storage industry is married to a storage management model that says that storage efficiency is a measure of how efficiently you used antediluvian architectures based around static partitioning.

Wonderful.

But the problem, and here's the rub Better Than Real Fibre Channel can do things that Real Fibre Channel can not. Because Better Than Real Fibre Channel can use block level reference counts, Better Than Real Fiber Channel can store 255 full copies of data at the cost of that actual unique data within those 255 copies.

So if I have a snapshot, a single NetApp snapshot, I have a full copy of the file system. But instead of being applauded for storing a huge amount of data on a small amount of storage, I have to explain the WAFL overhead.

I'll be blunt, the WAFL overhead allows you to store what to a client appears to be a full copy of the file system on the minimal amount of disk blocks necessary. You wanna do that using Real Fibre Channel, I really hope you like buying disk drives.

And if it was a couple of EMC bloggers I wouldn't mind. But this discussion has permeated the entire computer industry.

And I wouldn't care too much except the question of how we should measure storage system efficiency is not even being discussed because of the fusillade of FUD. And because at the end of the day no one really wants to talk about how inefficient their architectures are.

If they did, storage customers would learn exactly why it's better to use RAID10 to protect your storage instead of replicating your data at the application level.

It's almost 2010 ...

NetApp and Data Domain, and soon others, have demonstrated that a storage system that uses indirection between the disk block and the logical disk block can store more data on less storage than we ever imagined.

And instead of talking about how efficiently we do that, in terms of CPU/Memory, or effectively in terms of the impact to applications we continue to discuss what is the most pointless of conversations:

If I treat my very sophisticated storage array like a set of big dumb disk drives I bought at the local electronics discount store, what's the usable capacity?

Blech.

An update to system management

Steve Klinkner, in his usual understated way, has begun an excellent series on the new system manager
user interface that was just released for our FAS system.

I am, personally, very pleased with the product and am using it to manage my home filer.

I strongly urge folks who are interested to read Steve's posts on this topic.

A new program for Real FC customers who want Better Than Real FC utilization

Last night I wrote about how in every recession there is one good idea (storage virtualization) and one bad idea (fine grained HSM). And in that discussion I observed that whereas HSM is a bad idea whose time should never come, storage virtualization is actually a very good idea that has been poorly executed.

Todays NetApp announcement about extending our space efficiency program should really be the time and place where everyone re-examines the value of storage virtualization.

So what was the underlying problem storage virtualization tried to solve:

• How to use storage that was purchased during the good times more efficiently.

So let's recap, the problems with Storage Virtualization Classic were:

1. The virtualization layer really didn't add any features above and beyond migration
2. The storage efficiencies obtained through pooling of storage were not that significant
3. The vendors supporting it were not able to offer the same customer experience as the underlying storage providers.

The real problem was more insidious than that. Storage virtualization isn't something that you deploy during a recession, rather it should be part of your infrastructure to begin with. But to be part of the infrastructure it had better have compelling features, provide real demonstrable efficiences beyond pooling and the vendor supporting it better have a first class global support organization. Storage virtualization to be useful has to be useful in good and bad times.

Thankfully NetApp has a such a storage virtualization layer (WAFL). Customers who have deployed this technology benefit from a layer that:

1. Adds a whole slew of features like REAL snapshots, FlexClone
2. Really saves space through things like RAID-DP and dedup
3. And is backed by a company that has the heft to provide global customer support.

For those of you who have Real FC, the good news is that you can get the benefits of Better Than Real FC as well through a NetApp v-series. And what's really interesting is this announcement by NetApp that extends our program about space efficiency to Real FC when you deploy v-series.

And why do EMC and HP etc hate this program?

Well as advocates of Real FC, and sadly the biggest provider of underutilized disk out there, this program is really targetted at them. Look the recession has left with customers with disks that are under-utilized or not utilized. How about taking advantage of this program to get some real efficiences from the sunk cost in your Real FC infrastructure?

SANscreen 5.0

Today NetApp announced the latest version of our SANscreen product, SANscreen 5.0. This is a big release that enhances, what I think is, the best heterogeneous SRM product.

Delivering a service oriented view of storage

We are in the process of watching the evolution of storage from an interesting set of devices with capabilities to a service driven infrastructure. What do I mean by service driven infrastructure? Prosaically, it's the aggregation of a set of capabilities, things like performance, reliability, protocol, cost and a commitment that the storage will provide those capabilities, also known as the Service Level Agreement.

For some companies this has already happened. For others, it's a work in progress.

One of the key challenges is how to provide the visibility into the storage tier to ensure that the services promised are the services delivered. 

SANscreen, originally built by Onaro, has continued to evolve and extend it's core value proposition, the ability to quickly and easily see what is going on throughout the storage infrastructure. Using an agentless infrastructure, SANscreen is able to see more, faster and more reliably than any other tool.

Building on top of this very efficient data collection mechanism, SANscreen allows storage teams to monitor the service levels of their storage infrastructure.

So what's new?

The latest release extends SANscreen in three interesting ways:

1. A data warehousing infrastructure that allow storage architects to slice and dice their data
2. Extend SANscreen to iSCSI infrastructures.
3. Deliver the kind of analytic tools necessary to execute a storage tiering strategy.

iSCSI and SANscreen 

It's more than just an FC world out there, and any SRM tool needs to understand iSCSI infrastructure. And SANscreen is no different.

Storage architects that have used SANscreen understand how providing visibility into the infrastructure in an almost real time way makes it possible to mange the storage sprawl. Onaro got it's start clarifying the complex SAN fabrics that were put together over the last decade.

An increasingly important part of the storage landscape is iSCSI. An increasingly challenging part of deploying iSCSI is how to manage the fabric and the storage attached to that fabric.

With the latest release of SANscreen, NetApp is providing a tool that although not providing an end-to-end fabric view, does provide insight into the fabric end-points.

Storage Tiering 

When you're flush, time is money, so you spend money to save time. When you're short on cash you spend time to save money. The trick is to get enough bang for the buck for the time you spend.

Storage analysts talk about the need to have a comprehensive storage tiering strategy that spans the entire enterprise, and then stare dumbly as storage architects ask the inevitable question; how? Hetero SRM tools never really got enough information to be useful, homogeneous tools only told you how to use their stuff effectively.

And don't get me started on the question of FC vs iSCSI.  Never mind what is real FC, how about the basic question of how much FC do I need and how much iSCSI do I need?

To produce a reasonable tiering strategy you need complete, up-to-date, comprehensive data and the analytics necessary to do something with that data. SANscreen has always had the comprehensive, up-to-date and complete part covered, now we added the analytics.

What exactly do I mean by analytics?

1. Provide reports into how capacity is being used

    

2. Map capacity used by business group to some notion of dollars per terabyte.

    

Using both of these mechanisms it's possible to understand the current behavior in your storage infrastructure, and more importantly change it.

Let's be precise, if you want to drive to a particular global view of $/TB, understanding what's going is the first step. But once you have that information, it's really important to act on it. And once you've acted on the information see what happened.

I call this the find, act, verify cycle of storage management.

Too many tools make the find and verify phase an unending adventure. Make those two phases quick and painless, and you'll spend more time doing and less time trying to figure out what to do. Net result, you'll get more efficiency out of your infrastructure.

What's really interesting is that when you combine SANscreen's ability to monitor storage SLA's with the new tiering tools, it becomes possible to correctly identify storage that is on the wrong tier.  And even better, if storage is moved to the wrong tier, then the storage group can discover that and move the storage back without having to deal with the dreaded scream at storage group, storage group fixes problem, scream storage group for creating problem cycle.

But for any tiering strategy to be effective, the consumer has to be involved. And that's where SANscreen's new charge back capabilities come into play.

 

If you measure behavior, and show how the measured behavior is deviating from the goal, you'd be surprised at how quickly the behavior moves towards the goal.

For more information check out the website where you'll find a couple of new whitepapers that describe the product in more detail.

MAD Blog: Onaro is not as much rubbish!

I couldn't help but laugh when I read Martin's blog about ECC.

When I first started working in the storage management space, i tried to understand who the competition was and understand why they were successful.

It was only after I experienced ECC in all of its pointless glory, did I truly appreciate how much opportunity for innovation there was in this space. And it was precisely because of that opportunity that I worked in storage management for about 5 years. One of the things we tried to do at NetApp was to try and build products our customers wanted to use. Products that solved real problems. And I wish I could say we had the perfect product, but that would be foolish. I can say that there were a lot of customers who really liked what they got. I hope some of them didn't think it was rubbish...

But I was delighted to read in a later post that Onaro may be useful. That was indeed high praise from someone so obviously and justifiably frustrated with the state of the art in vendor SRM tools.

When I first met the Onaro guys i was genuinely impressed with what they had. This was a very smart team, with a very cool product underpinned by some very interesting technology that solved a problem that, frankly, EMC and ECC should have solved a years ago.

To this day, Chuck's comments about NetApp's Onaro acquisition remind me of this parable.

IOPS data density and latency

As I started to write about flash and storage tiers, I realized there were some basic underlying concepts that were not clear in my head. So I will first try to clarify those before I get to flash.

Earlier I wrote about the IOPS density of spinning media. The point of the post was this graph:

What this graph shows, other than my inability to use SnagIt to correctly cut-n-paste an image, is that at some point in time the number of hot bytes of data on a disk drive is negligible compared to the number of cold bytes of data.

After I started writing my series on storage tiers, I realized that there is a corresponding notion of IOPS density, but this time for the data not for the media.

So I am going to define a new term:

IOPS data density: The ratio of hot vs cold data for any dataset at any given point in time.

For some datasets, for example, home directories, the IOPS data density is quite low. If you read the paper Meaurement and Analysis of Large Scale File System Workloads by Andrew Leung et al, you'll notice in section 4.2 that the amount of data that is transferred between the client and the server is a fraction of the overall amount of data that is available.

The problem with the term IOPS data density is that as defined, it does not capture the notion of latency. Users of data don't care about IOPS, they care about the latency (or time) it takes to perform an operation.

I'll state that the performance of an application, where an application may be a database or an NFS client,  is related to the average performance of the average IO. I'll also state that applications expect consistent performance rather than variable performance and perform worse as variability increases.

So to expand my definition a little bit:

IOPS data density: The ratio of hot vs cold data for any dataset at any point in time. The application using the dataset expects the hot data to have a sufficiently low latency that is reasonably uniform across all of the hot data.

Next we explore the different kinds of IOPS data density and how they map to the different storage tiers.

Unified Storage Architecture: Archiving

NetApp today announced product integration with CommVault, Quest and Mimosa in the archival space. What make this announcement interesting is that it demonstrates the value of a unified storage architecture. Earlier this year, I defined the properties of such an architecture, but never really articulated the customer value. That's what I hope to do today.

So what is an archive?

Talk to someone about archive, and you get a hundred different answers.

In practice, an archive is whatever you want it to be. But there are a couple of key properties that are meaningful and relevant to this discussion.

1. An archive contains the last remaining copy of the data.
2. Finding data within an archive must be fast enough.
3. Data in an archive will survive for a very long time
4. The probability of accessing any bit of data in an archive is very low, but the archive as a whole will be routinely accessed.
5. Getting the data out of the archive into a format that can be accessed is very important.
6. An archive must survive operational, hardware and site failures.

Two kinds of an archive: passive and active

There are really two kinds of an archive, a passive and an active archive. A passive archive  is an archive that is created through the pre-existing backup process. As I observed in earlier post, the backup process is used to create archives by cranking up the retention period. In a tape world, where the cost of storing tape is very low, this has been a remarkably successful technology at the expense of large amounts of wasted space.

However the time to find data and the time to recover data pushes IT architects to look at disk for a piece of or all of their archive needs.

But the old model of archive everything just doesn't work. If you archive everything forever, you end up retaining lots of data which costs money because disks have to keep spinning and even if they can be spun down, consume rack space.

So we have seen the emergence of active archives, like CommVault, Quest and Mimosa, where specific pieces of data are moved from wherever they are to the archive. In this kind of archive, the move is either a cut and paste, or a copy and paste of the data. The choice of cut and paste or copy and paste is not meaningful for the rest of this discussion.

Architecture of an archive solution

At the core every archive solution has four basic components: an application you are archiving, a database, a content index and a repository. Looking at the storage, the application, database and content index require fast performance whereas the repository requires good enough performance but more importantly, must be affordable and reliable. Some archives will also require some compliance based storage to meet regulatory requirements.

A picture of such a solution would look this:.

 

 

Each colored box represents a specific class of storage that needs to be deployed and managed.

But in addition, each class of storage must be protected against site failures, so the real picture looks like this:

 

And what the picture does not show is the complex co-ordination required to ensure that the archive and application are available on the DR site.

The challenges of a traditional storage architecture.

Looking at the last picture, it's pretty clear that a fairly complex storage infrastructure must be created.

Using traditional schemes, you either rely on a single architecture that is stretched across all of your tiers, or bite the complexity bullet of managing four distinct tiers with four distinct management paradigms and four distinct replication schemes.

But even if you could bite off the four architectures, there is a hidden cost of such an approach called over provisioning. If the underlying disk can not be effectively load balanced across different uses, guess wrong and the disk sits idle consuming power and cooling. Any unused disk represents lost money. Especially if you consider the cost of disk and the capacity growth of disk, buying more disk than you need is a horribly inefficient.

Unified Storage Architecture

NetApp, on the other hand, has a different answer. Our unified storage architecture delivers the benefits of right-sized performance and availability and cost to your archive infrastructure.

The picture below shows a single controller for clarity, but you could imagine multiple controllers if you needed additional performance or capacity. 

The operational simplicity of a single storage infrastructure with a single management paradigm is obvious.

What's not so obvious, is that there is a real cost advantage to this scheme. Through the power of things like thin provisioning, flexible volumes and the ability to select which datasets are or are not deduplicated, the underlying space can be effectively load balanced across the different uses. As a result, the amount of storage required can be right-sized rather than over-provisioned.

And what about data protection?

The real power of a unified storage architecture, however, comes to play when you consider the operational challenges of protecting the archive.

Consider this archive application which is running:

At some point in time you want to replicate the archive to a remote site. Using our snapmirror technology, a single mechanism and a single management process creates the remote replica:

And what we don't show here, is that the remote archive can be made up of different hardware to further cost reduce the overall solution.

This approach of replicating a set of snapshots as a cascade is, you guessed it, is called cascading snapshots ...