Extensible NetApp: Thinking outside of the box

The DAS Disruption part 6: AMD64

The beginning of the DAS disruption doesn't begin in storage but begins with the introduction of the AMD64 instruction set through the AMD K8 family of processor.

It's funny to think about it now, but in the late 1990's the question of how mainstream x86 servers would deliver 64bit memory architectures was unclear.

For the UNIX servers built on top of 64 bit architectures, this represented a well defined limit of the x86 system architecture, acting as a firewall between the high and low-end servers.

Intel's strategy, which was in retrospect wrong, was to introduce a new processor architecture the Itanium. The new architecture had a new instruction set built on a new instruction set paradigm (EPIC). The problem was that for applications and OS's to leverage the new 64 bit architecture everything would have to change.

The problem was that the commodity space wasn't going to move to the new system architecture until the market moved, which meant that 64 bit computing was out of the cost/performance sweet spot.

If you go back to my earlier posts, and remember this picture:

the lack of a standard 64 bit processor  meant that theindividual boxes could not use 64 bit processor architectures. And more importantly the individual servers could not have memory system designs that assumed very large memories. Which meant that memory on a per server basis was low.

If you were in the storage business this created a firewall that protected your business.

So while the market tried to figure itself out, with demand for increasing memories pushing more and more Frankenstein solutions or requiring major rewrites of software, AMD, pretty much out of nowhere does exactly what everyone wished Intel had done: extend the 32 x86 architecture to support 64 bit addressing.

When Intel saw AMD's traction, Intel released a similar processor, and the death warrant of boutique processors whose only differentiation was 64 bit addressing was signed.

Very quickly it became possible to put 64 bit processors in those small server boxes, and the memory system design necessary to support them effectively also became cost effective.

Which meant that now you could have servers with lots and lots of memory at low cost.

The DAS Disruption part 5: The 90's assault on DAS

Last week I described how UNIX server vendors collapsed as the total value individual servers brought shrunk over time. With a gross margin investment that assumed that individual servers would have a high value, they were stuck with a bad allocation of investment and disappeared into the ether.

Out of that detritus emerged the storage vendors that consumed a chunk of that value.

How did that happen?

I don't want to spend too much time on the business, but on the technology trend.

In the 1990's we had the convergence of a set of technologies. The emergence of low cost commodity compute power, coupled with the emergence of increasingly fast networks and the increasing divergence between CPU, DRAM and Disk speed made it possible to put storage in the network.

Effectively what happened was that, relative to the CPU speed, it was taking more and more time to get data from disk, which meant that waiting for data to be read from disk over the network vs waiting for data to be read from disk locally was lost in the noise.

Let me repeat that for emphasis, the cost of the network hop was rendered irrelevant if you actually had to go to disk.

If disk speeds were slowing in relative terms, the only way you could keep the CPU's fed was to rely on more and more aggressive in memory caching. I talk a lot about this problem in a different context here. So if the only way to improve performance is to add more memory, then the obvious solution is to put more memory on each server. And here's where things get costly. Because as you add more and more memory you increase the cost of the servers. First you increase the cost in the obvious way, more RAM costs more money. Secondly you also increase it in the not-so-obvious way of requiring more and more sophisticated electronics to handle more and more RAM. So even for the servers that don't require the IO performance, you're paying for the infrastructure on the motherboard.

So we have a case of waste and inefficiency which is the mother of innovation.

If we take a deeper dive into the world of Operating Systems, we note that every OS has a thing called the buffer cache. The buffer cache is used to store data that has been read from disk, so as to avoid going to disk later if you need to access the same information twice. Given the temporal locality of application this is a good optimization.

What the storage guys observed is that if you put the buffer cache on the network, sure the latency killed you compared to getting the data locally from memory, but the latency over the network would still be better than going to disk. So the trick was to have a lot more memory on the storage device compared to the server, so that you could deliver on average better performance. And that was made possible because the server had a small amount of memory that had to be shared between the application usage and the storage.

Because the storage was shared across a number of servers, that expensive large memory was amortized across a number of users.

So what was the pitch for shared storage in the 1990's? I don't know what EMC was saying, but I very vividly remember Dan Warmenhoven saying in 1999 on CNBC: We're faster than local disk. And since I can't find the video, let me quote from an interview on Motley Fool

Warmenhoven: And another problem that you probably have on your H drive is availability -- you can't count on it being there. So if I could solve those two problems from a end-user viewpoint -- and I can be very competitive on price -- and simplify the life of your system administrator, it really just makes it so he doesn't have to do anything. Not only would you be happy, but he'd probably be happy. And that's basically the heart of the sales pitch. It's been a real challenge getting the message out. Now I've got to tell you when I make claims like "faster than local disk," I wouldn't be surprised if you were a skeptic because I haven't met anybody who believes that when you first make the claim.

And if you look at the early Symm's they had a lot of memory. I vividly remember stories about competing with some of those original Symm's and noticing that you never had to go to disk because of the amount of memory they put into those devices.

The point is not that shared storage was just faster, but it was architecturally faster AND cheaper.

And it's not just reads

Both the Symm and WAFL offered excellent write performance as well. However, the two companies made fateful choices. EMC chose to layout data in such a way that reads were trivially easy to do. WAFL chose to layout data in such a way that some thought had to be done to do a read, but allowed for excellent write performance. In a world where read workloads dominated, this didn't seem to be too big of a distinction.

Make it faster and cheaper and the world will follow

Shared storage, whether it was NAS or SAN, was making it possible to put much more memory in front of the disk drives than you could ever hope to put in a DAS configuration. The net effect was a market shift.

In the UNIX server world, SAN's replaced DAS very quickly as the only effective way to get performance. EMC coming down from the mainframe world quickly dominated that part of the Universe. I remember that in the 1990's the standard deployment for Oracle was on Solaris using Veritas with EMC.

In another part of the storage world, where customers were struggling with how to get performance out of file servers, NetApp took off. Customers in engineering organizations looking to get more performance moved to NetApp.

And once external storage became prevalent, alternative architectures for server resiliency became possible.

So what made NetApp and EMC possible was a happy convergence of CPU, Memory, Network and Disk speeds that made it possible for customers to get better performance at a lower cost out of a storage device on the network than out of storage inside of the server.

The DAS Disruption part 4: The server collapse in pictures

In parts 2 and 3 on the DAS disruption I wrote a lot about servers and storage, but I had no pictures.

Today I’ll do a recap but include pictures.

1980’s

In the 1990’s UNIX workstation vendors built single processor machines where the sheet metal contained the storage, the memory and the CPU.

Towards the end of the 1980’s and early 1990’s the server vendors increased performance by adding multiple processors. An important consequence is that they had to engineer more bandwidth into their systems, and the amount of electronics increased.

 

By the middle of the 1990’s a UNIX server within a single piece of sheet metal provided all of the performance and availability as a single monolith.

The emergence of external storage.

In the 1990’s external storage showed up. This was enabled by improvements in commodity networking infrastructures, CPU and memory performance, and RAID.

In the server domain, what this meant was that the servers were no longer providing all of the value, but a subset. Another way of expressing this is that customers were now spreading their money between two vendors.

And then the final collapse this decade.

With the emergence of commodity processors that out preformed boutique processors, and the emergence of the commodity networks that provided vast amounts of bandwidth, it became possible to create compute infrastructures out of lots of individual components.

If you’re a server vendor this meant more components were being sold, but the value of each component shrunk because it offered a smaller piece of the overall pie.

And the application reaction

As each individual server provided an increasingly smaller share of the overall performance and availability applications evolved to solve those in software.

As an example, consider Exchange and availability.

In 2000 Exchange leveraged clustering for CPU and Memory availability, but relied on highly available storage with data protection. By 2009, with Exchange 2007, Microsoft was arguing that hardware availability was entirely passe and that all of the availability was to be provided by the application software.

The DAS Disruption part 3: The Role of Storage in the Server Collapse

In part 2 of the DAS disruption series I described the Unix server market collapse. In this companion post I want to drill into that collapse because of the crucial role storage played.

In the 1980's UNIX workstation vendors began to chase ever increasing performance with ever increasing sophisticated system design. At the time two fateful decisions were made. The first was to abandon CISC commodity processor architectures in favor of their own RISC designs, and the second was to put more processors into each box.

Of the two decisions, it was, perhaps, the decision to build increasingly larger SMP systems that was more fateful.

To build a large multi-processor system, it is necessary to build a memory bus that has sufficient bandwidth. In addition to the memory bandwidth, applications expected a symmetric memory behavior, and the electronics necessary to make that work just add to the cost.

In many ways, what the system vendors did was instead of relying on external commodity network infrastructure to hook up processors, they built their own custom networks inside of their own sheet metal. In the 1980's and 1990's they had no choice. The commodity network infrastructure was simply too slow.

But the consequence of both decisions was to increase the total system cost.

As system performance increased, another trend emerged in the UNIX market, increasing system reliability.

Although nowadays UNIX has a well deserved reputation for 6 9's reliability, in the 1980's the UNIX Hater's Handbook had testimonials that mocked UNIX reliability. The authors remarked that it was a good thing that the reboot loop was fast, because after all the system rebooted a lot.

What does this mean?

As systems increased in performance, and increased in reliability an increasingly larger share of the compute requirements of customers were being serviced by a smaller set of big systems. Ironically, the vendors that began life to compete with the monolithic big mainframe had built a mainframe.

As a result making these big systems robust became increasingly more important. Every minute of downtime could cost millions of dollars to a company that depended on a few of these machines.

Enter RAID

As the UNIX vendors began to build increasingly more robust systems, it became  apparent that disk drives were going to be a performance and reliability bottleneck. 

Although disk drives are remarkable feats of mechanical and materials engineering they suffer from two distinct technology challenges. The first is that they are mechanical devices, and therefore suffer the wear and tear of mechanical devices. The second is that because they are mechanical devices their performance does not track Moore's law.

From a UNIX server vendor this created two problems. If you needed performance you needed to combine many of the drives and spread your workload across the drives, but the more disk drives you spread your workload across the increasing likelihood that a single disk drive would take down your system. If you needed capacity you had the same problem. Increasing disk drives gave you more capacity, but reduced reliability.

An obvious solution to the reliability problem was to mirror the disk drives. The problem with mirroring, if you weren't a disk drive manufacturer, was the cost.

Thankfully David Patterson and Randy Katz squared the circle with RAID and made it possible to have both reliability and performance and capacity at a fraction of the cost of mirroring.

What RAID enabled, along with the decline in the cost of compute infrastructure, and the increase in network speeds was the emergence of shared storage.

Let's be precise, adding a shared storage device increases cost because you have to add networking infrastructure and compute infrastructure. That cost can only be justified if that infrastructure is adding tremendous value or it's taking out cost. RAID made it possible for the storage vendors to do both: add value and take out cost.

NAS and SAN vs DAS in the 1990's

In the 1990's NAS and SAN disrupted DAS. The reasons are many, but basically boil down to the fact that EMC SAN and NetApp NAS performed better than the alternatives, either DAS or UNIX file servers.

As the EMC and NetApp devices proliferated, it became clear that they also enabled unique functionality that could solve real business problems.

But we were talking about servers...

If you remember the mental picture I drew earlier, the UNIX system vendors were building these increasingly big systems and making those systems increasingly robust.

What data center architects, and computer visionaries realized was that the UNIX system, like the mainframe, had absorbed lots of commodity elements into a boutique design. As the capabilities of the commodity elements improved, then the value of the boutique elements dropped.

And in the 1990's the UNIX server market got hit by a perfect storm. Storage was outside of the UNIX system. The commodity processors were getting fast enough. The network infrastructure had gotten fast enough.

Of all of those things: CPU, network and Storage, it was,, perhaps, external storage that killed the UNIX server market.

External storage allows you to store your entire state outside of a server. So, if the state is outside of the server, then what's the point of having a highly available server? What are you, protecting? Well an obvious answer is that you don't want your applications to keep stopping and starting. But what if you could solve that problem differently? Suppose instead of having one very reliable computer system, I could have two cheaper computer systems that were less reliable individually but more reliable as a pair? That could work if and only if I didn't have to replicate the state, if only I could share the state ... and oh yes, external storage makes it possible for me to share the state.

And that's what happened.

Application reliability was solved in two ways. The availability of the state was addressed by the storage. The availability of the application was addressed through clustering. Effectively, applications were designed to be able to restart quickly from state that was stored externally to the server.

So in summary

External storage outperformed direct attached storage. External storage made it possible to protect application state outside of a server, which meant that the availability of the server was less important.

Improved commodity processor performance made the value of the RISC processors less compelling.

Improved network bandwidth made it possible to build very large compute infrastructures without requiring specialized memory buses. 

All three of these elements made it possible to replace big iron with lots of smaller less reliable pieces. Except that's not strictly true. The servers were replaced with less reliable pieces, but the storage became more reliable.

The net effect, in the data center, was that server costs were being traded off against software costs, networking costs and storage costs, and given the dollar amounts involved, everyone but the boutique server vendors came out ahead.

The DAS Disruption part 2: The server collapse

With the final disintegration of SGI and the disappearance of SUN the 20+ year history of UNIX open systems computing can finally be written. The reason the story is interesting is because it foreshadows the DAS disruption going on in the data center today.

1980’s

In the 1980’s a slew of UNIX workstation vendors emerged (SUN, SGI, Apollo, and others) that attempted to address the need for increasingly cost effective compute resources that could be used by engineers to solve complex problems.

The early targets were the minicomputers that had displaced the mainframes, but were still too expensive to hand out to individual developers.

The entire period from about 1984 until 1992 was an era of tremendous technological innovation. We had multiple new high-end processors emerge (SPARC, MIPS, PA-RISC, POWER-PC), and multiple distinct operating systems (SUN OS, IRIX, HP-UX, AIX) and multiple distinct 3D graphics architectures as engineers and the market tried to figure out what the right mix of features and cost would succeed.

As the decade evolved, customers observed that these workstations had enough horsepower to become servers for multiple workstation clients.

In effect the UNIX server began it’s assault on the mainframe as a backend server.

Customers who were looking to replace their mainframes noticed that the systems that were running on engineers desktops could be servers for the very same workstations.

1990’s

In the 1990’s the workstation vendors transitioned to becoming server vendors.

SUN, who was able to achieve the greatest amount in the UNIX server market, was the last independent vendor. The Ultra-Enterprise 10000 because of it’s performance, began to displace mainframes, and became the defacto standard for high performance Oracle database infrastructures.

As the UE10k proliferated and became an increasingly important part of the datacenter, customers demanded an ever increasing slew of RASM features.

A direct consequence of that was that the Total Cost of Ownership of Solaris based systems continued to improve over time.

In fact the whole trend of the decade was an increasingly larger chunk of the UNIX server gross margin was being spent on making big SMP systems more and more reliable.

Those big SMP systems had and have upwards of 5 9’s reliability.

The Pentium

As the UE10k took on it’s assault on the high-end of compute infrastructure, Intel built the Pentium processor. The Pentium Processor and it’s successors suddenly created the potential for cheaper workstations and potentially over time, cheaper servers.

The original assault on the UNIX server and workstation market was blunted by the biggest IT bubble in the history of IT bubbles (.com + Y2k) but the seeds of doubt were set.

Customers observed that they could get performance that was as good as the UE10k if all they needed was a single processor. But what if you need more than 1 processor?

Enter Clustering

It’s funny to talk about it now, but back in the 1990’s there was a vigorous debate in the computer industry around whether the future of computation would be clusters or large SMP systems.

The problem with clusters was that they required a change in the programming model, whereas SMP systems did not or so went the argument.

The argument being made was the cost of the networking infrastructure inside of an SMP system was going to make clusters more cost effective and ultimately drive a change in the programming model. In 1998 I remember participating in discussions at SGI where we argued over this exact point.

Clustering solved the performance problem

The emergence of clustering enabled customers to get the performance of those big SMP systems at a fraction of the cost. Furthermore, clustering allowed them to add incremental performance at a much lower cost.

A side effect of clustering for performance, is that it also provided a method for delivering high availability for the cluster.

The use of shared storage provided data availability, and clustering software provide high availability for the CPU and Memory resources.

So for a fraction of the cost, customers were getting the performance and the availability of those large SMP systems. The only problem was that all of the software that assumed an SMP system. And software takes forever to change.

And that TCO advantage, never forget that TCO advantage…

2000’s

Did you see what happened?

SGI was obliterated, and SUN became part of Oracle.

The cost advantage of clustered systems moved the software vendors to adapt their software to those infrastructures. For example, Oracle moved to Linux clustered systems.

And once the software changed, software infrastructures emerged to provide the kind of availability that used to be embedded in the high end servers.

Essentially what happened was that the huge investment in RASM capabilities that took place in the 1990’s to build implicit hardware availability got replaced with a software layer that ran on commodity hardware.

Lessons Learned

Once the performance advantages of SMP systems disappeared, there was a huge incentive to adapt software to the new architecture. Once the software adapted to take advantage of the performance, additional software infrastructures emerged to address the availability.

And once you got the performance and availability the big iron UNIX market shrunk.

But that’s now what killed the big-iron guys. What killed them was that they spent their high gross margin dollars on improving their infrastructure along a dimension that the market no longer cared about. And when they needed the money to invest in a transitioned market, they no longer had the dollars.

The DAS Disruption part 1: Points of Gross Margin

One of the more intriguing parts of following the computer industry is that, especially for hardware vendors, gross margin is a key determinant of the value the market attaches to their value proposition.

Gross Margin is what enables a vendor to invest and derive further value over time. The more gross margin, the more money, the less gross margin the less money.

Given that engineering investments have to be justified in terms of future value, and given the lag between an engineering investment and market delivery, how you spend your gross margin makes or breaks a company.

The problem is that at some point in time your gross margins start to fall. And when that happens, what the market is telling you is that the value you bring to the market is less than what you invested.

And the real problem is that you never know when that will happen.

From a market perspective, the difference between a high gross margin business and a low gross margin business is twofold. The first is a slow down in the rate of innovation, which is fairly obvious. Less dollars for product development means fewer features. The second is a reduction in the number of highly differentiated products. Because differentiation costs money, the amount of differentiation the market is able to sustain is smaller. This doesn’t mean that there is no differentiation, just that the number of differentiated products is smaller.

So what?

Let’s consider the server vendors.

The server vendors, using their high gross margin business, ploughed investment dollars into CPU design, memory design, storage system design and OS design. At some point in time, the value the server vendors ploughed into their hardware stopped being valued by the industry. What that meant was that they had over-invested in things the market was simply not interested in paying for. For example, the market was not interested in paying for their custom high-end processors.

This manifested itself in a collapse of gross margins from north of 50% to less than 30%.

So what happened?

You had a reduction in the number of differentiated offerings, (where are you SGI?), and the amount of differentiation in each product (x86 for everyone, Windows and Linux everywhere!).

In my next post, I want to explore what happened in the server industry over the last 15 years because I believe it will say a lot about the upcoming DAS disruption.

A video of unified storage

After blogging about the value of unified storage, I thought a video would be nice…