Extensible NetApp: Thinking outside of the box

Thin Provisioning and Fragmentation or what is the CLARiiON defrag utility?

Thin Provisioning (or virtual provisioning) is the most obvious feature that Better Than Real Fiber Channel enables as a result of it’s architecture.

To repeat, in a Real Fiber Channel architecture, the physical disk blocks and the logical disk blocks are one and the same. As a result the logical disk layout reported through the LUN is the physical disk layout.

In a Better Than Real Fiber Channel architecture, the physical disk block has no correspondence to the logical disk layout. Better Than Real Fiber Channel systems are able to achieve better performance, and provide new and novel features as a result.

A good example of a feature enabled by Better Than Real Fiber Channel is Thin Provisioning.

Thin Provisioning is the storage industry designation for the process of making physical disk block allocations on demand rather than pre-allocating those physical disk blocks. By making the physical disk block allocations on demand, it is possible to reduce the total amount of physical storage required. The theory behind thin provisioning is that actual resources requested are typically greater than the resources required.

An important consequence of thin provisioning is that since physical disk block allocations happen on-demand, the storage array can no longer guarantee physical disk layout. More succinctly, thin provisioning inherently creates fragmentation of the storage array.

Because NetApp has a 15 year old history in understanding how to deal with fragmentation and deliver superior performance, we were able to publish benchmark results with Thin Provisioning enabled and still beat a Real Fiber Channel system.

The point here is not to mock the competition, but to observe that the ability to perform well with a disk layout that is different than the logical layout the client assumes is a core capability that storage systems must be able to handle, and handle well.

Another important point to realize is that because handling fragmentation and delivering high performance is a hard problem, not every thin provisioned architecture offers the same capabilities.

For example, to the best of my knowledge, the V-Max architecture supports 1GB chunks as their thin provisioning granularity. In other words, if a disk block within a 1GB region is written to, the entire GB is allocated. Doing this limits the amount of fragmentation that can occur on disk. This is a reasonable strategy if your architecture does not support 4KB chunks like WAFL. But that implies that you get less space savings from thin provisioning.

Circling back to the beginning, though, the CLARiiON finally has added support for thin provisioning. So I must ask, how do they deal with fragmentation? How do they perform as their file system ages?

And my point isn’t to mock the CLARiiON but to observe that what was once a discussion about a peculiar NetApp challenge is now a broader conversation that the entire storage industry has to engage in.

Image vs Logical Backup

What I love about mismanaging my own IT shop (my house), is that I learn first hand what not to do.

So for example, a logical backup scheme (backup incremental files according to some schedule) is great if you want to do a logical restore (restore a single) but sucks wind if you want to do a physical restore.

And yes, I knew that.

So for my laptop I rely on a image based backup utility because I don't really care about the data on the laptop as I keep all of my important data on a Filer.

This does make single file restore painful, but do-able.

Unfortunately, the other day, my wife really needed an image to do a full image restore of her laptop. And unfortunately, the clever logical scheme she was relying on got FUBAR'ed making it impossible to do a restore.

So now we have two backup strategies for her laptop: a logical scheme so she can recover individual files, and a physical schemes so she can recover her entire hard drive quickly. The theory being that any file she needs can be recovered, and she can be back on her feet working without having to wait for some logical scheme to try and piece together all of the files that need to be restored.

The problem with this approach is that it doubles the cost of our storage, never mind the complexity of having two distinct approaches to data protection.

Which reminds me again, why I love snapshots. A snapshot is an image based backup, but it allows you to browse the backup as if it were a logical image, providing logical restore level granularity. But thanks to the magic of SnapRestore, provides a constant time technology that allows you to restore all of the data all at once.

Understanding WAFL Performance: How RAID changes the performance game.

In my last post I explained why PAM improves performance, but I didn’t answer the other question of why WAFL kicked ass at low utilization, and more interestingly why is the PAM a read-only cache?

A number of our competitors have correctly pointed out that their cache’s are read-write and have asked, fairly, is our cache the wrong kind of cache?

To answer the question I need to talk about RAID, something I dutifully ignored in my last set of posts.

Understanding how RAID affects storage performance

This is not a primer on RAID but rather an attempt to explain how designing a high performance storage system is impacted by RAID.

Yesterday I pointed out that a Real FiberChannel RAID array can defer writes in memory such that the writes are sequential. I then said that WAFL is different, but what I described and showed didn’t seem that different. So what is different?

When you’re writing data to disk using RAID, it’s not sufficient to just write a disk block to a specific location, but you must also update the RAID parity information to allow reconstruct to take place.

So let’s assume you have this picture:

Each column represents a disk drive, and each row represents a RAID stripe.

The application sending writes to the array will force the Real FiberChannel array to update locations A1, B3, D2, D4, F3 and H1.

However if you modify location A1, you also have to modify location AP, ditto for BP, DP, FP, and HP. But because AP etc are parity computations for the whole stripe to modify them you have to first read them.

So write updates A1, B3, D2, D4, F3 and H1 really turn into 6 reads and 12 writes.

Alright, but if we look at this a little bit more carefully, because of the write cache, I don’t really need 6 reads but 5 reads. Consider stripe D, since I know D2 and D3 are updating the same stripe I can only do one read for the entire stripe.

Now that’s why write cache is good!

The more writes the storage array defers, the more likely the storage array get writes to the same stripe, the fewer additional reads the storage array will end up doing.

But WAFL is better, after all it’s Better Than Real FiberChannel

WAFL relaxes the requirement that Real FiberChannel imposes that an update to the same logical disk block must be to the same physical disk block.

So for WAFL here’s what the would happen.

Suppose the original layout looked liked this

 

Then a sequence of updates to A1, B3, D2, D4, E3, F3, G3 and H1 happen. What WAFL will do is instead of over-writing the same blocks it will write the data as a full stripe in I1 and J2 and then free up the original blocks. So for WAFL this sequence of 8 writes would appear as one write where you were sequentially writing more data.

The net layout will look like this:

So that’s why WAFL doesn’t need as much write cache

Unlike Real FiberChannel, WAFL only has to wait for enough data to write out a single stripe to disk. If there is enough free space, WAFL’s write performance is constrained by Data ONTAP’s sequential write throughput.

So that’s why WAFL performance kick’s ass!

A Real FiberChannel array has to work harder than WAFL (more disk IO’s) when there is a lot of free space. That’s why a Real FiberChannel array will benefit from additional write cache. However, there is a limit to how much you can cache, and ultimately you will end up doing more work.

A Better Than Real FiberChannel array like WAFL, will do very little work when the array has plenty of free space and therefore perform much better. A side effect, though, is that as the array free space shrinks, WAFL’s will have to do more work to find free space, and that more work will translate into lower performance. This is why as the storage utilization increases WAFL performance reaches the same level as other storage arrays.

WAFL Performance: Disk Drives, Latency, Throughput and IOPS

Cisco decides to explode the server market, and I want to talk about disk drives. What's wrong with me?

Last week I spent time trying to disentangle notions of randomness and sequential access. And after a long meadering presentation, what I think I landed on was a useful and meaningful definition namely:

A device is sequential if the software accessing the device can detect differences in access time depending on how data is accessed. The sequential access is the fast access, the random access is the slow access

An important consequence of this definition is that because time moves forward for every man, the number of random and sequential accesses that can be performed over any time period are effectively time/T for random access or time/D for sequential access. But disk drives have an interesting property that changes this a little bit. Because of the way disk drives work, to move between two random location you need to, in general, perform a sequential operation, so the sequential operation is in, fact, free once you've done the random operation.

Now it turns out that because of the peculiar relationship between random and sequential operations, the only interesting measure for disk is the number of random operations, and that's what we call IOPS.

Although IOPS are important, another user-visible aspect of disk drives is their latency and throughput. Now the latency of a disk drive is pretty horrible compared to it's throughput. So there is a lot of value if you can hide the latency of a disk drive. 

The core trick to improving latency is to avoid going to disk as much as possible. Imagine if you could read all of the data into memory. Then the latency of the data access is the latency of accessing memory! The problem with storing everything in memory is cost, and the annoying property that memory is volatile, so you have to go to disk.

Now it turns out, that you only have to go to disk when you write data and when data that you need is not in memory.

So now let's consider the picture I am painting for you. A disk drive has a certain number of IOPS. Those IOPS need to be used to write data to disk and to read data from disk that is not in memory. If you can cache more data in memory then you can improve your write throughput because you can have a larger share of the IOPS be for write operations. The interesting side effect, is that as you reduce the number of read operations, you reduce the number of total IOPS you require for a given workload!

So the question is then is if the disk drives are all the same, why does WAFL write performance kick ass? After all given the same amount of memory, and the same workload, why does WAFL perform so well? And why does a relatively small amount of memory allow WAFL to cut the total number of spindles it requires in half and sustain the same throughput and latency?

Next post, promise.

Understanding WAFL Performance: Disk Drives

Disclaimer:

If you've been working in the storage industry, bought storage or in general have a good understanding of storage system design, the rest of this blog entry is really uninteresting. So skip ahead to the next post. If you want to read about disk drive performance and storage system design at a fairly basic level, then read on. This post will serve as a bridge to my follow on post.

The world is really sequential

My first computer was an IBM PC circa 1981. That computer had among many other things a 5.25" floppy drive. And I remember reading that the floppy drive unlike the tape drive was a random access media.

 

But if you looked at the floppy you noticed something really bizarre, that it had this spinning platter that spun like LP on a round table. When I compared it to the tape drives my friends had on their

Commodore 64's I couldn't for the life of me figure out why a tape drive was sequential but a floppy was random access. In both cases you had to move this physical media backwards and forwards.

Confused, perplexed and bewildered, I asked a buddy of my dad's why the floppy was random and the tape was sequential, and his response was simple: speed of access. Of course, I had no idea what he was talking about, I nodded sagely filed away the story in my brain, and went on my merry life.

Some 20+ years later it became fairly obvious what he meant. Something Dave explained in an earlier post, but I'll repeat in my own words.

The pompous definition

Every technology that can be used to store data cost-effectively (memory, disk, tape, flash) has the property that getting some data is faster than getting other data.

More precisely, suppose it takes some time T to seek to a specific location to get some data, if the next piece of data is sequentially located, it will take time D to seek to that location where D is very much less to T.

From the picture on the left, it takes time T to get to the place where the 1 is stored, but it takes time D to read the 0 next to the 1.

A medium appears to be random access if the time it takes for an application to make each request of data is approximately on the order of T. If the time to make a request is T, then the application doesn't care about how the data is layed out and can make any request whenever it wants.

On other hand if the time it takes an application to request data is on the order of D or even less than D, then the medium the data is stored appears to be sequential. The reason it appears to be sequential, is because the application that is accessing the data will change it's requests to access the data to be sequential so as to get better performance. In other words, the way the data will get requested will be a relatively small number of random seeks followed by a much larger number of sequential seeks.

Huh?

Every media has a time it takes to get to where the data is located (seek time), and a time to get the data.  But if you drill down a little bit further, it turns that seeking to a completely different location takes more time than just getting the next piece of data. Using a DVD as an example, playing the next frame frame of the movie takes a lot less time than skipping to the next scene. In both cases you have to seek to get the next frame, just in one case you have to seek for a less time.

So fine.

So given that in both cases there is seek, what makes something sequential and something random? The device asking for the data. Consider a modern processor that can request billions of bits a data second. If the thing feeding the data can not get feed the processor at that rate, then the speed of the processor is defined by the feeder. If the seeks can't get any faster, then you need to minimize the number of seeks. If you minimize the number of seeks, then you're basically operating the device as if it were a sequential medium, because you seek to a specific location and then try and get as much information as fast as possible by reading the next piece of information that is located at the next sequential location.

So what does this have to do with disk drives?

Well it turns out that disk drive performance, today, is so slow compared to processor speeds, that the entire design of high performance storage systems is how to convert random access operations into a set of sequential operations.

In other words, whereas the application wants to seek all over the disk drive, the storage system using a variety of tricks minimizes the amount of disk movement that is necessary.

There are two basic approaches depending on whether you are doing a read or a write.

Read

Most applications exhibit a certain locality of reference. What that means is that the next piece of data they require happens to be close to the previous piece of data. Using that principle, storage systems use large amounts of memory to store data to avoid going to disk, and when they go to disk read more than was requested.

Write

It turns out that how to make write operations sequential is a bit of black magic. And that will be the topic of my next post.

Flash Point: Missing the point on TLAs

Martin G, as he does so well, cut through the recent Flash brouhaha by pointing out that stuffing an SSD with a FiberChannel connector into a storage array is more a statement about the sad state of storage innovation than any miracle of modern engineering.

So what about NetApp?

NetApp's plans were announced in a whitepaper that described our strategy for integrating flash into our storage systems.

The PAM announcement, even though it's not Flash, is a proof point, another execution point on our roadmap to our destination. .

NetApp's recent announcement on TMS RAMSAN, doesn't really count as the great leap forward either. In my mind it's just more proof/evidence of that the roadmap.

But in spite of what  I just said, I think the TMS RAMSAN announcement is interesting in and of itself.

Why?

Because of the value of our software.

Our software lets you use less storage, make faster copies and keep more backup images than any other storage software out there. And flash just makes it all go faster. And by making it go faster, you're allowing customers who never thought they could benefit from our storage software benefit from it. And all of a sudden they are keeping more backups, using less storage and making faster copies than they were before. And all of a sudden that creates a new set of opportunities for them.

Because flash isn't a destination. Flash is an enabling technology.

The question is, enables what?

The short answer is it enables much faster reads at a fraction of the cost of DRAM.

But that isn't that interesting either.

What's really interesting is how does software adapt itself to Flash.

And no one knows the answers to those questions.

But here's what I do know, everytime you make something faster and cheaper and more plentiful, innovative people figure out a way to exploit that plenty to do new things.

So let me give you an example of what more faster reads enables.

Add more memory to our systems, and all of a sudden you don't need a FiberChannel disk tier in some infrastructures. And that suddenly frees up more money because you're buying cheaper disk, and it suddenly frees up time because you're no longer doing tiering. And that suddenly is very very interesting.

You make reads more plentiful, and guess what things change.

So focusing on the speed of Flash or how it's packaged is, in my mind, about as boring a discussion as you can imagine.

But hey, leave it to a storage customer to point out when storage vendors are too busy examining their plumage instead of focusing on the real value to the customer.

Adventures in consumer flash

I recently purchased a netbook. And to protect the innocent, I'll keep vendor names to myself.

As a technologist, the idea of an 8.9" laptop was intriguing. When I first bought my first smartphone 3 years ago, what I really wanted was a tiny laptop with an awesome battery life. I looked into UPMC laptops, but the problem was the cost (2000+ dollars) and the battery life (2 hours). If I bought one it had to replace my real computer because of the cost, and there was no way I was going to have as my only computer something that small.

Asus, recently, invented the netbook category. Essentially what they did was to look at the OLTPC (one laptop per child) guys and figure out how to make that product at a higher price point but with all of the functionality you really want.  And in doing that, Asus made a lot of money and it may mean that I won't be buying a new smartphone.

On a personal note, this is a lot like in the 80's when the Amstrad PC was released. Up until the Amstrad, the average Greek could not afford a PC. So computers based on the Z80 etc were very popular. When the Amstrad came out at a slighly higher price point than the Z80's but still ran all of the PC software, the Z80's etc  disappeared over night. 

So I got my netbook and discovered that the flash drive they had installed was slow. Not slow in a kinda-sorta-slow, but slow in a why-did-i-ever-buy-this-product slow. I wasnt' sure if I was ready to give up on the category entirely, so I bought a faster (more expensive) flash drive (really a mini pci-e module) to see if it was really the flash or something else.

Well it turns out that my new netbook is fast, and fun and furious with it's new storage.

If the performance delta between drives is so fantastic, (I've never seen such performance delta's between equivalent hard drives) this tells me that  software is going to determine the Flash experience.

More interestingly, it validated my faith that just putting in a flash drive into your system is about leveraging someone's else's IP. The real magic will happen when flash get's integrated into the virtualization layer directly. 

Real FiberChannel vs Better than Real FiberChannel and Flash

Although I promised to not respond to the question of how real or not real NetApp's FiberChannel is or is not, the recent flurry about Flash brings to stark relief a core fundamental difference between NetApp and EMC on how we believe storage systems should be architected. They, EMC believe, that the right answer is Real FiberChannel , I believe, it's Better than Real FiberChannel.

So let me make my case.

On Monday I discussed how CPU schedulers exploit the fact that not every program needs to have it's own dedicated CPU to improve utilization of the overall system. This is a well known approach to computer system design that dates back to IBM in the 1970's. What the scheduler does is create, implicitly and not explicitly, a virtual CPU for every application and then schedules those virtual CPUs based on some notion of priority.

Building a CPU scheduler that is both fair and efficient is hard. I recently looked at the Linux Kernel and noticed that they have had three different scheduler implementations over the last several kernel releases.

There is an easier way to schedule processes. That approach is called batch scheduling with static partitioning. Effectively each process is pre-allocated a set of CPU resources, and the OS provides those resources. If the process doesn't use the CPU resources those resource go to waste. At SGI I wrote a system called miser that attempted to combine the best of batch scheduling with dynamic scheduling.

It turns out that scheduling disk IOPS and disk capacity is analogous to scheduling CPU time. And it turns out that NetApp and EMC's approach to scheduling disk IOPS and capacity boils down to whether you believe in static partitioning or dynamic scheduling.

More prosaically, we believe in putting scheduling of space and IOPS in the array, they believe in static partitioning outside of the array.

Huh?

The Real FiberChannel advocates will observe that FiberChannel disk drives do not exploit spatial and temporary locality to deliver better performance and utilization. They will also observe that software layers on top of the disk drives make assumptions about how disk blocks get mapped. They will also observe that higher layers (file systems and volume managers) will exploit temporal and spatial locality for better performance and utilization. And the whole point of Real FiberChannel is that doing it in the storage array is the wrong thing to do. So instead what you should do in the storage array is provide static partitioning. They add value to storage by making those statically partitioned disks more available and better performing. Their fundamental argument is that the storage array should not get in the business of out-thinking the higher layers in the stack.

 

The Better than Real FiberChannel advocates will observe that static partitioning is wasteful. Instead we should be providing virtual FiberChannel disks whose physical resources get mapped on demand rather than before they get used. And because we control the allocation on demand, we can get better performance by exploiting memory and IOPs better. The exploitation does come at a cost and getting it right is hard but you can minimize and eliminate those costs. After all it's been 40+ years since the first CPU scheduler was released and we're still innovating in that space. But if you can deliver the performance and the availability you can deliver better utilization and throughput. I call this approach Better than Real FiberChannel, because this group is trying to do something better than what the fundamental disk drives can do.

At the core, if you rely on static partitioning you have to do a lot of pre-planning and be willing to tolerate waste and if you rely on dynamic partitioning you get better utilization and better throughput with, if you do it right, just as good performance. 

And you know what, this last point is true independent of NetApp or EMC or Disk drives, it's just a fundamental axiom of scheduling.

And going back to flash

If you look at EMC's approach to flash and our approach to flash, you see our fundamental philosophical approaches on display. They believe that the right thing to do is make the flash disk better by throwing hardware at it. So they put an EFD into an array, add some hardware on top of that and proclaim their technological brilliance. We believe in making the logical disk drive we present to the application better by leveraging flash in our system through things like caches and proclaim our technological brilliance.

And at the end of the day, this is as much a religious war as it is a technical war. 15 years ago I believed in dynamic resource allocation and I still believe in it, because I think we can do better than just carving up hardware resources.

But in the end the market not EMC nor NetApp will decide who is right.

Deeper into the scheduler

When I wrote about the failure of hybrid multi-processor CPU schedulers, I made the surprising leap that I saw the same failure occurring in the storage space. Namely that multiple conflicting virtualization layers in the storage stack are creating significant inneficiences.

But before I get to the storage discussion, I think it's worthwhile to explain my point about the hybrid scheduling a little bit more. But before I do that, let me try and explain what a CPU scheduler is and does.

So what is a CPU scheduler?

Wikipedia has it's definition, which I'll copy because I think it's a good one:

The scheduler is concerned mainly with:

• CPU utilization - to keep the CPU as busy as possible.
• Throughput - number of process that complete their execution per time unit.
• Turnaround - amount of time to execute a particular process.
• Waiting time - amount of time a process has been waiting in the ready queue.
• Response time - amount of time it takes from when a request was submitted until the first response is produced.

More specifically the scheduler is the piece of software in an operating system that performs those tasks.

But in English?

As a user of a computer, you have a bunch of things you want to accomplish: you want to surf the web, you want to write comments in blogs, and you want to send email to your buddies. To perform each of those tasks you use a variety of programs. For example, you may use FireFox to surf the web and Outlook to send email to your buddies.

To perform any task, the program you want to use has to run. To run the program has to use CPU resources and memory resources.

In any modern operating system (OS), the CPU and Memory resources are not available to your programs without going through the OS. The OS decides who and how much of the physical resources your programs can use.

Why?

After all you want to use Firefox, and you're god of your computer, why not just give all of the CPU and all of the memory to Firefox?

Because it turns out that there are lots of other things that you want to run at the same time, for example, you want to get notifications from Outlook about your meetings, and you wnat notifications from your IM that you're buddy is online, and you want to respond to that email while you're downloading that program from the internet.

So ideally you would like to have all of those things going on at the same time.

But there is a problem. You want ten different things to go on at the same time, but you only have one physical processor! Remember, for anything to happen the programs have to use some CPU time. And if there is only CPU and only one program can be using the CPU at a time, you can't do more than one thing at a time!

Enter human fraility.

Human beings are very slow. Consider a web page. After the page is downloaded we take several minutes to read the page's contents. And during that time, the computer is effectively just sitting around doing nothing.

Well, wait-a-sec, that's when I can run all of those other programs (like Outlook and IM). During the time, when I am just wasting the computer's time trying to respond to the information it just produced, the computer can use the CPU to do other things, like see if I have a meeting invite or check to see if my buddy is on IM!

And that's what the OS scheduler does, it checks to see if there is any other program that wants to run and gives the CPU to that other program (which is what we mean by the term CPU Utilization).

Well okay, so far so good, but I am the computer's god, and I want the computer to always give CPU time to the program I want to run, not to some other johnny come lately.

And that's the other thing the scheduler does. Not every program is created equal. The program the user wants to use are the highest priority programs. Other things are background tasks that have to give way for the user's program. So, in our example, when Firefox needs to run, the CPU scheduler gives the CPU to Firefox, because it a higher priority program. And this is what we mean by Response Time.

But there is one more thing the CPU scheduler does, suppose I have two programs of equal priority. Let's say the mp3 ripping software and my web browser. And let's say that I want both of them to use as much CPU as possible. In this scenario the CPU is never idle, it's running at 100% utilization.  In that case, what the CPU scheduler does is make sure that both programs, my mp3 software and my browser, get a fair share of the CPU so that their respective waiting time is in proportion to their priority.

In the spirit of the day, CPU schedulers virtualize the real CPUs! By virtualizing the CPU, the throughput of the system as a whole improves as idle CPU's are made busy.

Memories of a different life

In 1996 I graduated from Brown University to go work at SGI on what was a rather interesting technical problem: how to make hybrid scheduling work.

Just to recap, in the early 90’s the UNIX vendors were adding threading to their kernels and there was intense debate over what the right model was:

In the 1 on 1 model, each user thread had a kernel  thread. In the N on 1, all of the user threads were multiplexed on one kernel thread. Finally in the M on N model, many user level threads were multiplexed on a smaller number of kernel threads.

The N on 1 model was the “simplest” to implement, but did not allow the application to exploit multiprocessors. One property of the N on1 model was that synchronization between different threads did not require a context switch.

Although 1 on1 was simple enough to implement it had, what at the time seemed serious, issues. The two most discussed were the memory resources necessary to maintain the kernel data-structures, and the second was the performance implications of making every synchronization operation a kernel operation.

So the M on N model was promoted as a compromise between the N on 1 model and the resource intensive 1 on 1 model. To implement the M on N model, two schedulers were required : a kernel and user level thread scheduler. The central notion was that the kernel scheduler would schedule processors and the user level thread scheduler would assign threads to the available processors represented by the kernel threads. The nice property of this model is that if synchronization was required between user-level threads, the performance would be as good as the N on 1 model.

The challenge of this model was that the thing scheduling the physical resources, the kernel scheduler, had incomplete information and was prone to making mistakes. And the cost of mistakes was idle processors and therefore worse performance.

Ultimately a variety of attempts were made to fix the problem including scheduler activations and scheduler aware synchronization and nano-threads but none good enough to stop the ultimate triumph of the 1 on 1 model.

It was the need to co-ordinate between the two schedulers that eliminated the advantages of the M on N model.

One could almost argue that this was a computer science boondoggle.

So what does this have to do with storage?

At the end of the day the lesson I learned from the whole experience is that having multiple layered resource schedulers scheduling the same resources is difficult to make work.

If you look at the storage stack, what you’ll see is multiple distinct storage layers each managing virtualized pools of storage that they have no visibility into. And what’s worse the thing that actually does the physical resource management has no visibility into what the resources are being used for.

So to me it looks that the storage stack has evolved over the last 20 or so years to look a lot like the hybrid thread scheduling model.

And that is enough to give me reason to pause and think.