I/O Efficiency and Alignment – the Cloud Demands Standards

I was surprised to read in the relatively recent EMC report: VMware ESX Server Optimization with EMC® Celerra® Performance Study that when it comes to aligning the partitions inside of virtual machines EMC has diametrically opposed recommendations based on the storage protocol and storage array which one deploys with VMware. I have to say that such recommendations raise a red flag that I believe needs some additional light shed on it.

A one-off design for a single array architecture in the cloud?

I don’t have all of the details that I’d like to have in order to explain why EMC would promote alignment for VMFS deployments on iSCSI, FC, and FCoE but not promote alignment for deployments of NFS on Celerra. Maybe Chad, Chuck, Zilla or another representative from EMC can jump in here. I believe in fairness that my comments may be harsh and I don’t want to misrepresent the Celerra technology.

The cloud requires standards

What I do know is that the cloud allows customers the flexibility to dynamically migrate VMs based on their needs between various physical servers and storage platforms and this last capability includes various storage protocols. How cloud friendly does the Celerra NFS solution sound if you have to have separate VM templates based on whether they run on VMFS or NFS?

Who wants to have to convert a VM in order to take advantage of advancements in storage technologies, application support tools, or to be able to leverage the cloud to move a VM to an off-site location or external service provider who may not be running Celerra NFS?

If you are a cloud services provider, would you support a separate physical infrastructures for VMs created on NFS connect Celerra storage arrays? I don't think so.

The market needs standards, in order to make the cloud a reality. We should all be asking EMC to share their plans to eliminate this one off design with Celerra and NFS? If I were in the position to select infrastructure architectures for my cloud deployment I would have to rule out Celerra until this issue was resolved.

A bit of background for those interested in the how and why

With virtualization I often state that everything old is new again and this mantra applies with partition alignment. Years ago, when one deployed a system on a SAN they had to properly create the starting partition boundaries on the LUN prior to installing the operating system in order to ensure optimal I/O between the operating system and the storage array. Storage arrays were enhanced and removed the need for this manual process by allowing LUNs to be provisioned based on the operating system which would be installed on the LUN. In this manner the storage array vendor set the starting partition boundaries at the time of storage provisioning and thus streamlining the OS installation process.

Fast-forward to today

Now we are in the age of deploying virtual machines and we have the same issue but instead of having it with LUNs we have it with virtual disks (VMDKs, VHDs, etc). What is almost universally agreed upon by storage vendors, server virtualization vendors, and operating system vendors is that for optimal I/O efficiency the partitions within the virtual disk need to be aligned to the underlying storage array. NetApp along with VMware, Citrix, and Microsoft have jointly published Technical Report 3747 specifically to address this issue.

As I am much more familiar with Windows than Linux I’ll speak to the specifics of alignment with Windows as the GOS. Past versions of the Windows operating system, specifically NT, 2000, 2003, and XP, reserves the first 32,256 bytes of this disk for the boot sector. Customers virtualizing systems running these operating systems end up with a VM that has a local file system that isn’t in alignment with the underlying storage array. Thus have the reprise of the old LUN issue.

The storage and virtualization industry have repeatedly communicated a recommended start partition offset for windows based virtual machines based on 4,096 bytes (4KB) value, which is greater than 32,256. These recommendations are commonly either 32,768 (32KB) or 65,536 (64KB). Note either value is acceptable as it is large enough to store the boot sector and evenly divisible by 4KB.

Here’s a brief list of industry based references for on this point...

IBM: Storage Block Alignment with VMware Virtual Infrastructure

EMC: Celerra IP Storage with VMware Virtual Infrastructure

Dell: Designing and Optimizing SAN Configurations

EMC: CLARiiON Integration with VMware ESX Server

Vizioncore: vOptimizer Pro FAQ

Now I would be remiss if I left out that it is very critical to align the start VMFS partition to the underlying block in one’s storage array, but as this is already addressed by most modern storage array vendors I will stick to the discussion at hand which is alignment of the VMs partition.

Microsoft advances the cause

In the most recent version of the Windows operating system, Vista, 2008 Server, and 7, Microsoft has removed the need to manually align the partition as it begins at 1 MB.

Score one for the guys in Redmond!

There is no need to make any change to the partitions of VMs created from a clean install of one of these operating systems. Note this is not the case for systems upgraded to one of these releases.

This also leads me to ask "Um EMC guys, does deploying Windows Vista, 2008 Server, and 7 as a VM via NFS on a Celerra array change your recommendations around alignment?" From what is contained within the performance report it appears that these deployments might suffer. Can you clarify for us?

So why does alignment matter?

Alignment will have a direct effect on how hard a storage array has to work to service the I/O requests of the virtual machines it hosts. Now you may think that the majority of the VMs you have deployed are not very demanding in terms of their I/O request, but remember the storage array has to serve the aggregated requests of all of the VMs deployed. As your virtualization footprint expands, so does the associated I/O load. This is where I/O inefficiency can raise its ugly head.

How to tackle the alignment challenge

First update the VM templates that aren’t running an operating system that avoids this situation. This relatively simple exercise will stop you from deploying future VMs that are unaligned. Next tackle your most I/O demanding VMs.

BTW – if you really need performance out of a VM you may want to consider upgrading the virtual disks on these systems to the eagerzeroedthick format for some additional performance gains.

As a side note - If you do this on a storage array virtualized by NetApp’s Data ONTAP (this includes NetApp FAS arrays, IBM N-Series arrays, or traditional legacy arrays virtualized by the NetApp vSeries) you can have the highest virtual disk performance while consuming less storage than what is available with a thin provisioned virtual disk on a traditional legacy storage array architecture. Psst - This is one of the many values of virtualizing your storage infrastructure.

Once you addressed these systems the question remains, should I optimize my existing VMs? This is a fair question which has an answer based on it depends. You can either align your VMs (this will cause a small service disruption with each) or you can offset the I/O inefficiency by adding more disks on the back end. While all storage vendors would love to sell you more storage, over the long haul it might be a wiser investment to align the VMs.

In addition, I’d like to ask anyone distributing applications as virtual appliances to ensure that the file system within the appliance is properly aligned along a 4KB boundary.

There are many tools that can help you with correcting this challenge; my personal favorites are MBRAlign and vOptimizer Pro and vConverter from Vizioncore. For clarification MBRAlign and vOptimizer Pro can address the alignment of previously deployed VMs; however, if your infrastructure still has servers to convert from physical to virtual vConverter can ensure that the migration results in a properly aligned partition.

As always VCE or Virtualization Changes Everything (except for some old issues we used to have that were solved but are now back again)

Comments

[Richarb Barlow]

At first when I read the EMC pdf I thought that perhaps it was old info. But now I see that its from April 2009 - this is relatively hot off of the press.
I hope EMC answers the question that you're posing becuase I'm sure their customers will be asking as well, especially in mixed environment shops where there may not be a single storage standard. Do I need to keep different templates for my Cellera NFS versus my NetApp NFS/FC/iSCSI or HDS FC storage? Inquiring minds want to know . . .

[BK]

MBRAlign isn't officially supported by NetApp. This is disappointing. We would love to align our 500 guests but we aren't going to walk blindly off a cliff with an unsupported tool to do it.

What are the chances this will become supported and make it into best practices?

[Nick Triantos]

MBRalign and MBRscan are now part of NetApp's ESX Host Utilities Kit v5.1 and such are officially supported.

The newly released Kit also introduces support for vSphere and continues to support previous ESX versions.

[Vaughn Stewart]

BK - Nick is spot on here. The versions in the EHU 5.1 and the upcoming VSC are officially supported.

If you need additional assistance NetApp global support center can also assist you and your team.

Thanks for the feedback.

[Chad Sakac]

Vaughn - thanks for investigating. Digging into this with the Celerra team.

I agree - there can be only one standard guest-level VM alignment, as customers need to be able to know that moving from one storage platform to another won't require re-aligning.

Errors (as you well know from the old NFS disable lock episode) can be made - we are all human after all - but it's important to catch and correct them if indeed it is an error.

Will get you and everyone else and update.

[Vaughn Stewart]

Chad - thanks for jumping in to address this issue from the EMC perspective. Customers should be able to have all features on all arrays over any protocol and on-off configs prevent this from happening. Sometimes change requires a little effort (ala like correcting the old NFS locking issue).