Why Run VMware Over NAS?

At VMWorld yesterday, I was surprised how excited customers are about using NFS to access VMDKs, even for virtual machines hosting Windows. (A VMDK is a VMware Virtual Disk, and it holds the boot image for its virtual machine.)

Since a VMDK is a virtual disk, I had assumed that block-based protocols like iSCSI and Fibre Channel would make more sense than NAS, so I asked several customers why they prefer NFS.

The answer is simple: Managing .vmdk files is much easier than managing LUNs. If you have 20 or 30 virtual machines, then VMFS is great for consolidating the VMDKs into a single LUN. But NAS is much easier and more scalable if you have hundreds or thousands of virtual machines.

The big advantage is that you can use all your file management tools. Group the VMDKs for Exchange servers in one folder, SQL servers in a second, virtual desktops in a third, and so on. Instead of backing up LUNs or virtual machines individually, simply backup a directory tree of VMDKs all at once. (This is much less expensive than buying a backup license for each virtual machine, and also easier to manage.) For disaster recovery, you can replicate the data for a whole group of virtual machines as a single unit.

Some people are surprised that you can use NFS for Windows virtual machines, since Windows can’t boot from NFS. This works because VMware has built NFS into ESX’s disk virtualization layer. ESX handles the NFS protocol so that the operating system doesn’t have to. This is very similar to Oracle’s recent project to build NFS support directly into their database.

I rarely find excuses to praise Chuck Hollis from EMC, but in a recent blog he said:

I think that, in the long term, we'll find high-end NAS much more friendly for high-end VMotion / DRS farms than today's SANs. And I think that NAS has the potential to offer a few benefits that we might not find in the SAN world.

Brilliant!

Comments

[Reinoud Reynders]

Dear Dave,

When you use FCP or iSCSI on VMWare (with VMFS), your NetApp is changed from a great storage device to a just ordinary storage solutions. You loose all the fun part: snapshots, thin provisioning, snapvault (is possible in theory but not in real live), ...

With NFS, you don't use the file system of VMWare. It's back WAFL who rules the world. This is also a big advantage of using NFS.

And to go one step futher, what do you think about the marriage between VMWare and OnTap GX over NFS? I believe this will be one with a lot of passion and firework.

Reinoud

[Matt]

I concur with Reinound, my current NetApp FAS 3050c are just plan SAN's hooked up to my ESX 3 systems. I'm seriously considering the move to NFS to better use the snapshots built into my filers.

-- Matt
University Technology

[Mike Riley]

To add to Dave and Reinoud's comments (and even Chuck's) Nick Triantos, a NetApp Global SE, has posted on his blog some VMware backup ideas if you're considering VMware on NetApp using NFS. VMware has a link to it as well from their VMTN blog but you can get directly to it here:

http://storagefoo.blogspot.com/2007/09/vmware-on-nfs-backup-tricks.html

Great read. Thanks, Nick.

[Dave Salisbury]

We implemented NFS over ISCSI when we saw the results of ASIS on the vmdk files. We are currently deploying XP workstations through VMware for all of our contractors. All of the OS VMDKs are the same, with applications being installed on a seperate VMDK.

We are finding > 40% savings using ASIS on these volumes.

[Dan Pancamo]

We have been running vmware over NFS for a year now using the fas3050 and fas3070. I can't imagine moving back to FC or ISCSI.

ASIS is great, but please rush the Aggregate level ASIS!

Also, I hope you guys are going watching the Site Recovery Manager (SRM) feature closely. It would be nice to see Netapp as one of the 1st vendors to support this feature.

[Dan Pancamo]

Why Run VMware Over NAS?

Nick Triantos (Netapp Employee) answers the question with several reasons.

http://storagefoo.blogspot.com/2007/09/vmware-over-nfs.html

[Markus Pettenkofer]

Dear Dave,

some of my Customers has bought NetApp and use NetApp together with VMWare´s ESX. With the new VMWare Release 3.0.2 it should now be possible for NetApp to offer Snap Drive for ESX Servers, so that the guest servers have the Possibilty to use Snap Manager Modules.

When can i offer snap drive for ESX to my customers ?

Yours sincerely
markus

[Chad Sakac]

Another interesting thing in NFS's "advantage" - at least in the near term - column is there is more robust multipathing/failover using network link-layer protocols than with iSCSI's path-based models.

This will get better in future VMware releases, but right now, multipathing iSCSI on VMware either means you use the link-layer methods (which doesn't follow the iSCSI path-based MPIO model). In the past, the lack of advanced VMware feature support kept customers away, but everything but VCB works great on NFS.

I gotta tell you, in my mind right now - it's NFS if easy is the primary design goal, and FC if high throughput, low ESX host CPU impact or VCB (although Peter Learmonth had a good session at VMworld on alternative solutions to VCB when using NFS) the primary design goal.

Lots of our customers are deploying on NFS - it's completely a legitimate VMware option.

[Ian Forbes]

I'm a little confused. Hopefully someone can shed some light. I absolutly agree that NFS and ESX are a nice fit with many advantages. What I'm having trouble with is the agruements that you cannot achieve the same things with iSCSI/FC LUNS/ESX. Dave states:

The big advantage is that you can use all your file management tools. Group the VMDKs for Exchange servers in one folder, SQL servers in a second, virtual desktops in a third, and so on. Instead of backing up LUNs or virtual machines individually, simply backup a directory tree of VMDKs all at once. (This is much less expensive than buying a backup license for each virtual machine, and also easier to manage.) For disaster recovery, you can replicate the data for a whole group of virtual machines as a single unit.

What is a VMFS Datastore? It's a LUN presented to ESX. From a Netapp perspective that LUN is contained within a Flexible Volume. When I provision that volume I set it up thin provisioed for maximum storage utilization. I can also thin provision the LUN to achieve the same benefits. So the arguement (from Reinoud) doesn't hold true about losing the ability to thin provision.

So, now I've set-up my LUN and ESX has turned it into a VMFS datastore. That datastore will hold several vm's (vmdk's). I can setup my datastores in the same manner Dave talks about - datastore1 - all Exchange servers; datastore2 - all SQL servers; datastore3 - all virtual desktops. I could also group all of those servers within the same datastore. When it comes to backup I'm not backing up LUNS or vm's individually. I'm backing up the flexible volume that contains the LUN - which holds several vm's.

An NFS datastore is still provisioned from a flexible volume. The only difference is instead of seeing one big file (LUN) you see a bunch of files. But, when you backup you'd backup the flexible volume in order to capture all the vm's contained. How is that any different than backing up the volume conating the LUN (which contains several vm's)?

Snapshots are done at a volume level. If that volume contains a LUN (which contains vm's) or a bunch of files (vmdk's) as long as I'm snapping the volume I'm achieving the same level of data protection.

Replication? Well, replicate the volume. Doesn't matter if that volume contains a LUN or a bunch of files - I'm only interested in replicating the volume.

Myabe someone can explain to me if ASIS can be achieved on block data. Is it not possible to get the beneifts of de-duplication from data stored within a LUN. If I decided to house all of my VMWare VDI desktops onto a FC or iSCSI LUN and all those dektops were identical (i.e. running Windows XP for a call center), ASIS wouldn't benefit me? Is there some benefit for de-duplication when using NFS datastores instead of VMFS datastores?

Cheers

Ian

[Richard Barlow]

Ian,

Actually, the best results that we have seen are from running ASIS against NFS rather than VMFS and here's why.

When you dedupe a VMFS3 LUN within a NetApp volume you are simply reducing the number of blocks that the LUN uses from the NetApp point of view. From the point of view of the VMware server, nothing has changed. Your 500GB LUN with 10 40GB .VMDK files is still using 400GB of space. You have no new room for VMs. The actually LUN may look smaller from the NetApp point of view which gives more room for snaps, etc. But from the POV of the VMware server nothing at all has changed.

But with NFS, this all changes dramatically. Last month we cloned 6 identical sysprepped Windows XP clients to a NFS volume. They were 20GB each, for a total storage usage of 200GB. You can see this from both the NetApp df command AND from within VMware.

Then we ran an ASIS run against the volume. After it completed, only 22GB was in use. You could see this from both NetApp and VMware. Then, we FlexCloned this volume 6 times with no file guarantees for a total of 36 VMs. We ASIS ran the resulting clone volumes again after provisioning and setup for a total storage usage of about 45GB! That's 45GB for 36 fully functional XP VMs. Without NFS and ASIS we would have used 720GB. My boss still really can't believe it and keeps muttering something about "voodoo", but I keep telling him that its simply the magic of WAFL and an intelligent filesystem.

Absolutely unbelievable and it is totally changing the way that we use VMware and NetApp.

All hail the 6070c, king of NFS!

Rich

[Richard Barlow]

Ian,

Actually, the best results that we have seen are from running ASIS against NFS rather than VMFS and here's why.

When you dedupe a VMFS3 LUN within a NetApp volume you are simply reducing the number of blocks that the LUN uses from the NetApp point of view. From the point of view of the VMware server, nothing has changed. Your 500GB LUN with 10 40GB .VMDK files is still using 400GB of space. You have no new room for VMs. The actually LUN may look smaller from the NetApp point of view which gives more room for snaps, etc. But from the POV of the VMware server nothing at all has changed.

But with NFS, this all changes dramatically. Last month we cloned 6 identical sysprepped Windows XP clients to a NFS volume. They were 20GB each, for a total storage usage of 200GB. You can see this from both the NetApp df command AND from within VMware.

Then we ran an ASIS run against the volume. After it completed, only 22GB was in use. You could see this from both NetApp and VMware. Then, we FlexCloned this volume 6 times with no file guarantees for a total of 36 VMs. We ASIS ran the resulting clone volumes again after provisioning and setup for a total storage usage of about 45GB! That's 45GB for 36 fully functional XP VMs. Without NFS and ASIS we would have used 720GB. My boss still really can't believe it and keeps muttering something about "voodoo", but I keep telling him that its simply the magic of WAFL and an intelligent filesystem.

Absolutely unbelievable and it is totally changing the way that we use VMware and NetApp.

All hail the 6070c, king of NFS!

Rich

[Rich Barlow]

Sorry for the double tap :^)

[ML]

All,

I found this : EMC confirm the use of NFS... http://france.emc.com/techlib/pdf/H2756_using_emc_celerra_ip_stor_vmware_infra_wp_ldv.pdf

page 21 : VMFS is an excellent choice along with NFS where various virtual machine Luns don't require specific IO performance

page 33 : VMware has executed a test suite that achieved similar performance between the two protocols.

Best regards
ML

[ML]

I just see the link is cut :
The URL (without CR) is :
http://france.emc.com/techlib/pdf/H2756_using_
emc_celerra_ip_stor_vmware_infra_wp_ldv.pdf

[Karl Pottie]

We set up a VMWare NFS datastore and did some testing. We enjoy the ease of management (e.g. being able to snaprestore a single vmdk file).

Much to our surprise, we're also seeing performance gains of 30% to 45% compared to ISCSI (all else being the same).

Eliminating LUNS & VMFS seems to have nothing but advantages.

[kc giverupper]

You haven't helped at allThank-you so much!!just joking!!lol!!thanks you have helped me so much!!!If i could say thank you a million tims it would not be enough thanks!!!!!!!!!!!!!!!

[Dan Pancamo]

I've started keeping track of all the reasons to use Netapp NFS for VMware here:

http://viroptics.blogspot.com/2007/11/why-vmware-over-netapp-nfs.html

If I missed any reasons, let me know

dp

[Adam Stern]

Do I still need to run a script similar to the one in this link to get a crash-consistent snapshot of a vmdk file when using NFS?

>>>Snapshots are done at a volume level. If
>>>that volume contains a LUN (which
>>>contains vm's) or a bunch of files
>>>(vmdk's) as long as I'm snapping the
>>>volume I'm achieving the same level
>>>of data protection.

I'm late to the conversation but...

Yes, you can snapshot volumes containing LUNs just as easy as volumes containing NFS but snapshot backups are only the first half of the data protection story. Restores from snapshots are the second half of the story.

If the VMware datastore uses a Netapp LUN and we snapshot the volume containing that LUN and want to restore, we restore the whole LUN which includes the whole VMFS and all VM's on it.

If the VMware datastore uses a volume containing an NFS and we snapshot the volume containing the NFS and want to restore, we can restore individual files which make up virtual machines including .VMX configuration files and .VMDK disk files.

Granular restore capability is one of the beauties of NFS based VMFS backed by netapp snapshots.

And yes, FAS DeDupe (new name) will work with LUNs but you will not see the free space in the volume unless the LUN is thin provisioned.

I also have many customers looking forward to SMVI3.

-Tim-

[TimH]

Sorry... you can restore individual virtual machines or files from LUN based storage but it is a more lengthy procedure.

1) Create a LUN clone or flexclone the lun based on the snapshot from which you are going to restore.
2) LVM.enable.resignature on the ESX host
3) Mount the clone as a VMFS
4) Copy the VM

-Tim-