VMware over NetApp NFS: A Customer's Testimonial

A while back, in a previous blog entry, I've explained the benefits of implementing VMware over NFS on NetApp. That particular entry received almost 9500 hits in a single day. That, by itself, shows the high interest folks have in this capability, and understandably so, if one were to weigh the pros and the cons of such implementation.

In fact, we have not been alone in suggesting that NFS in VMware environments make a lot of sense, you can find other entries like this one here.

The fact of life is that Virtual Machines are files, and NAS protocols are designed to manage files. So why complicate things?

We can sit down and debate all day about the performance of FC vs NFS vs iSCSI but the reality is that in typical VMware implementations you'll be hard pressed to tell the difference.

Where FC typically gets deployed is in environments that already have available infrastructure components (i.e FC switch ports/HBAs).

Smaller businesses, tend to gravitate towards iSCSI implementations, typically because they do not want to deal with the complexity FC brings, as well as, because of lack of IT expertise.

But what if you're a "big house" and have considerable FC and NAS investments? What we're seeing is that these folks tend to prefer NFS as the protocol of choice.

In fact we have customers that have migrated their VMware enviroments from FC running on 3rd party disk arrays, to NetApp NFS implementations.  One such customer has posted his thoughts of such implementation on NetApp and the compelling value proposition. 

Comments

[Terry]

what is your recommendation for Site Recovery Manager? I hear that SRM will only support VMFS datastores. Also, what about Application Consistency?

The VMFS filesystem uses a larger block size 1MB to 8MB depending on size of the file system, so wouldn't it perform better on VMFS then on NFS?

[Nick Triantos]

That's my understanding as well, initially.

What you need to focus is the application I/O sizes rather than filesystem block size. Most windows apps do a 4k-8k block size typically in a random pattern.

What you also need to consider is that VMFS is a clustered file system with *multiple readers and a single writer*. That means that only 1 vmkernel can write to a vm at a time....Now scale this out and image where the bottleneck will be... NFS doesn't have this issue.

If Server virtualization enables a grid type of architecture, the FC or iSCSI are the wrong protocols because they don't scale well. Most grid environments don't run these protocols with clustered filesystem. They run NFS.

Application consistency - This is an challenge pretty much across this segment. We made an announcement yesterday for our SnapDrive and SnapManager products which allow for app consistency (SQL,Exchange, Sharepoint). The initial support is for RDM only but we're constantly looking for ways to improve this capability that right now is almost non-existent in this market segment.

[Bert dB]

No, VMFS is not "multiple readers - single writer". VMFS is multi-reader and multi-writer-to-files, but single-writer-to-filesystem-metadata.

The main benefit for FC/iSCSI over NFS seems to be the possibility for offloading the overhead to a dedicated card, which you can't do for NFS.
The main benefit for VMFS over NFS seems to be guaranteed non-fragmented files. This simplifies the translation from virtual SCSI bus to real iSCSI/FC/SCSI commands. With NFS, it's the NFS server that handles this, but it can't guarantee non-fragmentation, and therefore has to do more filesystem block lookups to find the correct blocks in a vmdk file.

[Nick Triantos]

You are absolutely correct. In fact I phrased it *completely* wrong when I said single writer. It's not. It's single writer as far as the metadata is correct. In fact, this morning i've been meaning to correct this statement.

[Nick Triantos]

I meant to say "as far as the metadata is concerned"

For large scale implementations, I will take NFS over VMFS any day of the week and twice on sundays...

Without a doubt both of them have pros and cons, but when it comes down to the managability and operational efficiencies NFS is no match (and this is coming from a guy with a blocks background).

People are starting to warm up to the idea that using a file based protocol to manage...uuuh files (vmdk) is the way to go.

Cheers and thanks for correcting my mistake

[Mark]

Nick- VMware just published a paper comparing FC,iSCSI and NFS storage protocols. NFS does very well and in most cases matches the performance of both hardware and software iSCSI. When you add up the manageability of NFS with performance equal to iSCSI, it really is no wonder people are going with an NFS solution for their virtual infrastructure. You can download the paper at http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

[Mark]

Nick- VMware just published a paper comparing FC,iSCSI and NFS storage protocols. NFS does very well and in most cases matches the performance of both hardware and software iSCSI. When you add up the manageability of NFS with performance equal to iSCSI, it really is no wonder people are going with an NFS solution for their virtual infrastructure. You can download the paper at http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

[Nick Triantos]

Hi Mark,

I've seen the whitepaper. Although it's a good start, to be honest with you, I expected to see something a bit more involved that would depict a typical ESX implementation.

I'd expected to see several ESX servers in a clustered config, with VMFS and datastores split across FC and iSCSI given that most VMware customers do not implement RDM, never mind a single ESX server.

In mind, given that ESX enables a grid type of architecture, then any protocol comparisson should take that into consideration.

[Charlie]

Can you give a brief overview of the environment you use this in? What throughput is required per host? etc...

[Steven Craig]

A little on topic, but from a different topic. A while back I asked questions about restoring files from a snapshot. Well I tried it, using your instructions and it mounts the boot volume. Since all of our linux VMs and im sure a lot of linux VMs out there all use LVM the offset you provided and the fdisk command provides is incorrect for where the data is actually stored, at least it appears to me that way. I wanted to see if you knew how to mount a LVM volume to get the files from a flexclone from a snapshot. I tried with now luck... thanks.

[Nick Triantos]

Hi Steven,

I remember you. When I posted the process on storagefoo I wasn't using LVM on my linux box. So to be honest with you I don't know whether it'd work of not with LVM.

If you're looking to restore Files within a vmdk from a Snapshot and you are using NFS as the protocol, then you may want to take a look at UFS Explorer. It supports a whole bunch of filesystems (ntfs/ntfs5/ufs/ufs2/ext2/ext3/reiserFS/XFS/fat16/fat32). Since it runs on windows, you'll need to create a FlexClone from an existing snapshot and share it to a windows host. Then use UFS explorer and pick the source vmdk and you'll be able to see and extract files which you then can copy to your target VM using cp or scp.

There's a trial version you can download with a restore limit of 64k.

http://www.ufsexplorer.com/

http://www.filetransit.com/screenshot.php?id=32497

[Marty]

This link is broken:
http://scalethemind.com/2007/09/nfs-unjust-vmware-storage-outcast.html

Should be:
http://scalethemind.com/2007/09/nfs-unjust-vmware-storage-outcast/

[Steven Craig]

Hey Nick,
I used the software, it worked good for non-LVM VMs, but i eneded up after a few days figuring it out myself...here are the steps if your VM has logical volumes...

-----------------------
create NetApp flexclone from snapshot:
-----------------------
1. vol clone create /vol/vmnfsXXXbk -s none -b parent_volume_name snapshot_name
(parent_volume_name = vmnfsXXX in our environment)
2. vol options clone_name nosnap on
3. snap sched clone_name 0 0 0

-----------------------
Run on linux VM:
-----------------------
1. mount your NFS snapshot flex clone in linux:
mount SAN_IP_ADDRESS:/vol/vmnfsXXXbk /mnt/vmnfs
2. losetup /dev/loop0 /mnt/vmnfs/SERVER_NAME/SERVER_NAME-flat.vmdk
(replace SERVER_NAME with actual VMware server path)
Example: losetup /dev/loop0 /mnt/vmnfs/WEB1/WEB1-flat.vmdk
3. kpartx -av /dev/loop0
4. vgscan
this will return something like:
Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2
5. vgchange -ay VolGroup00
This will return something like:
2 logical volume(s) in volume group "VolGroup00" now active
6. lvs
This will return something like:
LogVol00 VolGroup00 -wi-a- 5.06G
LogVol01 VolGroup00 -wi-a- 30.00G
7. mount /dev/VolGroup00/LogVol00 /mnt/vm

Thats it...

--------------------------------------
Unmount the vmdk:
--------------------------------------
1. umount /mnt/vm
2. vgchange -an VolGroup00
3. kpartx -d /dev/loop0
4. losetup -d /dev/loop0

[Roberto Ruggiero]

Hi Nick,

Just curious if you could direct me to a best practices with Netapp and VMware ESX using NFS.

Additionally, setting up VM on a netapp using NFS, is it better to use one export or have multiple exports for each VM? What are the advantages of either way?

thanks

[Nick Triantos]

Roberto,

Best practices for all protocols are described on TR3428

http://www.netapp.com/us/library/technical-reports/tr-3428.html

we do not recommend having multiple exports for the same underline NFS volume, with a different Datastore label, using IP aliases, all of which, are use to mount the volume on the same physical host.

While this can be done in order to leverage IP load balancing and utilize multiple host NICs, from a management standpoind you need to start keeping track which VMs reside on which datastore. Although the datastores will have different labels, the underline NFS volume is the same and you will be able to see all of the VMs.

In a large environment keeping track which VM is started in which datastore can be a difficult task and is prone to errors.