VMware over NetApp NFS: A Customer's Testimonial

A while back, in a previous blog entry, I've explained the benefits of implementing VMware over NFS on NetApp. That particular entry received almost 9500 hits in a single day. That, by itself, shows the high interest folks have in this capability, and understandably so, if one were to weigh the pros and the cons of such implementation.

In fact, we have not been alone in suggesting that NFS in VMware environments make a lot of sense, you can find other entries like this one here.

The fact of life is that Virtual Machines are files, and NAS protocols are designed to manage files. So why complicate things?

We can sit down and debate all day about the performance of FC vs NFS vs iSCSI but the reality is that in typical VMware implementations you'll be hard pressed to tell the difference.

Where FC typically gets deployed is in environments that already have available infrastructure components (i.e FC switch ports/HBAs).

Smaller businesses, tend to gravitate towards iSCSI implementations, typically because they do not want to deal with the complexity FC brings, as well as, because of lack of IT expertise.

But what if you're a "big house" and have considerable FC and NAS investments? What we're seeing is that these folks tend to prefer NFS as the protocol of choice.

In fact we have customers that have migrated their VMware enviroments from FC running on 3rd party disk arrays, to NetApp NFS implementations.  One such customer has posted his thoughts of such implementation on NetApp and the compelling value proposition. 

Comments

[Terry]

what is your recommendation for Site Recovery Manager? I hear that SRM will only support VMFS datastores. Also, what about Application Consistency?

The VMFS filesystem uses a larger block size 1MB to 8MB depending on size of the file system, so wouldn't it perform better on VMFS then on NFS?

[Nick Triantos]

That's my understanding as well, initially.

What you need to focus is the application I/O sizes rather than filesystem block size. Most windows apps do a 4k-8k block size typically in a random pattern.

What you also need to consider is that VMFS is a clustered file system with *multiple readers and a single writer*. That means that only 1 vmkernel can write to a vm at a time....Now scale this out and image where the bottleneck will be... NFS doesn't have this issue.

If Server virtualization enables a grid type of architecture, the FC or iSCSI are the wrong protocols because they don't scale well. Most grid environments don't run these protocols with clustered filesystem. They run NFS.

Application consistency - This is an challenge pretty much across this segment. We made an announcement yesterday for our SnapDrive and SnapManager products which allow for app consistency (SQL,Exchange, Sharepoint). The initial support is for RDM only but we're constantly looking for ways to improve this capability that right now is almost non-existent in this market segment.

[Bert dB]

No, VMFS is not "multiple readers - single writer". VMFS is multi-reader and multi-writer-to-files, but single-writer-to-filesystem-metadata.

The main benefit for FC/iSCSI over NFS seems to be the possibility for offloading the overhead to a dedicated card, which you can't do for NFS.
The main benefit for VMFS over NFS seems to be guaranteed non-fragmented files. This simplifies the translation from virtual SCSI bus to real iSCSI/FC/SCSI commands. With NFS, it's the NFS server that handles this, but it can't guarantee non-fragmentation, and therefore has to do more filesystem block lookups to find the correct blocks in a vmdk file.

[Nick Triantos]

You are absolutely correct. In fact I phrased it *completely* wrong when I said single writer. It's not. It's single writer as far as the metadata is correct. In fact, this morning i've been meaning to correct this statement.

[Nick Triantos]

I meant to say "as far as the metadata is concerned"

For large scale implementations, I will take NFS over VMFS any day of the week and twice on sundays...

Without a doubt both of them have pros and cons, but when it comes down to the managability and operational efficiencies NFS is no match (and this is coming from a guy with a blocks background).

People are starting to warm up to the idea that using a file based protocol to manage...uuuh files (vmdk) is the way to go.

Cheers and thanks for correcting my mistake

[Mark]

Nick- VMware just published a paper comparing FC,iSCSI and NFS storage protocols. NFS does very well and in most cases matches the performance of both hardware and software iSCSI. When you add up the manageability of NFS with performance equal to iSCSI, it really is no wonder people are going with an NFS solution for their virtual infrastructure. You can download the paper at http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

[Mark]

Nick- VMware just published a paper comparing FC,iSCSI and NFS storage protocols. NFS does very well and in most cases matches the performance of both hardware and software iSCSI. When you add up the manageability of NFS with performance equal to iSCSI, it really is no wonder people are going with an NFS solution for their virtual infrastructure. You can download the paper at http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

[Nick Triantos]

Hi Mark,

I've seen the whitepaper. Although it's a good start, to be honest with you, I expected to see something a bit more involved that would depict a typical ESX implementation.

I'd expected to see several ESX servers in a clustered config, with VMFS and datastores split across FC and iSCSI given that most VMware customers do not implement RDM, never mind a single ESX server.

In mind, given that ESX enables a grid type of architecture, then any protocol comparisson should take that into consideration.

[Charlie]

Can you give a brief overview of the environment you use this in? What throughput is required per host? etc...

[Steven Craig]

A little on topic, but from a different topic. A while back I asked questions about restoring files from a snapshot. Well I tried it, using your instructions and it mounts the boot volume. Since all of our linux VMs and im sure a lot of linux VMs out there all use LVM the offset you provided and the fdisk command provides is incorrect for where the data is actually stored, at least it appears to me that way. I wanted to see if you knew how to mount a LVM volume to get the files from a flexclone from a snapshot. I tried with now luck... thanks.