File Level Recovery from within a VMDK backup

One of the frequently asked question when implementing VMware on NFS is how would you implement file recovery from within a VMDK backup and what tools are available to accomplish this?

Implementing array based snapshots to backup entire datastores and their contents is a simple and a well documented process, however, often times customers have the need to recover specific files from within VMDKs.

Depending on the Datastore type used (VMFS vs NFS) the process and the administrative steps required as well as the associated costs vary greatly.

With VMFS you can could treat the Virtual Machines as Physical servers and deploy backup agents inside each VM and have file level backup and restore granularity. All that at  the expense of typically higher CPU and Memory consumption, multiple backup schedules and in general higher acquisition and operational costs.

Another approach would be to use a combination of Vmware's Consolidated Backup (VCB) and Backup software and do file level backups and restores.

How about NFS?

The backup process using NFS is very simple. Because we control the filesystem we know what's in there and we can recover specific files (.vmdk, .vmx) or any file that's in a VM directory.

However, what is the process to provide further granularity and be able to restore a specific file from within a VMDK backup in an NFS implementation without using backup software agents or proxies?

There are several options to accomplish this however, today I'll talk about a couple of them.  Both require a CIFS share to be created onto the snapshot directory of the NFS volume (i.e /vol/vmdata/.snapshot).

In both cases you will be accessing the snapshot directory from the Virtual Machine that requires the needed file by specifying the UNC path. This process works for Windows VMs and requires VMDKs with a single partition.

Option 1

This option was recently discovered and further developed (Registry entries and Batch script) by one of the NetApp SE's (Mike Arndt) and it's very effective and free for those customers that already have a CIFS license with their NetApp arrays, which is a very large percentage. The other important factor is that Mike has made it a point-and-click process. Great job Mike!!!

As part of their VMware Disk Developer's Kit, VMware provides a vmware-mount.exe utility that allows for mounting an existing VMDK on a Windows Driver letter. We'll be using this utility as well to mount the VMDK as well as some Registry Entries and a Batch Script to further simplify the file recovery process.

Step 1: Download and install the kit from the above link 

Step 2: Registry Entries

The following registry entries can be added by putting them in a file with a .reg extension and double-clicking on the file. The entries will provide you with a  "VMDK Mount" and "VMDK UnMount" option when you right click in the Windows Explorer interface.

REGEDIT4 
; The following registry keys will put right click shortcuts in place for VMDKMounter.

[HKEY_CLASSES_ROOT\.vmdk]

@="VMware.VirtualDisk"

[HKEY_CLASSES_ROOT\.vmdk\VMware.VirtualDisk\ShellNew]

@="" 

[HKEY_CLASSES_ROOT\VMware.VirtualDisk\Shell\vmdkmounter]

@="VMDK Mount" 

[HKEY_CLASSES_ROOT\VMware.VirtualDisk\Shell\vmdkmounter\command]

@="cmd.exe /c "c:\\Program Files\\NetApp\\VMDKMounter\\vmdkmounter.bat" mount %1"

[HKEY_CLASSES_ROOT\Drive\shell\vmdkmounter]

@="VMDK UnMount"

[HKEY_CLASSES_ROOT\Drive\shell\vmdkmounter\command]

@="cmd.exe /c "c:\\Program Files\\NetApp\\VMDKMounter\\vmdkmounter.bat" unmount %1"

Step 3: Batch Script

The following batch script needs to be placed in C:\Program Files\NetApp\VMDKMounter in order for the previously mentioned registry entries to work properly.

Note: The Lines marked in red should be one line.

@ECHO OFF
SET vmware_mount_cmd_path="C:\\Program Files\\VMware\\

VMware Virtual Disk Development Kit\\bin\\vmware-mount.exe" 
IF EXIST %vmware_mount_cmd_path% GOTO CONTINUE
ECHO The vmware-mount utility is not installed.  Exiting.
GOTO EXIT0

:CONTINUE
IF "%1"=="mount" GOTO GETDRIVE
set drive_letter=%2
ECHO Unmounting VMDK on drive %2 ...
%vmware_mount_cmd_path% %drive_letter:~0,2% /d /f
GOTO EXIT0

:GETDRIVE
SET /P drive_letter="Enter Drive Letter [Z:] " 
IF NOT "%drive_letter%"=="" GOTO CHECKDRIVE
SET drive_letter=Z:
:CHECKDRIVE

IF NOT EXIST %drive_letter% GOTO DOMOUNT
ECHO Drive %drive_letter% appears to already be in use!  Exiting.
GOTO EXIT0

:DOMOUNT
ECHO Mounting VMDK on drive %2 ...
%vmware_mount_cmd_path% %drive_letter% %2 /m:n
explorer %drive_letter%

:EXIT0
@ping 127.0.0.1 -n 5 -w 1000 > nul

File Recovery Process 

Step 1: Use the UNC path to access the snapshot directory (Figure 1)

Step 2: Pick the snapshot you want and drill down and into the VM folder

Step 3: Select the *.vmdk (not the -flat.vmdk), right click the "VMDK Mount" 

Step 3: Drag and Drop the file and use"VMDK UnMount" option when done.

        Figure 1                                                             Figure 2

Option 2

This option is, to a certain extent, similar with option 1. The main difference here is that you don't need VMware's Virtual Disk Developer's Kit.

Instead, what you need is to license a tool that is able to crack open a VMDK. Such tool is UFS Explorer Standard Access or WinImage. A UFS Explorer Standard Access Business license runs around $65 and you can install it on as many Windows systems as you want , as long as you own the systems. I like UFS Explorer primarily because it supports several filesystem types (supports VHD images) versus WinImage which only supports NTFS.

The way the recovery process works is as follows:

Step 1: Install UFS Explorer into the VM(s) or into a VM used for recovery purposes

Step 2: Map a Network Drive to the NetApp snapshot folder using the UNC path or even all the way into the VM folder that resides under the snapshot you want

Step 3: Point UFS Explorer into the -flat.vmdk file (not the (.vmdk file).

Step 4: Explore the VMDK and drag and drop the file you need

 

Done. Simple, Easy and cost effective!!!!

 

 

 

Comments

[Brandon]

Nice post.

You can also do file level recoveries with a Linux host that has access to your filer's snapshots. Loopback mount the VMDK file from the snapshot on the Linux host and get the file(s) you need from the VMDK.

If you have Windows guests, you may need to build the NTFS kernel module depending on which Linux distribution you're using.

[Nick Triantos]

Thanks Brandon,

You are correct you can do that too. In fact, I've written about it a while back on an old blog I kept:

http://storagefoo.blogspot.com/2007/09/vmware-on-nfs-backup-tricks.html

http://storagefoo.blogspot.com/2007/10/vmware-over-nfs-backup-trickscontinued.html

[Markus]

Very nice post.

This is a very good (real world) example how NFS can help you to simplify your ESX environment.

[Seb]

Same can be done with VMware DiskMount GUI from http://petruska.stardock.net/Software/VMware.html

[Steven Craig]

Hey Nick,
I used the software, it worked good for non-LVM VMs, but it doesnt work with a VM that has linux LVM, which we do to split up the logs from the OS. I ended up after a few days figuring it out myself where you dont need that software at all...here are the steps if your VM has logical volumes.

-----------------------
create NetApp flexclone from snapshot:
-----------------------
1. vol clone create /vol/vmnfsXXXbk -s none -b parent_volume_name snapshot_name
(parent_volume_name = vmnfsXXX in our environment)
2. vol options clone_name nosnap on
3. snap sched clone_name 0 0 0

-----------------------
Run on linux VM:
-----------------------
1. mount your NFS snapshot flex clone in linux:
mount SAN_IP_ADDRESS:/vol/vmnfsXXXbk /mnt/vmnfs
2. losetup /dev/loop0 /mnt/vmnfs/SERVER_NAME/SERVER_NAME-flat.vmdk
(replace SERVER_NAME with actual VMware server path)
Example: losetup /dev/loop0 /mnt/vmnfs/WEB1/WEB1-flat.vmdk
3. kpartx -av /dev/loop0
4. vgscan
this will return something like:
Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2
5. vgchange -ay VolGroup00
This will return something like:
2 logical volume(s) in volume group "VolGroup00" now active
6. lvs
This will return something like:
LogVol00 VolGroup00 -wi-a- 5.06G
LogVol01 VolGroup00 -wi-a- 30.00G
7. mount /dev/VolGroup00/LogVol00 /mnt/vm

Thats it...

--------------------------------------
Unmount the vmdk:
--------------------------------------
1. umount /mnt/vm
2. vgchange -an VolGroup00
3. kpartx -d /dev/loop0
4. losetup -d /dev/loop0

[Nick Triantos]

Great stuff Steven. Thanks for figuring this out. Very helpful.

[Pat]

I have followed the instructions to a tee and I get an error when I try to mount the .vmdk file. It tells me that "This file does not have a program associated with it for performing this action. Create an association in the Folder Options control panel."

What did I miss?

[Thomas Joyce]

Pat,

I had the same issue. It seems the .reg file didn't enter the path into the registry eky.

Try going in and manually entering the paths to the executable into the registry in each location given in the script.

So, in the HKCR\Drive\Vmware.VirtualDisk\Shell\vmdkmounter\command

Double-click the (Default) key and paste the path "cmd.exe /c "c:\\Program Files\\NetApp\\VMDKMounter\\vmdkmounter.bat" mount %1" without the outside double-quotes.

Repeat with the other key.

[isi]

There seems to be a UFS Reader for free on
sourceforge. Haven't tested it yet.

http://mount-ntfs.sourceforge.net/

[isi]

sorry, does'nt work

[Steven]

Also a note about my mounting linux commands above. If your like me, I get afraid of deleting volumes in netapp, which would need to be done to get rid of the flexclone you made for mounting the linux VMDK...imagining myself deleting the wrong thing in one of my 14+ hour days. Well I have found out in the newest version of losetup there is a -r command that allows a read-only mount, which means you wouldn't need to create the netapp flexclone for the linux machine your want to restore from. So based on my example above, you would skip the netapp flexclone section and run the step 2 command like this:

losetup -r /dev/loop0 /mnt/vmnfs/SERVER_NAME/SERVER_NAME-flat.vmdk

Just an FYI for protection from myself and makes scripting restores a lot easier and safer.

[Keith A]

Thomas or Nick,
I had the same problem where the .reg script did not add the commands into the registry. I added the Mount command to the default key so mounting works great. Not sure where to add the unmount command though. Do I need to create a new path?

[Keith A]

Sorry, just reread the post. You place the second command in "HKEY_CLASSES_ROOT\Drive\shell\vmdkmounter\command" Works like a charm now. Thanks!

[Adrian Buchmann]

Is it right that I have to use mixed-security to restore with your method?

[Nick Triantos]

Hi Adrian,

No need to change the security style. I use the default which is "unix". Remember what we are doing though. We're not recovering the vmdk itself but rather files *within* the vmdk. The security style has no effect in this instance.

[Joe Soap]

I can only access vmware-x.log files in the snapshot directories. Access to .vmdk, .vmx, ... in the same directory is denied. Share Permission ist set to everyone:full Any ideas?

[Adrian Buchmann]

Thank you for the informatiom, Nick.
Got the same problem as Joe, i'm able to access the vmx and the logs, but I don't have access do the vmdk file.
So I have to ask the same question: Any ideas?

[Nick Triantos]

Guys,

Look at the following volume options:

a) convert_ucode
b) create_ucode

both of them should be set to on. In fact, that's has been the default behavior. If they are not, try the steps below in the following order:

1) remove the cifs share
2) vol options convert_ucode on
3) vol options create_ucode on
4) create cifs share
5) take a snapshot
6) Try to access the contents

Let me know if this changes things

[Nick Triantos]

It looks like I can't use the greater than or less than signs to specific the volume name so steps (1) and (2) omit the volume name. I'll use parenthesis instead:

1) vol options (volname) convert_ucode on
2) vol options (volname) create_ucode on

[Keith A]

One other thing you will want to watch out for is make sure you don't have any spaces in the snapshot name. Windows doesn't like spaces and will keep you from opening the VMDK file. Simply pop into Filerview and rename the snapshot and you are back in action.

[Gordon Garnt]

When i try and browse a .snapshot directory i get an access denied error message. Can anybody advise?