A Blog Less Ordinary

The blog of Dave Ingram

Restoring from LVM and VMWare disks

I recently had to restore a server that failed to boot after a power cut. This machine was a Linux VMWare host, and it had three Linux guest virtual machines that were running at the time. While we had full backups available, I decided to set myself the challenge of recovering the entire images, to save the pain of a complete rebuild.

The host server partitions were LVM-formatted volumes on top of hardware RAID-1, and each of the virtual hosts were partitioned with LVM too, internally. This means that the restore process will not be at all trivial. With just a complete image of the host system, I would need to restore (deep breath) files on a partition on an LVM logical volume (inside a volume group, on a physical volume) in a VMWare hard disk stored on LVM (logical volume inside volume group of physical volumes) inside a disk image that is itself a file on a disk. How very convoluted.

I used a Gentoo system to restore the data, although any Linux system with the appropriate packages should be able to do it. No searches turned up information on doing all of this, and I had to come up with some of it myself, so I thought I would document the process.

Just to illustrate the complexity of the data layout:

LVM Restore Diagram

Requirements

The machine that is performing the restore requires:

Procedure

Boot the dead server (vm-host) using a Linux LiveCD and copy its partitions to files on another drive. If you don’t know which partitions the LVM was stored across, you can either copy all of them, or use <tt>pvscan</tt> (possibly after running <tt>vgscan</tt>) to work out which belong to the appropriate volume group. This dump may take some time.

dd if=/dev/sda2 of=/mnt/ext-drive/sda2-data
dd if=/dev/sda3 of=/mnt/ext-drive/sda3-data
dd if=/dev/sda5 of=/mnt/ext-drive/sda5-data

Attach the drive with the partition images to the rescue machine (rescue-host), edit /etc/lvm/lvm.conf on rescue-host, and modify the filter line to allow LVM to scan loopback and network block devices. You’ll probably want to change this back afterwards, so remember what it was.

filter = [ "a|/dev/nbd.*|", "a|loop|", "a|/dev/[hs]d|", "a/.*/" ]

Set up the partition images as loop devices:

losetup /dev/loop0 /mnt/ext-drive/sda2-data
losetup /dev/loop1 /mnt/ext-drive/sda3-data
losetup /dev/loop2 /mnt/ext-drive/sda5-data

Scan for and activate the volume group:

vgscan
vgchange -ay

Find the logical volume with the data we’re after (LogVol06 in volume group VolGroup00 in this case) and mount it:

lvscan
mount /dev/VolGroup00/LogVol06 /mnt/rescue

Copy the VMWare hard disk files out of the LVM mount, because we know that the volume group stored on them will conflict with the existing one:

cp /mnt/rescue/vmware/vhostname/vhostname.vmdk /rescue/

Check the partition table on the virtual hard disk:

vmware-mount -p /rescue/vhostname.vmdk

This will give you something like:

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-80004
Copyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential
--------------------------------------------

Nr Start      Size       Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
1  63         208782     BIOS 83 Linux
2  208845     16563015   BIOS 8E Unknown

The second partition (with type 8E) is the one we wanted in this case, but it’s LVM-formatted. This means we can’t use vmware-mount because an LVM partition cannot be mounted normally.

We need to unmount the currently-active volume group on the loopback devices (i.e. the LVM from vm-host):

umount /mnt/rescue
# repeat the next line for all volume groups from the loopback devices
vgchange -an VolGroup00
losetup -d /dev/loop0
losetup -d /dev/loop1
losetup -d /dev/loop2

Now you need a dedicated terminal to mount the LVM partition from the virtual hard disk, as this process must stay running in order to access the virtual drive:

vmware-loop /rescue/vhostname.vmdk 2 /dev/nbd0

If you have more than one virtual partition as part of the LVM, you’ll need to have one terminal for each, and select partition numbers and NBD device numbers as appropriate.
Now find the volume group again:

vgscan
vgchange -ay

Find the appropriate logical volume and mount it:

lvscan
mount /dev/VolGroup00/LogVol06 /mnt/rescue

Do whatever you need to do with the data, then unmount it all:

umount /mnt/rescue
vgchange -an VolGroup00
# either press ctrl-c in the vmware-loop terminal(s), or:
killall -INT vmware-loop
# then finally refresh the volume group list to remove old entries
vgscan

And that’s it! I hope this helps someone other than me…

5 Responses to Restoring from LVM and VMWare disks

Aziz B says: March 4, 2009 at 17:06

Thanks for this well-written article. In light of what you have discussed, and the experience you have with LVM and VMWARE, do you suggest any best practices, as far as using LVM in host and guest Linux machines?
I have a 4 disk SCSI array, using RAID 5, which is about 400 Gig logical volume, I have partitioned it with LVM, but I am wondering if I need to use one VG or more? Do guest machines need to be on their own VG..etc..
I am still learning about LVM, so any details would be appreciated.
Thanks again for the good tips of how to restore a host gone bad, hopefully, I won’t have to use them. šŸ™‚

Dave says: March 4, 2009 at 21:08

Hi Aziz,

Thanks for the comment — it’s always nice to be appreciated šŸ™‚

As far as best practices go, I don’t really have any particular advice. It depends on exactly how you have things set up and organised.

The way I tend to configure a system is to have one VG for the operating system, with LVs for /usr, /usr/portage (on Gentoo), /opt, /var, /tmp and then another VG for user data, which might be split into /home and /vol (generic shared data) and possibly others (like /var/www or /var/lib/mysql, /vol/VMs or /vol/music) depending on the machine’s purpose.

Having said this, there’s nothing wrong with having just the one VG, as it provides the most flexibility with your LV partitions.

I also hope nobody else gets into a situation where they have to use this, but if they do I hope they find it helpful and realise that all is not lost!

Terry Stebbens says: March 9, 2009 at 15:44

You are a life saver! I had nearly given myself a brain tumour trying to work this out on some Xen virtual machines and ended up giving up and using straight partitions instead. Now I can go back to using LVMs and all the benefits they provide. Cheers!

Dave says: March 9, 2009 at 17:14

Thanks Terry — glad I could help!

Pingback: Talkwards » Blog Archive » links for 2009-04-03

Leave a Reply

Your email address will not be published. Required fields are marked *

*

*

 

GitHub Google+ Twitter