Offline Migration of KVM Guests

This post is in the category: Guides

Posts here are mostly step-by-step guides on how to replicate something I have set up in the past. Read over my About page to see how I show commands/output and read the disclaimer.

It’s time to enter into summer mode with my lab. By that I mean that it’s time to move important KVM guests off of the HP G5 server and shut that space heater down. Here are my notes with moving guests. If you are looking to move guests from a CentOS to Ubuntu hypervisor, I encourage you to read over my findings. You could also reverse-engineer the article if you are going from Ubuntu to CentOS as well.

Note that this procedure is called an “offline migration”. This means that the guest will be shut down and unavailable during the migration process.

The Long Story / Learning Phase

In this example, note the hostnames:

  • server-10 – the HP DL360 G5 hypervisor
  • atw-05 – my desktop machine to be holding the VMs while the G5 is powered down
  • s10-intel – the example guest to be moved

First, stop the guest and dump the VM settings.

[root@server-10]$ virsh shutdown s10-intel
[root@server-10]$ virsh dumpxml s10-intel > /root/vms/s10-intel.xml

Take a look at the image location.

[root@server-10]$ grep '.img' /root/vms/s10-intel.xml
      <source file='/var/lib/libvirt/images2/s10-intel.img'/>

Copy the XML dump to the new host.

[root@server-10]$ rsync -avSHP s10-intel.xml atw-05:/root/vms/

Copy the image to the new host.

[root@server-10]$ rsync -avSHP /var/lib/libvirt/images2/s10-intel.img atw-05:/var2/lib/libvirt/images/

Note that the image location on the new host is different. We should modify the XML file before importing into the new host.

[root@atw-05]$ vim /root/vms/s10-intel.xml

Now import it into the new host.

[root]$ virsh define /root/vms/s10-intel.xml
error: Failed to define domain from /root/vms/s10-intel.xml
error: Cannot find QEMU binary /usr/libexec/qemu-kvm: No such file or directory

Hmmm…I should point out that I am moving the guests from a Centos 6 hypervisor to a hypervisor running Ubuntu 12.04. These binaries must be in different places.

After taking a peak at an XML dump from a VM created in Ubuntu 12.04, I found that the emulator path is /usr/bin/kvm. So we have to edit the XML file one last time.

[root@atw-05]$ vim /root/vms/s10-intel.xml

Update this line:

<emulator>/usr/bin/kvm</emulator>

Now we shall try to import it again.

[root]$ virsh define /root/vms/s10-intel.xml
Domain s10-intel defined from /root/vms/s10-intel.xml

Yay, we have success this time. Shall we go for broke and start it up?

[root@atw-05]$ virsh start s10-intel
error: Failed to start domain s10-intel
error: internal error Process exited while reading console log output: Supported machines are:
pc         Standard PC (alias of pc-1.0)
pc-1.0     Standard PC (default)
pc-0.14    Standard PC
pc-0.13    Standard PC
pc-0.12    Standard PC
pc-0.11    Standard PC, qemu 0.11
pc-0.10    Standard PC, qemu 0.10
isapc      ISA-only PC

Darn, more issues. Let’s take another look at that XML file.

[root@atw-05]$ grep 'machine' /root/vms/s10-intel.xml
    <type arch='x86_64' machine='rhel6.5.0'>hvm</type>

Ah, we are not on a RH (CentOS) machine anymore. So what to we put there? Well, that error message above stated pc-1.0 to be the default, so I am going to use that.

[root@atw-05]$ vim /root/vms/s10-intel.xml

Update the machine parameter to pc-1.0.

We have to re-define the machine to apply the changes.

[root@atw-05]$ virsh define /root/vms/s10-intel.xml
Domain s10-intel defined from /root/vms/s10-intel.xml

Now attempt to start it again.

[root@atw-05]$ virsh start s10-intel
Domain s10-intel started

Migration complete.

The End Result / Condensed Procedure

This second example will be with the guest s10-jump, and will skip over the issues we are now aware of.

First, stop the guest and dump the VM settings.

[root@server-10]$ virsh shutdown s10-jump
[root@server-10]$ virsh dumpxml s10-jump > /root/vms/s10-jump.xml

Take a look at the image location.

[root@server-10]$ grep '.img' /root/vms/s10-jump.xml
      <source file='/var/lib/libvirt/images/s10-jump.img'/>

Copy the XML dump to the new host.

[root@server-10]$ rsync -avSHP /root/vms/s10-jump.xml atw-05:/root/vms/

Copy the image to the new host.

[root@server-10]$ rsync -avSHP /var/lib/libvirt/images/s10-jump.img atw-05:/var2/lib/libvirt/images/

Now we ssh into the new hypervisor and make a few tweaks before importing and starting.

[root@atw-05]$ vim /root/vms/s10-jump.xml

This box has images stored at /var2/lib/libvirt/images/. Update the source image line accordingly.

We learned that Ubuntu 12.04 has a different emulator binary. Update that from /usr/libexec/qemu-kvm to /usr/bin/kvm.

Finally, change the machine type from rhel6.4.0 to pc-1.0. Then save and close the file.

Now import the VM settings.

[root]$ virsh define /root/vms/s10-jump.xml
Domain s10-jump defined from /root/vms/s10-jump.xml

Yay, no errors this time! Now start it.

[root]$ virsh start s10-jump
Domain s10-jump started

Migration complete.

Nice! I have about 5 more VMs to go. Some of them I will not start up to save resources on my desktop, but once I am done, I will be in “low-power mode”.

Configuring Autostart

Once all VMs are moved off this machine, I want to make sure that these don’t startup automatically, causing MAC conflicts on my LAN when this machine gets booted up in the fall. I’m not sure how to list all autostart VMs, but I do know that they are configured by placing a symlink in a certain directory.

[root@server-10]$ ls -alh /etc/libvirt/qemu/autostart/
total 8.0K
drwxr-xr-x. 2 root root 4.0K Apr 12 22:39 .
drwx------. 4 root root 4.0K May 18 14:55 ..
lrwxrwxrwx. 1 root root   30 Apr 12 22:39 s10-jump.xml -> /etc/libvirt/qemu/s10-jump.xml
lrwxrwxrwx. 1 root root   31 Apr 12 22:39 s10-mc-01.xml -> /etc/libvirt/qemu/s10-mc-01.xml
lrwxrwxrwx. 1 root root   31 Apr 12 22:39 s10-mc-02.xml -> /etc/libvirt/qemu/s10-mc-02.xml
lrwxrwxrwx. 1 root root   31 Apr 12 22:39 s10-mc-03.xml -> /etc/libvirt/qemu/s10-mc-03.xml
lrwxrwxrwx. 1 root root   32 Apr 12 22:39 s10-nagios.xml -> /etc/libvirt/qemu/s10-nagios.xml
lrwxrwxrwx. 1 root root   32 Apr 12 22:39 s10-puppet.xml -> /etc/libvirt/qemu/s10-puppet.xml

So these are the machines that are set to start automatically. Let’s turn that off. While I probably could just delete these symlinks, I will run the virsh command for each one.

[root@server-10]$ virsh autostart --disable s10-jump
Domain s10-jump unmarked as autostarted

…and so-on. Finally, I configure some of these machines to autostart on the guests’ new hypervisor.

[root@atw-05]$ virsh autostart s10-puppet

Conclusion

If you came here looking for tips on moving VMs, I hope my findings helped in some way!

This entry was posted in Guides and tagged , on by .

About Andrew Wells

I have been developing on the LAMP stack since about 2006. I run Ubuntu XFCE on my desktop and have a history of managing Ubuntu and CentOS servers. I code web applications mostly in PHP but have experience with other languages as well. When I'm not working, I can be found working in my home lab or out snowboarding, hiking, camping, or biking depending on the season.

Leave a Reply

Your email address will not be published. Required fields are marked *