qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

Hi,
I'm very sorry for crossposting, but I'm not sure whether my problem
is related to libvirt or kvm itself.
I've spent some time testing save/restore and it seems it's still not really reliable. Starting guest, loading it with some applications
(ie kernel compilation) and saving/restoring it for few times usually leads to error:

[root@vbox4 hibernated]# virsh save vmroo55 /home/LVS/hibernated/vmroo55.vm Domain vmroo55 saved to /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh restore /home/LVS/hibernated/vmroo55.vm Domain restored from /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh save vmroo55 /home/LVS/hibernated/vmroo55.vm Domain vmroo55 saved to /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh restore /home/LVS/hibernated/vmroo55.vm
error: Failed to restore domain from /home/LVS/hibernated/vmroo55.vm
error: internal error unable to start guest: char device redirected to /dev/pts/2

in /var/log/messages, i see:

Jun 12 15:18:30 vbox4 libvirtd: 15:18:30.793: error : qemudReadMonitorOutput:696 : internal error Timed out while reading monitor startup output
Jun 12 15:18:30 vbox4 libvirtd: 15:18:30.820: error : qemudWaitForMonitor:994 : internal error unable to start guest: char device redirected to /dev/pts/2

/var/log/libvirt/vmroo55.log:
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/ /usr/bin/qemu-kvm -S -M pc -m 768 -smp 2 -name vmroo55 -uuid 987e7a85-0e15-11de-a522-00304892f672 -monitor pty -boot c -kernel /home/LVS/boot/vmlinuz-2.6.29lb.00_01_PRE14-x86_64 -initrd /home/LVS/boot/kvm-initrd-2.6.29lb.00_01_PRE14-x86_64.cpio.gz -append root=/dev/sda ro vmhostname=vmroo55 -drive file=/dev/vgshared/vmroo55-1,if=ide,index=0,boot=on -drive file=/dev/vgshared/vmroo55-2,if=ide,index=1 -drive file=/dev/vgshared/vmroo55-3,if=virtio,index=0 -net nic,macaddr=00:16:3e:18:37:00,vlan=0,model=virtio -net tap,fd=15,vlan=0 -serial none -parallel none -usb -vnc 0.0.0.0:24155 -incoming exec:cat char device redirected to /dev/pts/2
qemu_popen: returning result of qemu_fopen_ops
qemu_popen: returning result of qemu_fopen_ops

I wanted to try it using just kvm, but for some reason, I can't figure the proper way to execute it, is the -incoming exec:cat somewhere properly documented? I tried doing cat .../img.vm | qemu-kvm ... -incoming exec:cat but this doesn't seem to be the proper way.

Could somebody give me a hint on how to debug this?

thanks a lot in advance!

nik

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------

That should work fine. Your "..." sets up the drives, memory amount, etc., right? Why do you say it "doesn't seem to be the proper way"? What fails?

BTW, you might also do this: -incoming 'exec:cat .../img.vm', avoiding the need to pipe through stdin.

Hello Charlie,
thanks for the reply

sure, I'm (hopefully) providing all the needed arguments.

yup, I tried that, but it allways ends with following:

cat /home/LVS/hibernated/vmroo55.vm | /usr/bin/qemu-kvm -M pc -m 768 -smp 2 -name vmroo55 -uuid 987e7a85-0e15-11de-a522-00304892f672 -monitor pty -boot c -kernel /home/LVS/boot/vmlinuz-2.6.29lb.00_01_PRE14-x86_64 -initrd /home/LVS/boot/kvm-initrd-2.6.29lb.00_01_PRE14-x86_64.cpio.gz -append "root=/dev/sda ro vmhostname=vmroo55" -drive file=/dev/vgshared/vmroo55-1,if=ide,index=0,boot=on -drive file=/dev/vgshared/vmroo55-2,if=ide,index=1 -drive file=/dev/vgshared/vmroo55-3,if=virtio,index=0 -net nic,macaddr=00:16:3e:18:37:00,vlan=0,model=virtio -net tap,script=/etc/qemu-ifup.sh,vlan=0 -serial none -parallel none -usb -vnc0.0.0.0:24155 -incoming exec:cat

Executing /etc/qemu-ifup
Bringing up tap0 for bridged mode... Adding tap0 to br0...
char device redirected to /dev/pts/7
qemu_popen: returning result of qemu_fopen_ops load of migration failed
cat: write error: Broken pipe /etc/qemu-ifdown: could not launch network script

using
... -incoming exec:cat /home/...image.vm
seems to start the machine, but it seems to be stuck, even though I removed -S parameter.

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------

That may well be a guest issue rather than a qemu migration problem -- I've seen similar difficulties for guests being resumed via incoming migration when using guest kernels with TSC as their clocksource.

You might try interrogating the monitor to see your guest's state.

Hmm,
I just noticed that dump file saved using virsh contains some libvirt related headers, so I guess I can't just feed it directly to qemu-kvm. So I guess best way to test save/restore without libvirt is to save vm state using monitor and then restore it again.
But I can't find any clear documentation on how to do this. Could anyone give me a hint?
thanks a lot!
nik

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------

There's savevm and loadvm in the monitor.

error compiling committee.c: too many arguments to function

Hello Charles,

I tried that, saving seems to work OK,

but if I restore the domain, kvm always seems to be stuck, I'm not able to swith to monitor to do anything. running strace on the kvm process show that following is repeating:

clock_gettime(CLOCK_MONOTONIC, {453254, 437209647}) = 0 clock_gettime(CLOCK_MONOTONIC, {453254, 437244463}) = 0 clock_gettime(CLOCK_MONOTONIC, {453254, 437295938}) = 0
select(20, [5 10 17 19], [], [], {1, 0}) = 1 (in [5], left {0, 999998})
read(5, "\0", 512) = 1
read(5, 0x7fff60506840, 512) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {453254, 437458602}) = 0 clock_gettime(CLOCK_MONOTONIC, {453254, 437492440}) = 0
timer_gettime(0, {it_interval={0, 0}, it_value={0, 0}}) = 0
timer_settime(0, 0, {it_interval={0, 0}, it_value={0, 30000000}}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {453254, 437605596}) = 0 clock_gettime(CLOCK_MONOTONIC, {453254, 437639489}) = 0
select(20, [5 10 17 19], [], [], {1, 0}) = 1 (in [19], left {0, 970104})
read(19, "\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128 rt_sigaction(SIGALRM, NULL, {0x4083b0, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f25576e94c0}, 8) = 0
write(6, "\0", 1) = 1
read(19, 0x7fff605069c0, 128) = -1 EAGAIN (Resource temporarily unavailable)

I guess I maybe need to unfreeze the domain somehow after loading it?

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------