Skip to contentSkip to conversation

Block Migration and xbzrle

DISCUSSION Peter Lieven 4 participants 10 messages Sep 16, 2012 – Oct 4, 2012 KVM

lore
Share & tools
lore
#0 lore raw

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

Thanks,
Peter

#1 lore raw
Quoted 6 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration. What is the error you are getting?
What command line ?

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

#2 lore raw
Quoted 8 lines · Orit Wasserman
Quoted 7 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Quoted 5 lines
Quoted 1 line

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration.
What is the error you are getting?
What command line ?

Meanwhile I can confirm that it happens with and without block migration. I I observe 2 errors:
a)
qemu: warning: error while loading state section id 2
load of migration failed
b)
the vm does not enter running state after migration.

The command-line:
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=798,script=no,downscript=no,ifname=tap1 -net
nic,vlan=798,model=e1000,macaddr=52:54:00:ff:01:15 -drive
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-d85f4e007-3f30017ce11505df-ubuntu-tools-hd0,if=virtio,cache=none,aio=native
-m 4096 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'Ubuntu-Tools' -boot order=dc,menu=off -k de -incoming
tcp:172.21.55.34:5002 -pidfile /var/run/qemu/vm-250.pid -mem-path
/hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no-hpet
-vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU
L5640 @ 2.27GHz',-tsc

Thanks,
Peter

Quoted 11 lines

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

#3 lore raw
Quoted 23 lines · On 10/02/2012 10:33 AM, [email protected] wrote:
Quoted 8 lines · Orit Wasserman
Quoted 7 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Quoted 5 lines
Quoted 1 line

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration.
What is the error you are getting?
What command line ?

Meanwhile I can confirm that it happens with and without block migration. I I observe 2 errors:
a)
qemu: warning: error while loading state section id 2
load of migration failed
b)
the vm does not enter running state after migration.

Quoted 11 lines

The command-line:
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=798,script=no,downscript=no,ifname=tap1 -net
nic,vlan=798,model=e1000,macaddr=52:54:00:ff:01:15 -drive
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-d85f4e007-3f30017ce11505df-ubuntu-tools-hd0,if=virtio,cache=none,aio=native
-m 4096 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'Ubuntu-Tools' -boot order=dc,menu=off -k de -incoming
tcp:172.21.55.34:5002 -pidfile /var/run/qemu/vm-250.pid -mem-path
/hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no-hpet
-vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU

Migration with -cpu host is very problemtic, because the source and destination can have different cpu resulting in different cpu features.
Does regular migration works with this setup?
Can you try with a different cpu type?
What are the source and destination /proc/cpuinfo output ?

Cheers,
Orit

Quoted 23 lines

L5640 @ 2.27GHz',-tsc

Thanks,
Peter

Quoted 11 lines

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

Am 02.10.2012 um 11:28 schrieb Orit Wasserman:

Quoted 24 lines · On 10/02/2012 10:33 AM, [email protected] wrote:
Quoted 23 lines
Quoted 8 lines · Orit Wasserman
Quoted 7 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Quoted 5 lines
Quoted 1 line

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration.
What is the error you are getting?
What command line ?

Meanwhile I can confirm that it happens with and without block migration. I I observe 2 errors:
a)
qemu: warning: error while loading state section id 2
load of migration failed
b)
the vm does not enter running state after migration.

Quoted 16 lines
Quoted 11 lines

The command-line:
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=798,script=no,downscript=no,ifname=tap1 -net
nic,vlan=798,model=e1000,macaddr=52:54:00:ff:01:15 -drive
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-d85f4e007-3f30017ce11505df-ubuntu-tools-hd0,if=virtio,cache=none,aio=native -m 4096 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'Ubuntu-Tools' -boot order=dc,menu=off -k de -incoming
tcp:172.21.55.34:5002 -pidfile /var/run/qemu/vm-250.pid -mem-path
/hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no-hpet
-vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU

Migration with -cpu host is very problemtic, because the source and destination can have different cpu resulting in different cpu features.
Does regular migration works with this setup?
Can you try with a different cpu type?
What are the source and destination /proc/cpuinfo output ?

The CPUs are identical, we also check if flags and cpu types match if cpu type is set to host. Regular migration does work.

BR,
Peter

Quoted 28 lines

Cheers,
Orit

Quoted 23 lines

L5640 @ 2.27GHz',-tsc

Thanks,
Peter

Quoted 11 lines

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#7 lore raw
In reply to Peter Lieven

Il 16/09/2012 12:39, Peter Lieven ha scritto:

Quoted 7 lines

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

Block migration is broken by design. It will converge really slowly as soon as you have real load in the VMs, and it will hamper the convergence of RAM as well.

Hopefully a real alternative will be in 1.3 (based on drive-mirror on the source + an embedded NBD server running on the destination), then in 1.4 we can reimplement the block migration monitor commands using the alternative.

Paolo

#8 lore raw

Am 02.10.2012 um 11:38 schrieb Paolo Bonzini:

Quoted 9 lines

Il 16/09/2012 12:39, Peter Lieven ha scritto:

Quoted 7 lines

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

Quoted 8 lines

Block migration is broken by design. It will converge really slowly as soon as you have real load in the VMs, and it will hamper the convergence of RAM as well.

Hopefully a real alternative will be in 1.3 (based on drive-mirror on the source + an embedded NBD server running on the destination), then in 1.4 we can reimplement the block migration monitor commands using the alternative.

Hi Paolo, i know that block migration is not that good, but it seems that there is a bug in XBZRLE that is independent of block migration.

Peter

Quoted 2 lines

Paolo

maintainer
#9 lore raw

Il 02/10/2012 11:44, Peter Lieven ha scritto:

Quoted 16 lines

Am 02.10.2012 um 11:38 schrieb Paolo Bonzini:

Quoted 13 lines

Il 16/09/2012 12:39, Peter Lieven ha scritto:

Quoted 7 lines

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

Block migration is broken by design. It will converge really slowly as soon as you have real load in the VMs, and it will hamper the convergence of RAM as well.

Quoted 7 lines
Quoted 4 lines

Hopefully a real alternative will be in 1.3 (based on drive-mirror on the source + an embedded NBD server running on the destination), then in 1.4 we can reimplement the block migration monitor commands using the alternative.

Hi Paolo, i know that block migration is not that good, but it seems that there is a bug in XBZRLE that is independent of block migration.

Understood---but hopefully you can stop using it with 1.3, which would also work around the bug. :)

Paolo

#5 lore raw
In reply to Peter Lieven
Quoted 24 lines · Peter Lieven

Am 02.10.2012 um 11:28 schrieb Orit Wasserman:

Quoted 21 lines · On 10/02/2012 10:33 AM, [email protected] wrote:
Quoted 20 lines
Quoted 8 lines · Orit Wasserman
Quoted 7 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Quoted 5 lines
Quoted 1 line

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration.
What is the error you are getting?
What command line ?

Meanwhile I can confirm that it happens with and without block migration. I I observe 2 errors:
a)
qemu: warning: error while loading state section id 2
load of migration failed

Did you enabled XBZRLE on the destination also?
(migrate_set_capability xbzrle on)

Orit

Quoted 59 lines
Quoted 19 lines
Quoted 14 lines

b)
the vm does not enter running state after migration.

The command-line:
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=798,script=no,downscript=no,ifname=tap1 -net
nic,vlan=798,model=e1000,macaddr=52:54:00:ff:01:15 -drive
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-d85f4e007-3f30017ce11505df-ubuntu-tools-hd0,if=virtio,cache=none,aio=native -m 4096 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'Ubuntu-Tools' -boot order=dc,menu=off -k de -incoming
tcp:172.21.55.34:5002 -pidfile /var/run/qemu/vm-250.pid -mem-path
/hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no-hpet
-vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU

Migration with -cpu host is very problemtic, because the source and destination can have different cpu resulting in different cpu features.
Does regular migration works with this setup?
Can you try with a different cpu type?
What are the source and destination /proc/cpuinfo output ?

The CPUs are identical, we also check if flags and cpu types match if cpu type is set to host. Regular migration does work.

BR,
Peter

Quoted 28 lines

Cheers,
Orit

Quoted 23 lines

L5640 @ 2.27GHz',-tsc

Thanks,
Peter

Quoted 11 lines

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Am 02.10.2012 um 12:40 schrieb Orit Wasserman:

Quoted 20 lines · Peter Lieven
Quoted 19 lines

Am 02.10.2012 um 11:28 schrieb Orit Wasserman:

Quoted 16 lines · On 10/02/2012 10:33 AM, [email protected] wrote:
Quoted 15 lines
Quoted 13 lines · Orit Wasserman
Quoted 8 lines · Peter Lieven

Hi,

I remember that this was broken some time ago and currently with qemu-kvm 1.2.0 I am still not able to use
block migration plus xbzrle. The migration fails if both are used together. XBZRLE without block migration works.

Can someone please advise what is the current expected behaviour?

XBZRLE only work on guest memory so it shouldn't be effected by block migration.
What is the error you are getting?
What command line ?

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Meanwhile I can confirm that it happens with and without block migration. I I observe 2 errors:
a)
qemu: warning: error while loading state section id 2
load of migration failed

Did you enabled XBZRLE on the destination also?
(migrate_set_capability xbzrle on)

I was not aware that I have to enable it on both sides. I thought it had to be enabled only on the source side. However, it seems that it is enabled by default in 1.2.0?!
I will retry with enabling it with the above command
on both sides.

Peter

Quoted 62 lines

Orit

Quoted 59 lines
Quoted 19 lines
Quoted 14 lines

b)
the vm does not enter running state after migration.

The command-line:
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=798,script=no,downscript=no,ifname=tap1 -net
nic,vlan=798,model=e1000,macaddr=52:54:00:ff:01:15 -drive
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-d85f4e007-3f30017ce11505df-ubuntu-tools-hd0,if=virtio,cache=none,aio=native -m 4096 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'Ubuntu-Tools' -boot order=dc,menu=off -k de -incoming
tcp:172.21.55.34:5002 -pidfile /var/run/qemu/vm-250.pid -mem-path
/hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no-hpet
-vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU

Migration with -cpu host is very problemtic, because the source and destination can have different cpu resulting in different cpu features.
Does regular migration works with this setup?
Can you try with a different cpu type?
What are the source and destination /proc/cpuinfo output ?

The CPUs are identical, we also check if flags and cpu types match if cpu type is set to host. Regular migration does work.

BR,
Peter

Quoted 28 lines

Cheers,
Orit

Quoted 23 lines

L5640 @ 2.27GHz',-tsc

Thanks,
Peter

Quoted 11 lines

Regards,
Orit

Quoted 5 lines

Thanks,
Peter

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html