Qemu-KVM Livate Migration 0.12.2 -> 0.12.3/4 broken?

Hi Qemu/KVM Devel Team,

Live Migration from a 0.12.2 qemu-kvm to a 0.12.3 (and 0.12.4) does not work: "load of migration failed"

Is there any way to find out, why exactly it fails? I have a lot of VMs running on 0.12.2 and would like to migrate them to 0.12.4

cmdline:
-net tap,vlan=6,script=no,downscript=no,ifname=tap7 -net
nic,vlan=6,model=e1000,macaddr=52:54:00:fe:00:88 -net
tap,vlan=651,script=no,downscript=no,ifname=tap8 -net
nic,vlan=651,model=e1000,macaddr=52:54:00:ff:00:69 -drive
file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-b6eca7604-6280020bc4b4bde8-quagga,if=ide,boot=on,cache=none,aio=native -m 256 -monitor tcp:0:4090,server,nowait -vnc :90 -name 'Quagga' -boot
order=dc,menu=on -k de -incoming tcp:172.21.59.132:5090 -pidfile
/var/run/qemu/vm-148.pid -rtc base=utc,clock=vm -usb -usbdevice
tablet -no-kvm-irqchip -vga cirrus

Any hints would be appreciated!

Quoted 20 lines · Peter Lieven

Hi Qemu/KVM Devel Team,

Live Migration from a 0.12.2 qemu-kvm to a 0.12.3 (and 0.12.4) does not work: "load of migration failed"

Is there any way to find out, why exactly it fails? I have a lot of VMs running on 0.12.2 and would like to migrate them to 0.12.4

cmdline:
-net tap,vlan=6,script=no,downscript=no,ifname=tap7 -net
nic,vlan=6,model=e1000,macaddr=52:54:00:fe:00:88 -net
tap,vlan=651,script=no,downscript=no,ifname=tap8 -net
nic,vlan=651,model=e1000,macaddr=52:54:00:ff:00:69 -drive
file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-b6eca7604-6280020bc4b4bde8-quagga,if=ide,boot=on,cache=none,aio=native -m 256 -monitor tcp:0:4090,server,nowait -vnc :90 -name 'Quagga' -boot
order=dc,menu=on -k de -incoming tcp:172.21.59.132:5090 -pidfile
/var/run/qemu/vm-148.pid -rtc base=utc,clock=vm -usb -usbdevice
tablet -no-kvm-irqchip -vga cirrus

Quoted 2 lines

Any hints would be appreciated!

Can you try reverting this patch?

commit 3fa017e24b0a0f0e68619a689b9b02fe486dae9e

Author: Marcelo Tosatti <[email protected]>
Date:   Thu Feb 11 18:19:44 2010 -0200

ide save/restore pio/atapi cmd transfer fields and io buffer

and told me if it works?

Later, Juan.

Quoted 20 lines · Peter Lieven

Hi Qemu/KVM Devel Team,

Live Migration from a 0.12.2 qemu-kvm to a 0.12.3 (and 0.12.4) does not work: "load of migration failed"

Is there any way to find out, why exactly it fails? I have a lot of VMs running on 0.12.2 and would like to migrate them to 0.12.4

cmdline:
-net tap,vlan=6,script=no,downscript=no,ifname=tap7 -net
nic,vlan=6,model=e1000,macaddr=52:54:00:fe:00:88 -net
tap,vlan=651,script=no,downscript=no,ifname=tap8 -net
nic,vlan=651,model=e1000,macaddr=52:54:00:ff:00:69 -drive
file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-b6eca7604-6280020bc4b4bde8-quagga,if=ide,boot=on,cache=none,aio=native -m 256 -monitor tcp:0:4090,server,nowait -vnc :90 -name 'Quagga' -boot
order=dc,menu=on -k de -incoming tcp:172.21.59.132:5090 -pidfile
/var/run/qemu/vm-148.pid -rtc base=utc,clock=vm -usb -usbdevice
tablet -no-kvm-irqchip -vga cirrus

Quoted 2 lines

Any hints would be appreciated!

Can you try reverting this patch?

commit 3fa017e24b0a0f0e68619a689b9b02fe486dae9e

Author: Marcelo Tosatti<[email protected]>
Date:   Thu Feb 11 18:19:44 2010 -0200

ide save/restore pio/atapi cmd transfer fields and io buffer

and told me if it works?

Quoted 30 lines · Juan Quintela

Quoted 20 lines · Peter Lieven

Hi Qemu/KVM Devel Team,

Live Migration from a 0.12.2 qemu-kvm to a 0.12.3 (and 0.12.4) does not work: "load of migration failed"

Is there any way to find out, why exactly it fails? I have a lot of VMs running on 0.12.2 and would like to migrate them to 0.12.4

cmdline:
-net tap,vlan=6,script=no,downscript=no,ifname=tap7 -net
nic,vlan=6,model=e1000,macaddr=52:54:00:fe:00:88 -net
tap,vlan=651,script=no,downscript=no,ifname=tap8 -net
nic,vlan=651,model=e1000,macaddr=52:54:00:ff:00:69 -drive
file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-b6eca7604-6280020bc4b4bde8-quagga,if=ide,boot=on,cache=none,aio=native -m 256 -monitor tcp:0:4090,server,nowait -vnc :90 -name 'Quagga' -boot
order=dc,menu=on -k de -incoming tcp:172.21.59.132:5090 -pidfile
/var/run/qemu/vm-148.pid -rtc base=utc,clock=vm -usb -usbdevice
tablet -no-kvm-irqchip -vga cirrus

Quoted 2 lines

Any hints would be appreciated!

Can you try reverting this patch?

commit 3fa017e24b0a0f0e68619a689b9b02fe486dae9e

Author: Marcelo Tosatti<[email protected]>
Date:   Thu Feb 11 18:19:44 2010 -0200

Quoted 5 lines

ide save/restore pio/atapi cmd transfer fields and io buffer

and told me if it works?

Any idea why it fails? And how to fix it?

Quoted 3 lines

Any idea why it fails? And how to fix it?

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
     .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
     .name = "ide",
     .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

So, to make the story short: I know what is happening, and I know how to fix it, just that fix is not trivial. I just need time.

Quoted 28 lines · Juan Quintela

Quoted 3 lines

Any idea why it fails? And how to fix it?

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
     .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
     .name = "ide",
     .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Quoted 9 lines

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

end of next week?

Quoted 3 lines

So, to make the story short: I know what is happening, and I know how to fix it, just that fix is not trivial. I just need time.

Meanwhile, we have a broken 0.12.4. Is there a quick'n'dirty workaround that will be forward compatible with the real fix that we can push out?

We've regressed from failing some migrations to failing all migrations.

Quoted 5 lines

Quoted 5 lines

So, to make the story short: I know what is happening, and I know how to fix it, just that fix is not trivial. I just need time.

Quoted 4 lines

Meanwhile, we have a broken 0.12.4. Is there a quick'n'dirty workaround that will be forward compatible with the real fix that we can push out?

revert the patch. It almost never happen (being in the middle of one IO) while migrating.

Quoted 3 lines

We've regressed from failing some migrations to failing all migrations.

Humm, 0.12.4 -> 0.12.4 should work. My advise is just revert the patch and live with it for another week, what do you think?

On 05/16/2010 11:04 PM, Juan Quintela wrote:

Quoted 3 lines

Quoted 1 line

We've regressed from failing some migrations to failing all migrations.

Humm, 0.12.4 -> 0.12.4 should work. My advise is just revert the patch and live with it for another week, what do you think?

A week is fine. I'm not going to release 0.12.4.1 for the week until you fix it, and just changing things in git doesn't help users.

Quoted 1 line · Avi Kivity

On 05/16/2010 11:04 PM, Juan Quintela wrote:

[]

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

We've regressed from failing some migrations to failing all migrations.

Humm, 0.12.4 -> 0.12.4 should work. My advise is just revert the patch and live with it for another week, what do you think?

A week is fine. I'm not going to release 0.12.4.1 for the week until you fix it, and just changing things in git doesn't help users.

There are at least 1 other problem in the migration code that's worth looking at (IMHO anyway): namely, migration on 32 bit host is completely (as far as i can see) broken (instant kvm process crash) in 0.12, but it works on 64bits:

http://www.mail-archive.com/[email protected]/msg34051.html

I wonder why it is not noticed before -- it's broken since 0.12...

Michael Tokarev<[email protected]> wrote:

Quoted 2 lines

http://www.mail-archive.com/[email protected]/msg34051.html
I wonder why it is not noticed before -- it's broken since 0.12...

People has become rich and everybody has a 64 bit hardware :-)

Will take a look at the end of the week.

Quoted 1 line · Juan Quintela

Michael Tokarev<[email protected]> wrote:

[]

Quoted 3 lines

Quoted 2 lines

http://www.mail-archive.com/[email protected]/msg34051.html
I wonder why it is not noticed before -- it's broken since 0.12...

Quoted 1 line

People has become rich and everybody has a 64 bit hardware :-)

Actually I don't think there's a CPU that's 32bits and has necessary VT instructions. At least not the ones to care about - maybe there were some models but they'e not common and are gone long time ago.

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
    .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
    .name = "ide",
    .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

Quoted 25 lines · Juan Quintela

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
    .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
    .name = "ide",
    .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Quoted 8 lines

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

BTW, the IDE subsystem is yet lacking a proper vmstate section split-up along qdev boundaries (ie. vmstate_ide_pci should not contain drive structures). Do you plan to address this as well?

Quoted 34 lines · Jan Kiszka

Quoted 33 lines · Juan Quintela

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
    .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
    .name = "ide",
    .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

Quoted 3 lines

BTW, the IDE subsystem is yet lacking a proper vmstate section split-up along qdev boundaries (ie. vmstate_ide_pci should not contain drive structures). Do you plan to address this as well?

Not for Friday, and not for 0.12.

That is 0.13 material, and have to get one agreement on how to go. We can go for:
- good structure
- backward compatibility

I can't see any good way to get both at this stage :( But I am open to sugestions.

Later, Juan.

PD. BTW, very good work with printing the vmstate, that was one of the goals
when we added it, that was the next step after porting everything to vmstate :)

Quoted 38 lines · Juan Quintela

Quoted 36 lines · Jan Kiszka

Quoted 34 lines · Juan Quintela

Lack of "proper" subsections. IDE is something like:

const VMStateDescription vmstate_ide_drive = {
    .version_id = 4,

....
}

static const VMStateDescription vmstate_bmdma = {
.name = "ide bmdma",
.version_id = 4,
...
}

const VMStateDescription vmstate_ide_pci = {
    .name = "ide",
    .version_id = 4,

....
VMSTATE_STRUCT_ARRAY(bmdma, PCIIDEState, 2, 0,
vmstate_bmdma, BMDMAState),
VMSTATE_IDE_DRIVES(bus[0].ifs, PCIIDEState), VMSTATE_IDE_DRIVES(bus[1].ifs, PCIIDEState),
....
}

Notice that everything is at version 4. It used to be everything at version 3. Now the problem is that when migrating from v3 -> v4. We put in one place v3, But we only have a version number at the toplevel, rest of "subsections" don't sent a version number. There is no way to fix it in the general case. We can hack something around for ide, but that will just be a hack, or we can backport marcelo change and port it as a proper subsection (that is my plan). I expect to have time at the end of next time to work on this.

Quoted 6 lines

Quoted 4 lines

BTW, the IDE subsystem is yet lacking a proper vmstate section split-up along qdev boundaries (ie. vmstate_ide_pci should not contain drive structures). Do you plan to address this as well?

Not for Friday, and not for 0.12.

For sure. I missed that this was only a 0.12 issue.

Quoted 8 lines

That is 0.13 material, and have to get one agreement on how to go. We can go for:
- good structure
- backward compatibility

I can't see any good way to get both at this stage :( But I am open to sugestions.

Based on recent experiments with vmstate to enhance the hpet, I'm fairly optimistic that we can have both (just the code complexity suffers a bit): Split up the drive sections for new versions, but keep the legacy fields with attached .field_exists() filters for reading of old versions. But I may also underestimate issues of this particular case.

Quoted 7 lines

Later, Juan.

PD. BTW, very good work with printing the vmstate, that was one of the goals
when we added it, that was the next step after porting everything to vmstate :)

I'm sorry for stealing you the pleasure to add it. :)

Jan