Migration issues in qemu.git

DISCUSSION Avi Kivity 2 participants 5 messages Aug 2, 2010 KVM

lore

Share & tools

lore

Avi Kivity

author

2010-08-02 09:06 UTC #0 lore raw

I'm hitting some migration issues merging qemu.git into qemu-kvm.git:

1. Crash in mig_cancel test:

(gdb) bt

#0  0x0000003a91c83dbb in memcpy () from /lib64/libc.so.6
#1  0x000000000049c2ff in qemu_get_buffer (f=0x302d870, buf=<value optimized out>, size1=4096) at /usr/include/bits/string3.h:52
#2  0x0000000000409464 in ram_load (f=0x302d870, opaque=<value optimized out>, version_id=4) at /build/home/tlv/akivity/qemu-kvm/arch_init.c:407
#3  0x000000000049cb4c in qemu_loadvm_state (f=0x302d870) at savevm.c:1708
#4  0x0000000000494169 in process_incoming_migration (f=<value optimized out>) at migration.c:63
#5  0x0000000000494517 in tcp_accept_incoming_migration (opaque=<value optimized out>) at migration-tcp.c:163
#6  0x000000000041b67e in main_loop_wait (nonblocking=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:1300
#7  0x00000000004314e7 in kvm_main_loop () at /build/home/tlv/akivity/qemu-kvm/qemu-kvm.c:1710
#8  0x000000000041c67f in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /build/home/tlv/akivity/qemu-kvm/vl.c:1340

#9 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:3069

This is on the incoming side so the test completes successfully, only leaving a core dump to fill my disks.

2. Intermittent migration failures

Likely suspect is the IDE migration fixes. A merge of the commit just before these changes passes the test, I'm now testing the commit immediately after the changes.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

author

2010-08-02 09:42 UTC #1 lore raw

Quoted 15 lines · Avi Kivity

I'm hitting some migration issues merging qemu.git into qemu-kvm.git:

1. Crash in mig_cancel test:

(gdb) bt

#0  0x0000003a91c83dbb in memcpy () from /lib64/libc.so.6
#1  0x000000000049c2ff in qemu_get_buffer (f=0x302d870, buf=<value optimized out>, size1=4096) at /usr/include/bits/string3.h:52
#2  0x0000000000409464 in ram_load (f=0x302d870, opaque=<value optimized out>, version_id=4) at /build/home/tlv/akivity/qemu-kvm/arch_init.c:407
#3  0x000000000049cb4c in qemu_loadvm_state (f=0x302d870) at savevm.c:1708
#4  0x0000000000494169 in process_incoming_migration (f=<value optimized out>) at migration.c:63
#5  0x0000000000494517 in tcp_accept_incoming_migration (opaque=<value optimized out>) at migration-tcp.c:163
#6  0x000000000041b67e in main_loop_wait (nonblocking=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:1300
#7  0x00000000004314e7 in kvm_main_loop () at /build/home/tlv/akivity/qemu-kvm/qemu-kvm.c:1710
#8  0x000000000041c67f in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /build/home/tlv/akivity/qemu-kvm/vl.c:1340

Quoted 3 lines

#9 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:3069
This is on the incoming side so the test completes successfully, only leaving a core dump to fill my disks.

This appears to be

Quoted 16 lines

static inline void *host_from_stream_offset(QEMUFile *f,
ram_addr_t offset, int flags)

{
    static RAMBlock *block = NULL;
    char id[256];
    uint8_t len;

    if (flags & RAM_SAVE_FLAG_CONTINUE) {
        if (!block) {
            fprintf(stderr, "Ack, bad migration stream!\n");
            return NULL;
        }

        return block->host + offset;
    }

with block == NULL, if my gdb-fu got a static variable in an inlined function examined correctly.

I don't see any special reason for block to be NULL on a cancelled migration. Though perhaps the incoming stream was terminated without us noticing, and we're migrating from some random buffer and confusing the code?

Signature · 1 line

error compiling committee.c: too many arguments to function

Alex Williamson

2010-08-02 13:12 UTC #2 lore raw

Quoted 48 lines · Avi Kivity

Quoted 23 lines · Avi Kivity
I'm hitting some migration issues merging qemu.git into qemu-kvm.git:
1. Crash in mig_cancel test:
(gdb) bt
#0 0x0000003a91c83dbb in memcpy () from /lib64/libc.so.6 #1 0x000000000049c2ff in qemu_get_buffer (f=0x302d870, buf=<value optimized out>, size1=4096) at /usr/include/bits/string3.h:52 #2 0x0000000000409464 in ram_load (f=0x302d870, opaque=<value optimized out>, version_id=4) at /build/home/tlv/akivity/qemu-kvm/arch_init.c:407
#3 0x000000000049cb4c in qemu_loadvm_state (f=0x302d870) at savevm.c:1708
#4 0x0000000000494169 in process_incoming_migration (f=<value optimized out>) at migration.c:63
#5 0x0000000000494517 in tcp_accept_incoming_migration (opaque=<value optimized out>) at migration-tcp.c:163
#6 0x000000000041b67e in main_loop_wait (nonblocking=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:1300
#7 0x00000000004314e7 in kvm_main_loop () at /build/home/tlv/akivity/qemu-kvm/qemu-kvm.c:1710
#8 0x000000000041c67f in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
Quoted 6 lines
    at /build/home/tlv/akivity/qemu-kvm/vl.c:1340
#9 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:3069
This is on the incoming side so the test completes successfully, only leaving a core dump to fill my disks.
This appears to be
Quoted 14 lines
static inline void *host_from_stream_offset(QEMUFile *f,
ram_addr_t offset, int flags)
{
    static RAMBlock *block = NULL;
    char id[256];
    uint8_t len;

    if (flags & RAM_SAVE_FLAG_CONTINUE) {
        if (!block) {
            fprintf(stderr, "Ack, bad migration stream!\n");
            return NULL;
        }

Quoted 5 lines

Quoted 2 lines
        return block->host + offset;
    }
with block == NULL, if my gdb-fu got a static variable in an inlined function examined correctly.

If block == NULL, are you getting the fprintf?

Quoted 4 lines

I don't see any special reason for block to be NULL on a cancelled migration. Though perhaps the incoming stream was terminated without us noticing, and we're migrating from some random buffer and confusing the code?

Yeah, I don't understand that either, block == NULL should only be an initial state, once we've seen a block it shouldn't happen. Does this patch solve anything:

http://lists.nongnu.org/archive/html/qemu-devel/2010-07/msg01114.html

I could see this fixing it if the migration was re-attempted after the cancel.

Alex

Avi Kivity

author

2010-08-02 13:15 UTC #3 lore raw

Quoted 48 lines · Alex Williamson

Quoted 47 lines · Avi Kivity
Quoted 23 lines · Avi Kivity
I'm hitting some migration issues merging qemu.git into qemu-kvm.git:
1. Crash in mig_cancel test:
(gdb) bt
#0 0x0000003a91c83dbb in memcpy () from /lib64/libc.so.6 #1 0x000000000049c2ff in qemu_get_buffer (f=0x302d870, buf=<value optimized out>, size1=4096) at /usr/include/bits/string3.h:52 #2 0x0000000000409464 in ram_load (f=0x302d870, opaque=<value optimized out>, version_id=4) at /build/home/tlv/akivity/qemu-kvm/arch_init.c:407
#3 0x000000000049cb4c in qemu_loadvm_state (f=0x302d870) at savevm.c:1708
#4 0x0000000000494169 in process_incoming_migration (f=<value optimized out>) at migration.c:63
#5 0x0000000000494517 in tcp_accept_incoming_migration (opaque=<value optimized out>) at migration-tcp.c:163
#6 0x000000000041b67e in main_loop_wait (nonblocking=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:1300
#7 0x00000000004314e7 in kvm_main_loop () at /build/home/tlv/akivity/qemu-kvm/qemu-kvm.c:1710
#8 0x000000000041c67f in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
Quoted 6 lines
     at /build/home/tlv/akivity/qemu-kvm/vl.c:1340
#9 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /build/home/tlv/akivity/qemu-kvm/vl.c:3069
This is on the incoming side so the test completes successfully, only leaving a core dump to fill my disks.
This appears to be
Quoted 14 lines
static inline void *host_from_stream_offset(QEMUFile *f,
ram_addr_t offset, int flags)
{
     static RAMBlock *block = NULL;
     char id[256];
     uint8_t len;

     if (flags&  RAM_SAVE_FLAG_CONTINUE) {
         if (!block) {
             fprintf(stderr, "Ack, bad migration stream!\n");
             return NULL;
         }

Quoted 5 lines

Quoted 4 lines
Quoted 2 lines
         return block->host + offset;
     }
with block == NULL, if my gdb-fu got a static variable in an inlined function examined correctly.
If block == NULL, are you getting the fprintf?

Looked in qemu logs, but didn't see it. Maybe I missed it in the noise, or maybe autotest saw SIGCHLD and closed the logging channel.

Quoted 4 lines

Quoted 4 lines
I don't see any special reason for block to be NULL on a cancelled migration. Though perhaps the incoming stream was terminated without us noticing, and we're migrating from some random buffer and confusing the code?

Quoted 8 lines

Yeah, I don't understand that either, block == NULL should only be an initial state, once we've seen a block it shouldn't happen. Does this patch solve anything:
http://lists.nongnu.org/archive/html/qemu-devel/2010-07/msg01114.html
I could see this fixing it if the migration was re-attempted after the cancel.

I'll try it manually and see.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

author

2010-08-02 13:15 UTC #4 lore raw

In reply to Avi Kivity

Quoted 5 lines · Avi Kivity

2. Intermittent migration failures
Likely suspect is the IDE migration fixes. A merge of the commit just before these changes passes the test, I'm now testing the commit immediately after the changes.

Looks like this was a byproduct of the first problem - my disks filled up, or became extra slow during the core dump, and caused the failures. Seems to not to happen now.

Signature · 1 line

error compiling committee.c: too many arguments to function