qemu-kvm-1.1.0 crashing with kernel 3.5.0-rc6

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0

Cannot access memory at address 0xb694610c
(gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to get them) or test patches, etc

Thanks

Chris Clayton

Ping.

Have I committed a bug-reporting sin in the mail below or is everyone simply too busy to look at this kvm-related crash?

Quoted 45 lines

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0

Cannot access memory at address 0xb694610c
(gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to get them) or test patches, etc

Thanks

Quoted 1 line

Chris Clayton

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · Chris Clayton

Ping.

Have I committed a bug-reporting sin in the mail below or is everyone simply too busy to look at this kvm-related crash?

Since you have good and bad points can you bisect the problem?

Quoted 46 lines · Chris Clayton

Quoted 45 lines

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0

Cannot access memory at address 0xb694610c
(gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to get them) or test patches, etc

Thanks

Quoted 7 lines

Quoted 1 line

Chris Clayton

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

Quoted 6 lines · Gleb Natapov

Quoted 5 lines · Chris Clayton

Ping.

Have I committed a bug-reporting sin in the mail below or is everyone simply too busy to look at this kvm-related crash?

Quoted 2 lines

Since you have good and bad points can you bisect the problem?

Yes, I can bisect, but since the crash occurs with only only one combination of qemu-kvm (1.1.0) and kernel (3.5.0-rc6), I'm not sure which of those I should bisect. Any ideas on how I could narrow that down.

Thanks.

Quoted 57 lines

Quoted 46 lines · Chris Clayton

Quoted 45 lines

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0

Cannot access memory at address 0xb694610c
(gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to get them) or test patches, etc

Thanks

Quoted 7 lines

Quoted 1 line

Chris Clayton

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

Quoted 10 lines · Chris Clayton

Quoted 8 lines · Gleb Natapov

Quoted 5 lines · Chris Clayton

Ping.

Have I committed a bug-reporting sin in the mail below or is everyone simply too busy to look at this kvm-related crash?

Since you have good and bad points can you bisect the problem?

Quoted 5 lines

Yes, I can bisect, but since the crash occurs with only only one combination of qemu-kvm (1.1.0) and kernel (3.5.0-rc6), I'm not sure which of those I should bisect. Any ideas on how I could narrow that down.

Bisect qemu between qemu-kvm-1.0.1 & qemu-kvm-1.1.0.

Quoted 59 lines

Thanks.

Quoted 57 lines

Quoted 46 lines · Chris Clayton

Quoted 45 lines

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0

Cannot access memory at address 0xb694610c
(gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to get them) or test patches, etc

Thanks

Quoted 7 lines

Quoted 1 line

Chris Clayton

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

--
Gleb.

Quoted 11 lines · Gleb Natapov

Quoted 10 lines · Chris Clayton

Quoted 8 lines · Gleb Natapov

Quoted 5 lines · Chris Clayton

Ping.

Have I committed a bug-reporting sin in the mail below or is everyone simply too busy to look at this kvm-related crash?

Since you have good and bad points can you bisect the problem?

Quoted 7 lines

Quoted 5 lines

Yes, I can bisect, but since the crash occurs with only only one combination of qemu-kvm (1.1.0) and kernel (3.5.0-rc6), I'm not sure which of those I should bisect. Any ideas on how I could narrow that down.

Bisect qemu between qemu-kvm-1.0.1 & qemu-kvm-1.1.0.

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

Quoted 71 lines

Quoted 67 lines

Thanks.

Quoted 64 lines

Quoted 53 lines · Chris Clayton

Quoted 52 lines

Hi,

When I run WinXP SP3 through qemu-kvm-1.1.0 on linux kernel 3.5.0-rc6, I
get a segmentation fault within 3 or 4 minutes maximum. In dmesg I see:

qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition!
qemu-kvm: sending ioctl 5326 to a partition!
qemu-kvm: sending ioctl 801c0204 to a partition! qemu-kvm[860] general protection ip:b6abad77 sp:b52ff09c error:0 in libc-2.16.so[b697d000+1b4000]

The crash does not occur with qemu-kvm-1.0.1 on rc6. Nor does it occur
qemu-kvm-1.0.1 or qemu-kvm-1.1.0 on kernel 3.4.4. All three combinations
survive for 15 minutes or more

When I try to get a backtrace with gdb, the screen on which konsole and
qemu are running locks up until I kill qemu in another console. Consequently I can't get a full BT, but, although probably not very helpful, what I did get is:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6946b40 (LWP 506)] 0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7705d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e8d6e3 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e8c94c in g_hash_table_lookup () from

/usr/lib/libglib-2.0.so.0
Cannot access memory at address 0xb694610c (gdb)

Note that the gdb and dmesg outputs above are not from the same crash instance.

I'm not subscribed,so please cc me on any reply.

Happy to provide any additional diagnostics (but may need help on how to
get them) or test patches, etc

Thanks

Quoted 7 lines

Quoted 1 line

Chris Clayton

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

--
Gleb.

Quoted 7 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash. That would indicate that the problem is in the kernel. However, I pulled the latest and greatest from Linus yesterday evening and I now can't get the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly, so it's been fixed in the last few days.

Quoted 8 lines · Chris Clayton

Quoted 7 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

Quoted 6 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash. That would indicate that the problem is in the kernel. However, I pulled the latest and greatest from Linus yesterday evening and I now can't get the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly, so it's been fixed in the last few days.

There were no kvm changes post-rc7.

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Chris Clayton

Quoted 8 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

Quoted 9 lines

Quoted 6 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash. That would indicate that the problem is in the kernel. However, I pulled the latest and greatest from Linus yesterday evening and I now can't get the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly, so it's been fixed in the last few days.

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

I'm not inclined to bisect to find out which patch provided the fix, but this mail should at least close the mail thread down tidily.

Chris

Quoted 34 lines · Chris Clayton

Quoted 24 lines · Avi Kivity

Quoted 11 lines · Chris Clayton

Quoted 10 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't
used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on
linux-3.4.4. I'll report back in a day or two.

Quoted 10 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash.
That would indicate that the problem is in the kernel. However, I pulled
the latest and greatest from Linus yesterday evening and I now can't get
the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly,
so it's been fixed in the last few days.

Quoted 2 lines

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at

qom/object.c:94
#4 type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5 0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")

    at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
    typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60,
run=run@entry=0xb6239000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0 #12 0xb77bbbbe in clone () from /lib/libc.so.6

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built

Quoted 71 lines · Chris Clayton

Quoted 34 lines · Chris Clayton

Quoted 24 lines · Avi Kivity

Quoted 11 lines · Chris Clayton

Quoted 10 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't
used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on
linux-3.4.4. I'll report back in a day or two.

Quoted 10 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash.
That would indicate that the problem is in the kernel. However, I pulled
the latest and greatest from Linus yesterday evening and I now can't get
the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly,
so it's been fixed in the last few days.

Quoted 2 lines

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at

qom/object.c:94
#4 type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5 0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")

    at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
    typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60,
run=run@entry=0xb6239000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0 #12 0xb77bbbbe in clone () from /lib/libc.so.6

Quoted 1 line

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

Jan, why are we calling cpu_set_apic_tpr() with kvm_irqchip_in_kernel?

Quoted 4 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

Jan, why are we calling cpu_set_apic_tpr() with kvm_irqchip_in_kernel?

To sync the userspace state with what the kernel maintains. Will end up in kvm_apic_set_tpr which does precisely this. We always did, just the QOM modeling is new.

Quoted 3 lines · Jan Kiszka

Quoted 3 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

Quoted 5 lines

Quoted 1 line

Jan, why are we calling cpu_set_apic_tpr() with kvm_irqchip_in_kernel?

To sync the userspace state with what the kernel maintains. Will end up in kvm_apic_set_tpr which does precisely this. We always did, just the QOM modeling is new.

We should move it to the general register synchronization code, there is no reason to do this every exit (though the cost is likely minimal).

Quoted 11 lines · Avi Kivity

Quoted 8 lines · Jan Kiszka

Quoted 4 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

Jan, why are we calling cpu_set_apic_tpr() with kvm_irqchip_in_kernel?

To sync the userspace state with what the kernel maintains. Will end up in kvm_apic_set_tpr which does precisely this. We always did, just the QOM modeling is new.

Quoted 2 lines

We should move it to the general register synchronization code, there is no reason to do this every exit (though the cost is likely minimal).

The cost is, well, was close to nothing. But I'm not sure about that QOM type casting magic (and also it's locking requirements, long-term). However, if that is a problem, it's likely a much bigger one anyway.

Quoted 71 lines · Chris Clayton

Quoted 34 lines · Chris Clayton

Quoted 24 lines · Avi Kivity

Quoted 11 lines · Chris Clayton

Quoted 10 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't
used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on
linux-3.4.4. I'll report back in a day or two.

Quoted 10 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash.
That would indicate that the problem is in the kernel. However, I pulled
the latest and greatest from Linus yesterday evening and I now can't get
the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly,
so it's been fixed in the last few days.

Quoted 2 lines

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at

qom/object.c:94
#4 type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5 0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")

     at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
     typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60,
run=run@entry=0xb6239000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0 #12 0xb77bbbbe in clone () from /lib/libc.so.6

Quoted 1 line

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

Jan, why are we calling cpu_set_apic_tpr() with kvm_irqchip_in_kernel?

Quoted 2 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

I've just done 10 runs of WinXP SP3 and 5 of them crashed. Three crashed early as XP was starting up - well before the desktop would have appeared. The other two crashed as XP was closing down, having been running for a few minutes (but not doing much).

The error messages seen through dmesg are:

qemu-kvm[12778] general protection ip:b6c43d77 sp:b5e800fc error:0 in libc-2.16.so[b6b06000+1b4000]
qemu-kvm[12813] general protection ip:b6bf6d77 sp:b54ff0fc error:0 in libc-2.16.so[b6ab9000+1b4000]
qemu-kvm[12986] general protection ip:b6cd3d77 sp:b55ff0fc error:0 in libc-2.16.so[b6b96000+1b4000]
qemu-kvm[13045] general protection ip:b6c91d77 sp:b54ff0fc error:0 in libc-2.16.so[b6b54000+1b4000]
qemu-kvm[13225] general protection ip:b6c5bd77 sp:b54ff0fc error:0 in libc-2.16.so[b6b1e000+1b4000]

The other 5 were OK, although I only did a bit of web browsing for few minutes with IE.

Quoted 21 lines · Chris Clayton

Quoted 2 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

I've just done 10 runs of WinXP SP3 and 5 of them crashed. Three crashed early as XP was starting up - well before the desktop would have appeared. The other two crashed as XP was closing down, having been running for a few minutes (but not doing much).

The error messages seen through dmesg are:

qemu-kvm[12778] general protection ip:b6c43d77 sp:b5e800fc error:0 in libc-2.16.so[b6b06000+1b4000]
qemu-kvm[12813] general protection ip:b6bf6d77 sp:b54ff0fc error:0 in libc-2.16.so[b6ab9000+1b4000]
qemu-kvm[12986] general protection ip:b6cd3d77 sp:b55ff0fc error:0 in libc-2.16.so[b6b96000+1b4000]
qemu-kvm[13045] general protection ip:b6c91d77 sp:b54ff0fc error:0 in libc-2.16.so[b6b54000+1b4000]
qemu-kvm[13225] general protection ip:b6c5bd77 sp:b54ff0fc error:0 in libc-2.16.so[b6b1e000+1b4000]

Quoted 2 lines

The other 5 were OK, although I only did a bit of web browsing for few minutes with IE.

Failures always in the same place (I'm guess the variations are due to PIE -- please configure with --disable-pie for future tests).

Please generate a core and look around, esp. in frame 3 (type_table_lookup). Also try to dissect type_table (you may need to install the glib debug symbols for this).

Quoted 29 lines · Avi Kivity

Quoted 21 lines · Chris Clayton

Quoted 2 lines

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

I've just done 10 runs of WinXP SP3 and 5 of them crashed. Three crashed early as XP was starting up - well before the desktop would have appeared. The other two crashed as XP was closing down, having been running for a few minutes (but not doing much).

The error messages seen through dmesg are:

qemu-kvm[12778] general protection ip:b6c43d77 sp:b5e800fc error:0 in libc-2.16.so[b6b06000+1b4000]
qemu-kvm[12813] general protection ip:b6bf6d77 sp:b54ff0fc error:0 in libc-2.16.so[b6ab9000+1b4000]
qemu-kvm[12986] general protection ip:b6cd3d77 sp:b55ff0fc error:0 in libc-2.16.so[b6b96000+1b4000]
qemu-kvm[13045] general protection ip:b6c91d77 sp:b54ff0fc error:0 in libc-2.16.so[b6b54000+1b4000]
qemu-kvm[13225] general protection ip:b6c5bd77 sp:b54ff0fc error:0 in libc-2.16.so[b6b1e000+1b4000]

Quoted 2 lines

The other 5 were OK, although I only did a bit of web browsing for few minutes with IE.

Failures always in the same place (I'm guess the variations are due to PIE -- please configure with --disable-pie for future tests).

Quoted 6 lines

Please generate a core and look around, esp. in frame 3 (type_table_lookup). Also try to dissect type_table (you may need to install the glib debug symbols for this).

Could you include the output of "info registers" at the point where it crashed?

Quoted 3 lines · Eric Northup

Could you include the output of "info registers" at the point where it crashed?

Here you go:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6a78b40 (LWP 13249)] __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) bt

#0  __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1  0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319b82) at ghash.c:1704
#2  0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800,
key=0x8319b82, hash_return=0xb6a78178)
    at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319b82) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319b82 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319b82 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a313e0, typename=typename@entry=0x8319b82 "apic-common")

    at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a313e0,
    typename=typename@entry=0x8319b82 "apic-common") at qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a313e0, val=13 '\r')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb86c in kvm_arch_post_run (env=env@entry=0x8a29370,
run=run@entry=0xb6274000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269
#11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752
#12 0xb7a3ed9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e45ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132 (gdb) info registers

eax            0x8319b82        137468802
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb6a780ec       0xb6a780ec
ebp            0xb6a78118       0xb6a78118
esi            0x8a313e0        144905184
edi            0xc513   50451
eip            0xb7824f77       0xb7824f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51

Quoted 40 lines · Chris Clayton

Quoted 3 lines · Eric Northup

Could you include the output of "info registers" at the point where it crashed?

Here you go:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6a78b40 (LWP 13249)] __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) bt

#0  __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1  0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319b82) at ghash.c:1704
#2  0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800,
key=0x8319b82, hash_return=0xb6a78178)
     at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319b82) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319b82 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319b82 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a313e0, typename=typename@entry=0x8319b82 "apic-common")

     at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a313e0,
     typename=typename@entry=0x8319b82 "apic-common") at qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a313e0, val=13 '\r')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb86c in kvm_arch_post_run (env=env@entry=0x8a29370,
run=run@entry=0xb6274000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269
#11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752
#12 0xb7a3ed9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e45ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132 (gdb) info registers

Quoted 17 lines

eax            0x8319b82        137468802
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb6a780ec       0xb6a780ec
ebp            0xb6a78118       0xb6a78118
esi            0x8a313e0        144905184
edi            0xc513   50451
eip            0xb7824f77       0xb7824f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51

ds shouldn't be zero for a 32-bit process.

But that should have crashed *much* earlier, ds is accessed all the time.

Please add the following snippet to the beginning of kvm_arch_post_run():

{
     unsigned short ds;
     asm("mov %%ds, %0" : "=rm"(ds));
     assert(ds != 0);
}

if the assert triggers, then kvm corrupted the segment registers. If not, corruption happens somewhere above.

Quoted 69 lines · Avi Kivity

Quoted 44 lines · Chris Clayton

Quoted 3 lines · Eric Northup

Could you include the output of "info registers" at the point where it crashed?

Here you go:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6a78b40 (LWP 13249)] __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) bt
#0 __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1 0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319b82) at ghash.c:1704
#2 0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800, key=0x8319b82, hash_return=0xb6a78178)

     at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319b82) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319b82 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319b82 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a313e0, typename=typename@entry=0x8319b82 "apic-common")

     at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a313e0,
     typename=typename@entry=0x8319b82 "apic-common") at
qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a313e0, val=13 '\r')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb86c in kvm_arch_post_run (env=env@entry=0x8a29370,
run=run@entry=0xb6274000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #12 0xb7a3ed9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e45ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132
(gdb) info registers

Quoted 17 lines

eax            0x8319b82        137468802
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb6a780ec       0xb6a780ec
ebp            0xb6a78118       0xb6a78118
esi            0x8a313e0        144905184
edi            0xc513   50451
eip            0xb7824f77       0xb7824f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51

ds shouldn't be zero for a 32-bit process.

But that should have crashed *much* earlier, ds is accessed all the time.

Please add the following snippet to the beginning of kvm_arch_post_run():

Quoted 9 lines

{
     unsigned short ds;
     asm("mov %%ds, %0" : "=rm"(ds));
     assert(ds != 0);
}

if the assert triggers, then kvm corrupted the segment registers. If not, corruption happens somewhere above.

Thanks, Avi.

The assert didn't trigger - I got:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 2134)]
__strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) info registers

eax            0x8319ba2        137468834
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb60ff0ec       0xb60ff0ec
ebp            0xb60ff118       0xb60ff118
esi            0x8a44818        144984088
edi            0xc513   50451
eip            0xb7820f77       0xb7820f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51
(gdb) list
212     #endif
213             mov     %dx, %cx
214             and     $0xfff, %cx
215             cmp     $0xff0, %cx
216             ja      L(first4bytes)
217             movdqu  (%edx), %xmm2
218             mov     %eax, %ecx
219             and     $0xfff, %ecx
220             cmp     $0xff0, %ecx
221             ja      L(first4bytes)
(gdb) bt
#0  __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1  0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319ba2) at ghash.c:1704
#2  0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800,
key=0x8319ba2, hash_return=0xb60ff178)
    at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319ba2) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319ba2 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319ba2 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a44818, typename=typename@entry=0x8319ba2 "apic-common")

    at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a44818,
    typename=typename@entry=0x8319ba2 "apic-common") at qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a44818, val=7 '\a')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb874 in kvm_arch_post_run (env=env@entry=0x8a3ca60,
run=run@entry=0xb626d000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1702

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a3ca60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269
#11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a3ca60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752
#12 0xb7a3ad9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e05ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132

I think you are saying that the problem isn't in kvm, so where would you recommend I continue investigations. I'm not seeing a crash with any other applications.

Quoted 4 lines

I think you are saying that the problem isn't in kvm, so where would you recommend I continue investigations. I'm not seeing a crash with any other applications.

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

Quoted 5 lines · Avi Kivity

Quoted 4 lines

I think you are saying that the problem isn't in kvm, so where would you recommend I continue investigations. I'm not seeing a crash with any other applications.

Quoted 6 lines

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

You're running a 32-bit kernel, yes? Please confirm.

Quoted 144 lines · Chris Clayton

Quoted 69 lines · Avi Kivity

Quoted 44 lines · Chris Clayton

Quoted 3 lines · Eric Northup

Could you include the output of "info registers" at the point where it crashed?

Here you go:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6a78b40 (LWP 13249)] __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) bt
#0 __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1 0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319b82) at ghash.c:1704
#2 0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800, key=0x8319b82, hash_return=0xb6a78178)

     at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319b82) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319b82 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319b82 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a313e0, typename=typename@entry=0x8319b82 "apic-common")

     at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a313e0,
     typename=typename@entry=0x8319b82 "apic-common") at
qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a313e0, val=13 '\r')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb86c in kvm_arch_post_run (env=env@entry=0x8a29370,
run=run@entry=0xb6274000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a29370) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #12 0xb7a3ed9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e45ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132
(gdb) info registers

Quoted 17 lines

eax            0x8319b82        137468802
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb6a780ec       0xb6a780ec
ebp            0xb6a78118       0xb6a78118
esi            0x8a313e0        144905184
edi            0xc513   50451
eip            0xb7824f77       0xb7824f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51

ds shouldn't be zero for a 32-bit process.

But that should have crashed *much* earlier, ds is accessed all the time.

Please add the following snippet to the beginning of kvm_arch_post_run():

Quoted 9 lines

{
     unsigned short ds;
     asm("mov %%ds, %0" : "=rm"(ds));
     assert(ds != 0);
}

if the assert triggers, then kvm corrupted the segment registers. If not, corruption happens somewhere above.

Thanks, Avi.

The assert didn't trigger - I got:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 2134)]
__strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
217 movdqu (%edx), %xmm2
(gdb) info registers

eax            0x8319ba2        137468834
ecx            0xd58    3416
edx            0x8a0cd58        144756056
ebx            0xb7f7f2c4       -1208487228
esp            0xb60ff0ec       0xb60ff0ec
ebp            0xb60ff118       0xb60ff118
esi            0x8a44818        144984088
edi            0xc513   50451
eip            0xb7820f77       0xb7820f77 <__strcmp_sse4_2+23>
eflags         0x10283  [ CF SF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x33     51
(gdb) list
212     #endif
213             mov     %dx, %cx
214             and     $0xfff, %cx
215             cmp     $0xff0, %cx
216             ja      L(first4bytes)
217             movdqu  (%edx), %xmm2
218             mov     %eax, %ecx
219             and     $0xfff, %ecx
220             cmp     $0xff0, %ecx
221             ja      L(first4bytes)
(gdb) bt
#0  __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:217
#1  0xb7e82cb4 in g_str_equal (v1=0x8a0cd58, v2=0x8319ba2) at ghash.c:1704
#2  0xb7e8137a in g_hash_table_lookup_node (hash_table=0x89fe800,
key=0x8319ba2, hash_return=0xb60ff178)
    at ghash.c:422

#3 0xb7e821e5 in g_hash_table_lookup (hash_table=0x89fe800, key=key@entry=0x8319ba2) at ghash.c:1074
#4 0x0815c9cb in type_table_lookup (name=0x8319ba2 "apic-common") at qom/object.c:94
#5 type_get_by_name (name=name@entry=0x8319ba2 "apic-common") at qom/object.c:149
#6 0x0815cf93 in object_dynamic_cast (obj=obj@entry=0x8a44818, typename=typename@entry=0x8319ba2 "apic-common")

    at qom/object.c:416
#7  0x0815cf2d in object_dynamic_cast_assert (obj=obj@entry=0x8a44818,
    typename=typename@entry=0x8319ba2 "apic-common") at qom/object.c:478
#8  0x08192c1b in cpu_set_apic_tpr (d=0x8a44818, val=7 '\a')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#9  0x081cb874 in kvm_arch_post_run (env=env@entry=0x8a3ca60,
run=run@entry=0xb626d000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1702

#10 0x081c686f in kvm_cpu_exec (env=env@entry=0x8a3ca60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269
#11 0x08198c72 in qemu_kvm_cpu_thread_fn (arg=0x8a3ca60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752
#12 0xb7a3ad9e in start_thread () from /lib/libpthread.so.0 #13 0xb77e05ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132

Quoted 3 lines

I think you are saying that the problem isn't in kvm, so where would you recommend I continue investigations. I'm not seeing a crash with any other applications.

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

Quoted 8 lines

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

from entry_32.S:

.macro RESTORE_REGS pop=0
RESTORE_INT_REGS
1: popl_cfi %ds

	/*CFI_RESTORE ds;*/
2:	popl_cfi %es
	/*CFI_RESTORE es;*/
3:	popl_cfi %fs
	/*CFI_RESTORE fs;*/
	POP_GS \pop

.pushsection .fixup, "ax"
4: movl $0, (%esp)
jmp 1b
5: movl $0, (%esp)
jmp 2b
6: movl $0, (%esp)
jmp 3b
.popsection

this piece of code tries to restore %ds, and if it fails, zeros it, which is consistent with the core dump.

This could happen if kvm is failing to restore GDT correctly.

Quoted 29 lines · Avi Kivity

Quoted 8 lines

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

from entry_32.S:

.macro RESTORE_REGS pop=0
RESTORE_INT_REGS
1: popl_cfi %ds

	/*CFI_RESTORE ds;*/
2:	popl_cfi %es
	/*CFI_RESTORE es;*/
3:	popl_cfi %fs
	/*CFI_RESTORE fs;*/
	POP_GS \pop

.pushsection .fixup, "ax"
4: movl $0, (%esp)
jmp 1b
5: movl $0, (%esp)
jmp 2b
6: movl $0, (%esp)
jmp 3b
.popsection

Quoted 5 lines

this piece of code tries to restore %ds, and if it fails, zeros it, which is consistent with the core dump.

This could happen if kvm is failing to restore GDT correctly.

Possible culprit: b2da15ac26a0c00.

Quoted 33 lines · Avi Kivity

Quoted 32 lines · Avi Kivity

Quoted 8 lines

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

from entry_32.S:

.macro RESTORE_REGS pop=0
RESTORE_INT_REGS
1: popl_cfi %ds

	/*CFI_RESTORE ds;*/
2:	popl_cfi %es
	/*CFI_RESTORE es;*/
3:	popl_cfi %fs
	/*CFI_RESTORE fs;*/
	POP_GS \pop

.pushsection .fixup, "ax"
4: movl $0, (%esp)
jmp 1b
5: movl $0, (%esp)
jmp 2b
6: movl $0, (%esp)
jmp 3b
.popsection

this piece of code tries to restore %ds, and if it fails, zeros it, which is consistent with the core dump.

Quoted 6 lines

Quoted 2 lines

This could happen if kvm is failing to restore GDT correctly.

Possible culprit: b2da15ac26a0c00.

That commit isn't in qermu-kvm-1.1.1.

I'm testing a build with g_str_equal implemented in kvm.c and so far I haven't had a crash in 6 invocations. That hasn't been possible with vanilla qemu-kvm-1.1.{0,1}, but I'll do a few more, just to be sure.

Thanks for your help, Avi.

Quoted 37 lines · Chris Clayton

Quoted 36 lines · Avi Kivity

Quoted 34 lines · Avi Kivity

Quoted 8 lines

What might have happened is that the movdqu instruction faulted (as it's an fpu instruction), and on the way back from the fault, ds and es didn't get restored correctly.

You can test this by writing a trivial version of g_str_equal() somewhere in the qemu source code and rebuilding it.

from entry_32.S:

.macro RESTORE_REGS pop=0
RESTORE_INT_REGS
1: popl_cfi %ds

	/*CFI_RESTORE ds;*/
2:	popl_cfi %es
	/*CFI_RESTORE es;*/
3:	popl_cfi %fs
	/*CFI_RESTORE fs;*/
	POP_GS \pop

.pushsection .fixup, "ax"
4: movl $0, (%esp)
jmp 1b
5: movl $0, (%esp)
jmp 2b
6: movl $0, (%esp)
jmp 3b
.popsection

this piece of code tries to restore %ds, and if it fails, zeros it, which is consistent with the core dump.

This could happen if kvm is failing to restore GDT correctly.

Quoted 5 lines

Quoted 3 lines

Possible culprit: b2da15ac26a0c00.

That commit isn't in qermu-kvm-1.1.1.

It is in kernel.

Quoted 5 lines

I'm testing a build with g_str_equal implemented in kvm.c and so far I haven't had a crash in 6 invocations. That hasn't been possible with vanilla qemu-kvm-1.1.{0,1}, but I'll do a few more, just to be sure.

Quoted 1 line

Thanks for your help, Avi.

--
Gleb.

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

Possible culprit: b2da15ac26a0c00.

That commit isn't in qermu-kvm-1.1.1.

Quoted 2 lines

It is in kernel.

Sorry, so it is.

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines · Chris Clayton

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Possible culprit: b2da15ac26a0c00.

That commit isn't in qermu-kvm-1.1.1.

It is in kernel.

Sorry, so it is.

Quoted 3 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Quoted 15 lines · Chris Clayton

Quoted 13 lines · Chris Clayton

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Possible culprit: b2da15ac26a0c00.

That commit isn't in qermu-kvm-1.1.1.

It is in kernel.

Sorry, so it is.

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

[chris:~/kernel/linux-3.5.0]$ grep PREEMPT .config
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

Quoted 15 lines · Chris Clayton

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

Quoted 8 lines

[chris:~/kernel/linux-3.5.0]$ grep PREEMPT .config
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

Here's what I think that is happening

vcpu_load
...
vmx_save_host_state
vmx_vcpu_run
(ds.cpl, es.cpl cleared by hardware)

interrupt
push ds, es # pushes bad ds, es
schedule
vmx_vcpu_put
vmx_load_host_state
reload ds, es
pop ds, es # of other thread's stack
iret

  # other thread runs
  interrupt
    schedule  # back in vcpu thread
    interrupt return: pop ds, es  # <-- problem
    iret

   ...
   vcpu_put

   # bad ds, es, but !vmx->host_state.loaded

Marcelo, did I miss something here?

Unfortunately, my reproducer has ceased to reproduce. But the fix is easy if the analysis above is right.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 47 lines · Avi Kivity

Quoted 15 lines · Chris Clayton

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

Quoted 8 lines

[chris:~/kernel/linux-3.5.0]$ grep PREEMPT .config
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

Here's what I think that is happening

vcpu_load
...
vmx_save_host_state
vmx_vcpu_run
(ds.cpl, es.cpl cleared by hardware)

interrupt
push ds, es # pushes bad ds, es
schedule
vmx_vcpu_put
vmx_load_host_state
reload ds, es
pop ds, es # of other thread's stack
iret

  # other thread runs
  interrupt
    schedule  # back in vcpu thread
    interrupt return: pop ds, es  # <-- problem
    iret

Quoted 6 lines

   ...
   vcpu_put

   # bad ds, es, but !vmx->host_state.loaded

Marcelo, did I miss something here?

Don't think so.

Quoted 51 lines · Marcelo Tosatti

Quoted 50 lines · Avi Kivity

Quoted 15 lines · Chris Clayton

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

Quoted 8 lines

[chris:~/kernel/linux-3.5.0]$ grep PREEMPT .config
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

Here's what I think that is happening

vcpu_load
...
vmx_save_host_state
vmx_vcpu_run
(ds.cpl, es.cpl cleared by hardware)

interrupt
push ds, es # pushes bad ds, es
schedule
vmx_vcpu_put
vmx_load_host_state
reload ds, es
pop ds, es # of other thread's stack
iret

  # other thread runs
  interrupt
    schedule  # back in vcpu thread
    interrupt return: pop ds, es  # <-- problem
    iret

   ...
   vcpu_put

Quoted 5 lines

Quoted 3 lines

   # bad ds, es, but !vmx->host_state.loaded

Marcelo, did I miss something here?

Don't think so.

So the same problem should happen with %fs and %gs, no?

x86_64 is safe, since it entry_64.S never saves/restores segment registers.

Quoted 2 lines

So the same problem should happen with %fs and %gs, no?

AFAICS:

depends on CONFIG_X86_32_LAZY_GS for GS, unconditional for FS.

Quoted 2 lines

x86_64 is safe, since it entry_64.S never saves/restores segment registers.

Is the comment

/*
* The sysexit path does not restore ds/es, so we must set them
* to
* a reasonable value ourselves.
*/

Correct?

syscall_exit -> syscall_exit_work -> resume_userspace -> restore_all -> RESTORE_REGS

Quoted 15 lines · Chris Clayton

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

With kernel 3.5.0 with b2da15ac26a0c00 reverted, I have just had 15 clean invocations of vanilla qemu-kvm-1.1.1. So that commit would seem to be the problem.

Quoted 10 lines

Quoted 3 lines

Just to be sure, I've run some more tests today. No crashes occurred in 20 runs of vanilla qemu-kvm-1.1.1 on kernel 3.5.0 with b2da15ac26a0c00 reverted.

Ok. I'm trying to reproduce it here on a nested-virt setup, since the code looks correct.

What's your preemption settings?

Quoted 8 lines

[chris:~/kernel/linux-3.5.0]$ grep PREEMPT .config
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

Here's what I think that is happening

vcpu_load
...
vmx_save_host_state
vmx_vcpu_run
(ds.cpl, es.cpl cleared by hardware)

interrupt
push ds, es # pushes bad ds, es
schedule
vmx_vcpu_put
vmx_load_host_state
reload ds, es
pop ds, es # of other thread's stack
iret

  # other thread runs
  interrupt
    schedule  # back in vcpu thread
    interrupt return: pop ds, es  # <-- problem

iret

   ...
   vcpu_put

   # bad ds, es, but !vmx->host_state.loaded

Quoted 72 lines · Avi Kivity

Quoted 71 lines · Chris Clayton

Quoted 34 lines · Chris Clayton

Quoted 24 lines · Avi Kivity

Quoted 11 lines · Chris Clayton

Quoted 10 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't
used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on
linux-3.4.4. I'll report back in a day or two.

Quoted 10 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash.
That would indicate that the problem is in the kernel. However, I pulled
the latest and greatest from Linus yesterday evening and I now can't get
the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly,
so it's been fixed in the last few days.

Quoted 2 lines

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at

qom/object.c:94
#4 type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5 0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")

     at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
     typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60,
run=run@entry=0xb6239000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0 #12 0xb77bbbbe in clone () from /lib/libc.so.6

Quoted 4 lines

Quoted 1 line

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built

It looks like general memory corruption. Is this repeatable? What's the guest uptime when it happens (i.e. is it immediate?)

I've just done 10 runs of WinXP SP3 and 5 of them crashed. Three crashed

early as XP was starting up - well before the desktop would have appeared. The other two crashed as XP was closing down, having been running for a few minutes (but not doing much).

The error messages seen through dmesg are:

qemu-kvm[12778] general protection ip:b6c43d77 sp:b5e800fc error:0 in libc-2.16.so[b6b06000+1b4000]
qemu-kvm[12813] general protection ip:b6bf6d77 sp:b54ff0fc error:0 in libc-2.16.so[b6ab9000+1b4000]
qemu-kvm[12986] general protection ip:b6cd3d77 sp:b55ff0fc error:0 in libc-2.16.so[b6b96000+1b4000]
qemu-kvm[13045] general protection ip:b6c91d77 sp:b54ff0fc error:0 in libc-2.16.so[b6b54000+1b4000]
qemu-kvm[13225] general protection ip:b6c5bd77 sp:b54ff0fc error:0 in libc-2.16.so[b6b1e000+1b4000]

Quoted 28 lines · Chris Clayton

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Chris Clayton

Quoted 8 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

Quoted 9 lines

Quoted 6 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash. That would indicate that the problem is in the kernel. However, I pulled the latest and greatest from Linus yesterday evening and I now can't get the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly, so it's been fixed in the last few days.

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at qom/object.c:94
#4  type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5  0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")
    at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
    typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60, run=run@entry=0xb6239000)
    at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0
#12 0xb77bbbbe in clone () from /lib/libc.so.6

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built against 3.4.4 kernel headers. The glibc, the kernel headers and the kernel are vanilla and the only change to the qemu-kvm sources is:

configure 2012-07-15 22:39:09.000000000 +0100 renamed from configure~ 2012-07-15 22:38:39.000000000 +0100 +1 -1

@@ -2783,7 +2783,7 @@ int main(int argc, char **argv)
 }
 EOF
   if ! compile_prog "" "" ; then
-    CFLAGS+="-march=i486"
+    CFLAGS+="-march=i686"
   fi
 fi
 

Please let me know of anything I can do to help track this down.

Thanks

Chris

Quoted 4 lines

I'm not inclined to bisect to find out which patch provided the fix, but this mail should at least close the mail thread down tidily.

Chris

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Chris,

Could you please try this patch?
http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=commit;h=ccebf448daf7964ee2aff7947c0bbe4c7962d059

Quoted 89 lines · Chris Clayton

Quoted 28 lines · Avi Kivity

Quoted 11 lines · Chris Clayton

Quoted 10 lines

Quoted 9 lines

Change of diagnostics, unfortunately. qemu-kvm-1.0.1 can, in fact, crash
on 3.5.0-rc6 (and rc7). I didn't get it earlier because it takes many times more invocations before the crash occurs with 1.0.1 and I haven't used qemu-kvm much in the past few weeks.

I'm now checking whether I can get crashes (with 1.0.1 and/or 1.1.0) on linux-3.4.4. I'll report back in a day or two.

Quoted 9 lines

Quoted 6 lines

I've started up qemu-kvm on kernel 3.4.4 many times and not see a crash. That would indicate that the problem is in the kernel. However, I pulled the latest and greatest from Linus yesterday evening and I now can't get the crash there either, so whatever it was seems to have been fixed. If I checkout and build 3.5.0-rc[1..7], I can get the crash pretty quickly, so it's been fixed in the last few days.

There were no kvm changes post-rc7.

Yes, I'm aware of that, Avi. This thread started because I was getting a crash in qemu-kvm, which I thought was only in v1.1.0. Later it turned out the the problem was also present in v1.0.1, but much harder to hit. However, it only ever happened with 3.5.0 kernels. 3.4.4, with either version of qemu-kvm, was stable. So then it seemed that the problem was in the kernel, (but not necessarily in the kvm code).

Quoted 5 lines

Something that's changed since rc7 has either fixed the problem or made it much harder to hit. With rc7 and earlier I can recreate the crash quite easily with qemu-kvm-1.1.0 and with enough runs of 1.0.1. With rc7+, I haven't been able to get a crash at all.

Well, I'm getting the crash again, but this time I've managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb60ffb40 (LWP 9405)] 0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6 (gdb) bt

#0  0xb7803d77 in __strcmp_sse4_2 () from /lib/libc.so.6
#1  0xb7e65333 in g_str_equal () from /usr/lib/libglib-2.0.so.0
#2  0xb7e6458d in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#3  0x8014e2cf in type_table_lookup (name=0x802b0c50 "apic-common") at qom/object.c:94
#4  type_get_by_name (name=name@entry=0x802b0c50 "apic-common") at qom/object.c:149
#5  0x8014e933 in object_dynamic_cast (obj=obj@entry=0x80a5d818, typename=typename@entry=0x802b0c50 "apic-common")
     at qom/object.c:416
#6  0x8014e8b9 in object_dynamic_cast_assert (obj=obj@entry=0x80a5d818,
     typename=typename@entry=0x802b0c50 "apic-common") at qom/object.c:478
#7  0x80193462 in cpu_set_apic_tpr (d=0x80a5d818, val=8 '\b')
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/hw/apic_common.c:60
#8  0x801d0560 in kvm_arch_post_run (env=env@entry=0x80a55a60, run=run@entry=0xb6239000)
     at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/target-i386/kvm.c:1695

#9 0x801cb05f in kvm_cpu_exec (env=env@entry=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/kvm-all.c:1269 #10 0x80199d1e in qemu_kvm_cpu_thread_fn (arg=0x80a55a60) at /home/chris/rpm/BUILD/qemu-kvm-1.1.1/cpus.c:752 #11 0xb7a1fd9e in start_thread () from /lib/libpthread.so.0
#12 0xb77bbbbe in clone () from /lib/libc.so.6

This is with kernel 3.5.0 and qemu-kvm-1.1.1. glibc is 2.16.0 built against 3.4.4 kernel headers. The glibc, the kernel headers and the kernel are vanilla and the only change to the qemu-kvm sources is:

configure 2012-07-15 22:39:09.000000000 +0100 renamed from configure~ 2012-07-15 22:38:39.000000000 +0100 +1 -1

@@ -2783,7 +2783,7 @@ int main(int argc, char **argv)
  }
  EOF
    if ! compile_prog "" "" ; then
-    CFLAGS+="-march=i486"
+    CFLAGS+="-march=i686"
    fi
  fi
 

Please let me know of anything I can do to help track this down.

Thanks

Chris

Quoted 4 lines

I'm not inclined to bisect to find out which patch provided the fix, but this mail should at least close the mail thread down tidily.

Chris

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html