Skip to contentSkip to conversation

kvm crash on 5.7-rc1 and later

DISCUSSION Peter Zijlstra 3 participants 4 messages Jul 11, 2020 – Jul 12, 2020 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 30 lines · Woody Suwalski

I am observing a 100% reproducible kvm crash on kernels starting with 5.7-rc1, always with the same opcode 0000.
It happens during wake up from the host suspended state. Worked OK on 5.6 and older.
The host is based on Debian testing, Thinkpad T440, i5 cpu.

[   61.576664] kernel BUG at arch/x86/kvm/x86.c:387!
[   61.576672] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   61.576678] CPU: 0 PID: 3851 Comm: qemu-system-x86 Not tainted 5.7-pingu #0
[   61.576680] Hardware name: LENOVO 20B6005JUS/20B6005JUS, BIOS GJETA4WW (2.54 ) 03/27/2020
[   61.576700] RIP: 0010:kvm_spurious_fault+0xa/0x10 [kvm]

Crash results in a dead kvm and occasionally a very unstable system.

Bisecting the problem between v5.6 and v5.7-rc1 points to

commit 6650cdd9a8ccf00555dbbe743d58541ad8feb6a7
Author: Peter Zijlstra (Intel) <[email protected]> Date:   Sun Jan 26 12:05:35 2020 -0800

    x86/split_lock: Enable split lock detection by kernel

Reversing that patch seems to actually "cure" the issue.

The problem is present in all kernels past 5.7-rc1, however the patch is not reversing directly in later source trees, so can not retest the logic on recent kernels.

Quoted 5 lines

Peter, would you have idea how to debug that (or even better - would you happen to know the fix)?

I have attached dmesg logs from a "good" 5.6.9 kernel, and then "bad" 5.7.0 and 5.8-rc3

I have no clue about kvm. Nor do I actually have hardware with SLD on. I've Cc'ed a bunch of folks who might have more ideas.

#2 lore raw
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Woody Suwalski

I am observing a 100% reproducible kvm crash on kernels starting with 5.7-rc1, always with the same opcode 0000.
It happens during wake up from the host suspended state. Worked OK on 5.6 and older.
The host is based on Debian testing, Thinkpad T440, i5 cpu.

[   61.576664] kernel BUG at arch/x86/kvm/x86.c:387!
[   61.576672] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   61.576678] CPU: 0 PID: 3851 Comm: qemu-system-x86 Not tainted 5.7-pingu #0
[   61.576680] Hardware name: LENOVO 20B6005JUS/20B6005JUS, BIOS GJETA4WW (2.54 ) 03/27/2020
[   61.576700] RIP: 0010:kvm_spurious_fault+0xa/0x10 [kvm]

Crash results in a dead kvm and occasionally a very unstable system.

Bisecting the problem between v5.6 and v5.7-rc1 points to

commit 6650cdd9a8ccf00555dbbe743d58541ad8feb6a7
Author: Peter Zijlstra (Intel) <[email protected]> Date:   Sun Jan 26 12:05:35 2020 -0800

    x86/split_lock: Enable split lock detection by kernel

Reversing that patch seems to actually "cure" the issue.

The problem is present in all kernels past 5.7-rc1, however the patch is not reversing directly in later source trees, so can not retest the logic on recent kernels.

Quoted 9 lines
Quoted 5 lines

Peter, would you have idea how to debug that (or even better - would you happen to know the fix)?

I have attached dmesg logs from a "good" 5.6.9 kernel, and then "bad" 5.7.0 and 5.8-rc3

I have no clue about kvm. Nor do I actually have hardware with SLD on. I've Cc'ed a bunch of folks who might have more ideas.

I think this bug is the same as the one found by Sean, and is already fixed in 5.8-rc4.

https://lore.kernel.org/kvm/[email protected]/

#3 lore raw
Quoted 38 lines · Xiaoyao Li
Quoted 37 lines · Peter Zijlstra
Quoted 35 lines · Woody Suwalski

I am observing a 100% reproducible kvm crash on kernels starting with 5.7-rc1, always with the same opcode 0000.
It happens during wake up from the host suspended state. Worked OK on 5.6 and older.
The host is based on Debian testing, Thinkpad T440, i5 cpu.

[   61.576664] kernel BUG at arch/x86/kvm/x86.c:387!
[   61.576672] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   61.576678] CPU: 0 PID: 3851 Comm: qemu-system-x86 Not tainted 5.7-pingu #0
[   61.576680] Hardware name: LENOVO 20B6005JUS/20B6005JUS, BIOS GJETA4WW (2.54 ) 03/27/2020
[   61.576700] RIP: 0010:kvm_spurious_fault+0xa/0x10 [kvm]

Crash results in a dead kvm and occasionally a very unstable system.

Bisecting the problem between v5.6 and v5.7-rc1 points to

commit 6650cdd9a8ccf00555dbbe743d58541ad8feb6a7
Author: Peter Zijlstra (Intel) <[email protected]> Date:   Sun Jan 26 12:05:35 2020 -0800

     x86/split_lock: Enable split lock detection by kernel

Reversing that patch seems to actually "cure" the issue.

The problem is present in all kernels past 5.7-rc1, however the patch is not reversing directly in later source trees, so can not retest the logic on recent kernels.

Peter, would you have idea how to debug that (or even better - would you happen to know the fix)?

I have attached dmesg logs from a "good" 5.6.9 kernel, and then "bad" 5.7.0 and 5.8-rc3

Quoted 8 lines
Quoted 3 lines

I have no clue about kvm. Nor do I actually have hardware with SLD on. I've Cc'ed a bunch of folks who might have more ideas.

I think this bug is the same as the one found by Sean, and is already fixed in 5.8-rc4.

https://lore.kernel.org/kvm/[email protected]/

You are right, kvm works OK on 5.8-rc4.
The fix will need to be backported to 5.7.

Thanks, Woody

#4 lore raw
Quoted 45 lines · Woody Suwalski
Quoted 44 lines · Xiaoyao Li
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Woody Suwalski

I am observing a 100% reproducible kvm crash on kernels starting with 5.7-rc1, always with the same opcode 0000.
It happens during wake up from the host suspended state. Worked OK on 5.6 and older.
The host is based on Debian testing, Thinkpad T440, i5 cpu.

[   61.576664] kernel BUG at arch/x86/kvm/x86.c:387!
[   61.576672] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   61.576678] CPU: 0 PID: 3851 Comm: qemu-system-x86 Not tainted 5.7-pingu #0
[   61.576680] Hardware name: LENOVO 20B6005JUS/20B6005JUS, BIOS GJETA4WW (2.54 ) 03/27/2020
[   61.576700] RIP: 0010:kvm_spurious_fault+0xa/0x10 [kvm]

Crash results in a dead kvm and occasionally a very unstable system.

Bisecting the problem between v5.6 and v5.7-rc1 points to

commit 6650cdd9a8ccf00555dbbe743d58541ad8feb6a7
Author: Peter Zijlstra (Intel) <[email protected]> Date:   Sun Jan 26 12:05:35 2020 -0800

     x86/split_lock: Enable split lock detection by kernel

Reversing that patch seems to actually "cure" the issue.

The problem is present in all kernels past 5.7-rc1, however the patch is not reversing directly in later source trees, so can not retest the logic on recent kernels.

Quoted 9 lines
Quoted 5 lines

Peter, would you have idea how to debug that (or even better - would you happen to know the fix)?

I have attached dmesg logs from a "good" 5.6.9 kernel, and then "bad" 5.7.0 and 5.8-rc3

I have no clue about kvm. Nor do I actually have hardware with SLD on. I've Cc'ed a bunch of folks who might have more ideas.

I think this bug is the same as the one found by Sean, and is already fixed in 5.8-rc4.

Quoted 8 lines
Quoted 2 lines

https://lore.kernel.org/kvm/[email protected]/

You are right, kvm works OK on 5.8-rc4.
The fix will need to be backported to 5.7.

Thanks, Woody

I see it is already in 5.7.8. Great :-)
Thanks, Woody