Skip to contentSkip to conversation

New: Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

Bug 219787 [email protected] 2 participants 26 messages Feb 16, 2025 – Jul 6, 2025 KVM

lore
Share & tools
lore

[email protected]

author
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=219787

            Bug ID: 219787
           Summary: Guest's applications crash with EXCEPTION_SINGLE_STEP
                    (0x80000004)
           Product: Virtualization
           Version: unspecified
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 307665
--> https://bugzilla.kernel.org/attachment.cgi?id=307665&action=edit
Debugger attached to Steam.exe

Overview
========
Linux 6.13 update introduced problem with Windows guest's applications on AMD processors. Several applications crash with EXCEPTION_SINGLE_STEP (0x80000004). The list of confirmed software: CrystalDiskMark, Visual Studio Code, Steam, Looking Glass server, Windows Tweaker.
It never happened prior 6.13 update. I also checked 6.13.3rc and 6.14.1rc updates, problem persists there too. I did quick check differences in KVM/SVM between 6.12 and 6.13 and did not found anything that could set trapflag, so problem could be somewhere deeper inside kernel.

Steps to reproduce
==================
Run VM with Windows guest, launch any software from the list.

Hardware
========
CPU: AMD Ryzen 7 9800X3D (16) @ 5.27 GHz
MB: TUF GAMING X870-PLUS WIFI

Additional Information
======================
Steam crashes when downloading game, Looking Glass crashes on WinAPI QueryPerformanceCountrer call. Tested on Window 11 22H2/23H2/24H2.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#1 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

https://bugzilla.kernel.org/show_bug.cgi?id=219787

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
Kernel Version| |6.13

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#2 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #1 from [email protected] ---
Created attachment 307666
--> https://bugzilla.kernel.org/attachment.cgi?id=307666&action=edit
CrystalDiskMark installation

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#3 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Sean Christopherson ([email protected]) --- Are you able to bisect to an exact commit? There are significant KVM changes in 6.13, but they're almost all related to memory management. I can't think of anything that would manifest as an unexpected single step #DB, especially not with any consistency.

And just to double check, the only difference in the setup is that the host kernel was upgraded from v6.12 => v6.13? E.g. there was no QEMU update or guest-side changes?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#4 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from [email protected] ---
I have been able to reproduce this bug too on Linux 6.13.3 - Specifically whilst attempting to download/install any game via Steam in a GPU passthrough enabled Windows KVM guest.

Downgrading to Linux 6.12.9 - with no other changes made, immediately resolves the issue for me.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#6 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #4 from [email protected] ---
(In reply to Sean Christopherson from comment #2)

Quoted 8 lines

Are you able to bisect to an exact commit? There are significant KVM changes in 6.13, but they're almost all related to memory management. I can't think of anything that would manifest as an unexpected single step #DB, especially not with any consistency.

And just to double check, the only difference in the setup is that the host kernel was upgraded from v6.12 => v6.13? E.g. there was no QEMU update or guest-side changes?

I was not able to bisect yet, sorry. And yes, I double checked, the only change is kernel upgraded from v6.12.10 to v6.13.2 (did not checked v6.13.3 yet, but rc version had some behaviour).

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
reviewer
#5 lore raw

Re: [Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]
Quoted 4 lines · On Thu, Feb 20, 2025, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219787

[email protected] changed:

Quoted 8 lines

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from [email protected] ---
I have been able to reproduce this bug too on Linux 6.13.3 - Specifically whilst attempting to download/install any game via Steam in a GPU passthrough enabled Windows KVM guest.

Are you also running an AMD system?

author
#8 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #5 from Sean Christopherson ([email protected]) ---

Quoted 4 lines · On Thu, Feb 20, 2025, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219787

[email protected] changed:

Quoted 8 lines

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from [email protected] ---
I have been able to reproduce this bug too on Linux 6.13.3 - Specifically whilst attempting to download/install any game via Steam in a GPU passthrough enabled Windows KVM guest.

Are you also running an AMD system?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
reviewer
#7 lore raw

Re: [Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]
Quoted 9 lines · On Thu, Feb 20, 2025, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #4 from [email protected] ---
(In reply to Sean Christopherson from comment #2)

Quoted 5 lines

Are you able to bisect to an exact commit? There are significant KVM changes in 6.13, but they're almost all related to memory management. I can't think of anything that would manifest as an unexpected single step #DB, especially not with any consistency.

Quoted 5 lines
Quoted 3 lines

And just to double check, the only difference in the setup is that the host kernel was upgraded from v6.12 => v6.13? E.g. there was no QEMU update or guest-side changes?

I was not able to bisect yet, sorry.

No need to be sorry, you didn't introduce the bug :-)

Quoted 2 lines

And yes, I double checked, the only change is kernel upgraded from v6.12.10 to v6.13.2 (did not checked v6.13.3 yet, but rc version had some behaviour).

Please let me know if you'll be able to bisect (or not). Unless I have a random epiphany, this will likely require bisection.

author
#9 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #6 from Sean Christopherson ([email protected]) ---

Quoted 9 lines · On Thu, Feb 20, 2025, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #4 from [email protected] ---
(In reply to Sean Christopherson from comment #2)

Quoted 5 lines

Are you able to bisect to an exact commit? There are significant KVM changes in 6.13, but they're almost all related to memory management. I can't think of anything that would manifest as an unexpected single step #DB, especially not with any consistency.

Quoted 5 lines
Quoted 3 lines

And just to double check, the only difference in the setup is that the host kernel was upgraded from v6.12 => v6.13? E.g. there was no QEMU update or guest-side changes?

I was not able to bisect yet, sorry.

No need to be sorry, you didn't introduce the bug :-)

Quoted 2 lines

And yes, I double checked, the only change is kernel upgraded from v6.12.10 to v6.13.2 (did not checked v6.13.3 yet, but rc version had some behaviour).

Please let me know if you'll be able to bisect (or not). Unless I have a random
epiphany, this will likely require bisection.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#10 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #7 from [email protected] ---
(In reply to Sean Christopherson from comment #5)

Quoted 10 lines · On Thu, Feb 20, 2025, [email protected] wrote:
Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=219787

[email protected] changed:

What |Removed |Added

----------------------------------------------------------------------------

Quoted 2 lines

CC| |[email protected]

Quoted 7 lines
Quoted 3 lines

--- Comment #3 from [email protected] ---
I have been able to reproduce this bug too on Linux 6.13.3 - Specifically whilst attempting to download/install any game via Steam in a GPU

passthrough

Quoted 1 line

enabled Windows KVM guest.

Are you also running an AMD system?

Yep. I am running a 9800X3D in an X670E chipset motherboard. I honestly wonder if this bug only affects people using a 9800X3D.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#11 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #8 from [email protected] ---
(In reply to Sean Christopherson from comment #6)

Quoted 3 lines

Please let me know if you'll be able to bisect (or not). Unless I have a random
epiphany, this will likely require bisection.

Yes, I will try to bisect it tomorrow.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#12 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #9 from [email protected] ---
Created attachment 307690
--> https://bugzilla.kernel.org/attachment.cgi?id=307690&action=edit
bisection-log

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#13 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #10 from [email protected] ---
Created attachment 307691
--> https://bugzilla.kernel.org/attachment.cgi?id=307691&action=edit
bisection-config-culprit

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#14 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #11 from [email protected] ---
Here we go:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=408eb7417a92c5354c7be34f7425b305dfe30ad9

Double-checked both reverting commit or unsetting X86_BUS_LOCK_DETECT fixes the problem.

Added bisection log and config to attachments.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#15 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Ravi Bangoria ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #12 from Ravi Bangoria ([email protected]) --- Thanks for the bug report. This is what is probably happening:

BusLockTrap is controlled through DEBUGCTL MSR and currently DEBUGCTL MSR is saved/restored on guest entry/exit only if LBRV is enabled. So, if BusLockTrap is enabled on the host, it will remain enabled even after guest entry and thus, if some process inside the guest causes a BusLock, KVM will inject #DB from host to the guest.

I had a KVM patch[1] but couldn't get back to work on it. Let me try to spend some time and respin it.

[1] https://lore.kernel.org/all/[email protected]

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#16 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #13 from Sean Christopherson ([email protected]) ---

Quoted 11 lines · On Fri, Feb 21, 2025, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Ravi Bangoria ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #12 from Ravi Bangoria ([email protected]) --- Thanks for the bug report. This is what is probably happening:

Quoted 7 lines

BusLockTrap is controlled through DEBUGCTL MSR and currently DEBUGCTL MSR is saved/restored on guest entry/exit only if LBRV is enabled. So, if BusLockTrap
is enabled on the host, it will remain enabled even after guest entry and thus,
if some process inside the guest causes a BusLock, KVM will inject #DB from host to the guest.

*sigh*

Bluntly, that's horrific architecture. Why on earth isn't debugctl automatically
context switched when BusLockTrap is supported?

And does AMD do _any_ testing? This doesn't even require a full reproducer, e.g. the existing debug KVM-Unit-Test fails on my system (Turin) without ever generating a split/bus lock. AFAICT, the CPU is reporting bus locks in DR6 on #DBs that are most definitely not due to bus locks.

Quoted 4 lines

I had a KVM patch[1] but couldn't get back to work on it. Let me try to spend some time and respin it.

[1] https://lore.kernel.org/all/[email protected]

Virtualizing BusLockTrap won't do a damn thing. If the guest isn't using LBRs or BusLockTrap, then KVM won't enable LBR virtualization and so will run the guest with the host's DEBUGCTL.

Furthermore, running with the host's DEBUGCTL is a bug irrespective of BusLockTrap. It just happens to be fatal with BusLockTrap, but running with BTF=1 and whatever other bits may be enabled in the host most definitely isn't correct.

Bug reporters, can you test the attached patches? I have a reproducer in the form of a KVM test, but I haven't actually tested a Windows guest. Assuming squashing DEBUGCTL remedies the issue, I'll post patches after I've done a bit more testing.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#17 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #14 from Sean Christopherson ([email protected]) --- Created attachment 307695
--> https://bugzilla.kernel.org/attachment.cgi?id=307695&action=edit
0002-KVM-SVM-Manually-zero-restore-DEBUGCTL-if-LBR-virtua.patch

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#18 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #15 from [email protected] --- (In reply to Sean Christopherson from comment #13)

Quoted 5 lines

Bug reporters, can you test the attached patches? I have a reproducer in the form of a KVM test, but I haven't actually tested a Windows guest. Assuming squashing DEBUGCTL remedies the issue, I'll post patches after I've done a bit
more testing.

Tested, these 2 patches solves the issue.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#19 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #16 from Ravi Bangoria ([email protected]) --- (In reply to Sean Christopherson from comment #13)

Quoted 5 lines

And does AMD do _any_ testing? This doesn't even require a full reproducer, e.g. the existing debug KVM-Unit-Test fails on my system (Turin) without ever generating a split/bus lock. AFAICT, the CPU is reporting bus locks in DR6 on
#DBs that are most definitely not due to bus locks.

It seems, the CPU is preserving SW written DR6[BusLockDetected] while generating the #DB when the CPL is 0 and DEBUGCTL[BusLockTrapEn] is set.

Since most of the x86/debug.c KUT tests clears DR6[BusLockDetected] before executing the test, the bit remains cleared at the exception time which causes tests to fail.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#20 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Jon Betti ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #17 from Jon Betti ([email protected]) --- (In reply to whanos from comment #7)

Quoted 1 line

I honestly wonder if this bug only affects people using a 9800X3D.

I'm running a 9950X and had repro as another data point. (I had this issue for a few weeks but thought it was Steam until I started looking at crash dumps in Windows... and I thankfully stumbled onto this bug which restored my sanity :).)

(In reply to rangemachine from comment #15)

Quoted 3 lines

(In reply to Sean Christopherson from comment #13)

Quoted 1 line

Bug reporters, can you test the attached patches?

Tested, these 2 patches solves the issue.

+1. Patched my kernel and the issue went away (again: 'twas Steam for me that threw the exception).

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#21 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #18 from Ravi Bangoria ([email protected]) --- (In reply to Ravi Bangoria from comment #16)

Quoted 2 lines

It seems, the CPU is preserving SW written DR6[BusLockDetected] while generating the #DB when the CPL is 0 and DEBUGCTL[BusLockTrapEn] is set.

My bad, the behavior is same for CPL 3 as well. Apparently, it's a correct behavior as documented in the AMD Architecture Programmer's Manual. I've posted a KUT patch to KVM mailing list. (More details in the patch). Please review.

https://lore.kernel.org/r/[email protected]

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#22 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Brian ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #19 from Brian ([email protected]) ---
I can confirm I have the same problem.

CPU is AMD 9950 X3D, and I have the problem both in Virtual Box, and VMware Workstation.

Can anyone point to a guide to get around this problem now? I have been running Linux for many years, but it almost just as many years since I compiled my own kernel.

I am running Ubuntu 25.04

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#23 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #20 from Ravi Bangoria ([email protected]) --- The KVM fix went in v6.14-rc6. These are the corresponding patches:

https://git.kernel.org/torvalds/c/ee89e8013383    v6.14-rc6
https://git.kernel.org/torvalds/c/d0eac42f5cec    v6.14-rc6
https://git.kernel.org/torvalds/c/fb71c7959356    v6.14-rc6
https://git.kernel.org/torvalds/c/433265870ab3    v6.14-rc6
https://git.kernel.org/torvalds/c/189ecdb3e112    v6.14-rc6
https://git.kernel.org/torvalds/c/5ecdb48dd918    v6.16-rc1
Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#24 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

--- Comment #21 from Brian ([email protected]) ---
How can I still have the problem on Ubuntu 6.14.0-22-generic with VMware Workstation and VirtualBox?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
author
#25 lore raw

[Bug 219787] Guest's applications crash with EXCEPTION_SINGLE_STEP (0x80000004)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219787

Artem S. Tashkinov ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #22 from Artem S. Tashkinov ([email protected]) --- (In reply to Brian from comment #21)

Quoted 2 lines

How can I still have the problem on Ubuntu 6.14.0-22-generic with VMware Workstation and VirtualBox?

VMWare Workstation and VirtualBox use their own supervisors (kernel modules).

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.