New: QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
https://bugzilla.kernel.org/show_bug.cgi?id=197951
Bug ID: 197951
Summary: QEMU/KVM & VFIO & PCI passthru with Windows 10 x64
guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
Product: Virtualization
Version: unspecified
Kernel Version: 4.12.13
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Originally reported here: https://bugs.launchpad.net/qemu/+bug/1728256
The title explains it all. When running a Windows 10 x64 guest that passes through a PCIe device with VFIO (especially a GPU, or perhaps only if at least one passed device is a GPU, since nobody who isn't passing through a GPU has confirmed this) on kernel version 4.12.13 and up, the guest will crash with the CRITICAL_STRUCTURE_CORRUPTION blue screen of death randomly when accessing the host's memory. You have to be doing something memory intensive, like mining or gaming, for it to happen often enough to really notice, but if you do, it happens once an hour or every few hours.
I downgraded to 4.12.12 and the issue went away. Others disabled swap instead of downgrading, and the issue went away. Disabling swap isn't an option for me, so I can't upgrade the kernel until this is fixed.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #1 from Jimi ([email protected]) ---
Also, I use an AMD GPU and another user reported using NVidia, so this seems to not be vendor-specific, if it even is GPU-specific.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
Alex Williamson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #2 from Alex Williamson ([email protected]) --- Seems like a simple matter of bisecting between 4.12.12 and 4.12.13 then, it's a very short list:
$ git log --oneline v4.12.12..v4.12.13 5d7d2e03e0f0 Linux 4.12.13 9f7df0bca168 xfs: XFS_IS_REALTIME_INODE() should be false if no rt device presen da0f4931ec52 NFSv4: Fix up mirror allocation 3307d5f5099c NFS: Sync the correct byte range during synchronous writes 6f50e3a1b8c3 NFS: Fix 2 use after free issues in the I/O code 7714f302294d ARM: 8692/1: mm: abort uaccess retries upon fatal signal b9a489e1d4a3 ARM64: dts: marvell: armada-37xx: Fix GIC maintenance interrupt 8329b5e8c6cf Bluetooth: Properly check L2CAP config option output buffer length 99dc1296b47c rt2800: fix TX_PIN_CFG setting for non MT7620 chips 2bce0fe7d0cd KVM: SVM: Limit PFERR_NESTED_GUEST_PAGE error_code check to L1 gues 9d6412aa06ce ALSA: msnd: Optimize / harden DSP and MIDI loops 846073130799 mm/memory.c: fix mem_cgroup_oom_disable() call missing 46791eb9f13e mm/swapfile.c: fix swapon frontswap_map memory leak on error 637f25e5ba94 mm: kvfree the swap cluster info if the swap file is unsatisfactory 58989dc3af0d selftests/x86/fsgsbase: Test selectors 1, 2, and 3 9ed3dc1c0431 radix-tree: must check __radix_tree_preload() return value 0af760ab3882 rtlwifi: btcoexist: Fix breakage of ant_sel for rtl8723be 8004198bb025 btrfs: resume qgroup rescan on rw remount 9a5537a76b62 nvme-fabrics: generate spec-compliant UUID NQNs 02c54b35cad8 mtd: nand: qcom: fix config error for BCH f2339a072e47 mtd: nand: qcom: fix read failure without complete bootchain 71515c37777d mtd: nand: mxc: Fix mxc_v1 ooblayout c54a31845019 mtd: nand: hynix: add support for 20nm NAND chips 2b8b46b24217 mtd: nand: make Samsung SLC NAND usable again
Let us know the results.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #3 from Jimi ([email protected]) ---
Sure, I'll start bisecting next time I get the chance (maybe tomorrow). It'll take a long time, though, since the BSOD might only happen once a day. I'll have to run the same commit for a few days before I'm confident that it isn't BSODing. Thank god for binary search.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
Fabian Grünbichler ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #4 from Fabian Grünbichler ([email protected]) --- Did you have a chance to bisect yet? We are experiencing a similar issue with 4.13 and 4.14 based kernels, and our test case and bisect points to a series not contained in 4.12.13:
https://forum.proxmox.com/threads/blue-screen-with-5-1.37664/
https://lkml.kernel.org/r/<[email protected]>
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #5 from Jimi ([email protected]) ---
I've been doing it. Currently on "[3307d5f5099c186d1ae43205eb23c29fabc6f5b8]
NFS: Sync the correct byte range during synchronous writes" with 2 commits left to test after it. They've all been good commits so far.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #6 from Ladi Prosek ([email protected]) ---
I have seen this crash on a Windows 10 x64 guest *without* any kind of device assignment. Didn't keep track of exact kernel versions but it was Fedora 26, very likely 4.12.*.
If you've been able to build a kernel where this happens for you, try cherry-picking:
commit a2b7861bb33b2538420bb5d8554153484d3f961f
Author: Boqun Feng <[email protected]> Date: Tue Oct 3 21:36:51 2017 +0800
kvm/x86: Avoid async PF preempting the kernel incorrectly
Currently, in PREEMPT_COUNT=n kernel, kvm_async_pf_task_wait() could call schedule() to reschedule in some cases. This could result in accidentally ending the current RCU read-side critical section early, causing random memory corruption in the guest, or otherwise preempting the currently running task inside between preempt_disable and preempt_enable.
Keywords: "PF" (since the report mentions swap), "random memory corruption in the guest"
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #7 from Ladi Prosek ([email protected]) ---
Correction: It looks like a2b7861bb33b2538420bb5d8554153484d3f961f is more of a guest-side fix with no effect on non-Linux guests. Please ignore it.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
Jack Wang ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #8 from Jack Wang ([email protected]) ---
We've seen windows 10 BSOD with CRITICAL_STRUCTURE_CORRUPTION short after migration. host kernel version is 4.4.50, no device passthrough, no swap in our case.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #9 from Ladi Prosek ([email protected]) --- (In reply to Jack Wang from comment #8)
Quoted 3 lines
We've seen windows 10 BSOD with
CRITICAL_STRUCTURE_CORRUPTIONshort after migration. host kernel version is 4.4.50, no device passthrough, no swap in our case.
What kernel version was running on the migration source? Thanks!
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #10 from Jack Wang ([email protected]) --- Source server were running kernel 3.12.45.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #11 from Jimi ([email protected]) ---
I'm about to spend a few days with it installed to make sure, but it looks like this commit is probably our culprit:
$ git bisect good Bisecting: 0 revisions left to test after this (roughly 1 step) [9f7df0bca168528aba20794f400be134495551b8] xfs: XFS_IS_REALTIME_INODE() should be false if no rt device present
It looks like there's some evidence that this issue doesn't *only* come from 4.12.13. I want to reiterate, I was on 4.12.13 when this problem started happening to me, and I haven't had a single BSOD since downgrading to 4.12.12, including during this entire bisect. It was happening frequently enough that if 4.12.13 wasn't at least one of the cuprits, I definitely would've had a few BSODs by now.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #12 from Ladi Prosek ([email protected]) --- (In reply to Jimi from comment #11)
Quoted 7 lines
I'm about to spend a few days with it installed to make sure, but it looks like this commit is probably our culprit:
$ git bisect good Bisecting: 0 revisions left to test after this (roughly 1 step) [9f7df0bca168528aba20794f400be134495551b8] xfs: XFS_IS_REALTIME_INODE() should be false if no rt device present
A few things hint at this being a red herring.
* It's the first commit before the 4.12.13 tag which means that you marked 4.12.13 as bad and everything else as good.
* There's nothing in it that would explain why it affects only virt and only Windows guests.
Quoted 6 lines
It looks like there's some evidence that this issue doesn't *only* come from 4.12.13. I want to reiterate, I was on 4.12.13 when this problem started happening to me, and I haven't had a single BSOD since downgrading to 4.12.12, including during this entire bisect. It was happening frequently enough that if 4.12.13 wasn't at least one of the cuprits, I definitely would've had a few BSODs by now.
The bug is likely timing sensitive and just rebuilding the kernel, out of the same sources, may end up more (or less) prone to it just by how the binary is laid out, the exact compiler used etc.
Also, we should not rule out the possibility that the problem has existed for a long time and Windows 10 got the ability to detect certain corruptions recently via a Windows Update patch.
I hit it again yesterday and the BSOD analyzes to:
CRITICAL_STRUCTURE_CORRUPTION (109)
This bugcheck is generated when the kernel detects that critical kernel code or data have been corrupted. There are generally three causes for a corruption:
1) A driver has inadvertently or deliberately modified critical kernel code or data. See http://www.microsoft.com/whdc/driver/kernel/64bitPatching.mspx
2) A developer attempted to set a normal kernel breakpoint using a kernel debugger that was not attached when the system was booted. Normal breakpoints, "bp", can only be set if the debugger is attached at boot time. Hardware breakpoints, "ba", can be set at any time.
3) A hardware corruption occurred, e.g. failing RAM holding kernel code or data.
Arguments:
Arg1: a3a0206143b9d5b3, Reserved
Arg2: b3b72ce7963bad06, Reserved
Arg3: 0000032000000000, Failure type dependent information
Arg4: 0000000000000017, Type of corrupted region, can be
[...]
16 : Critical floating point control register modification
17 : Local APIC modification
18 : Kernel notification callout modification[...]
I'm pretty sure that last time I got it the type of corrupted region was 17 as well.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #13 from Fabian Grünbichler ([email protected]) --- (In reply to Fabian Grünbichler from comment #4)
Quoted 6 lines
Did you have a chance to bisect yet? We are experiencing a similar issue with 4.13 and 4.14 based kernels, and our test case and bisect points to a series not contained in 4.12.13:
FWIW, the 4.13 and 4.14 issue was caused by the linked series, and a subsequent patch[1] solved it completely for us.
1: https://lkml.kernel.org/r/<[email protected]>
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #14 from Jack Wang ([email protected]) ---
In my case, it's region 7 : Critical MSR modification, I guess in my case
just some under line MSR state changed during migration from 3.12 to 4.4 kernel.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #15 from Ladi Prosek ([email protected]) --- (In reply to Fabian Grünbichler from comment #13)
Quoted 4 lines
FWIW, the 4.13 and 4.14 issue was caused by the linked series, and a subsequent patch[1] solved it completely for us.
Thanks! I have added this patch to my kernel (4.13.16 based, built locally, reproduces the BSOD). Will report back in a few days.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 197951] QEMU/KVM & VFIO & PCI passthru with Windows 10 x64 guest: memory access intermittently causes CRITICAL_STRUCTURE_CORRUPTION BSOD unless swap is disabled on host, since 4.12.13
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=197951
--- Comment #16 from Ladi Prosek ([email protected]) --- (In reply to Ladi Prosek from comment #15)
Quoted 8 lines
(In reply to Fabian Grünbichler from comment #13)
Quoted 4 lines
FWIW, the 4.13 and 4.14 issue was caused by the linked series, and a subsequent patch[1] solved it completely for us.
Thanks! I have added this patch to my kernel (4.13.16 based, built locally, reproduces the BSOD). Will report back in a few days.
No crashes so far. The fix is in Linus's tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b1394e745b9453dcb5b0671c205b770e87dedb87
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.