New: Machine will not wake from suspend if KVM VM is running
https://bugzilla.kernel.org/show_bug.cgi?id=219112
Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Steps to reproduce:
1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a VM.
2) Suspend to RAM
3) Wake from suspend
Expected outcome: Machine should resume from suspend when a KVM VM is running
Observed outcome: Machine will not resume from suspend
I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.
This is on an AMD Ryzen 7 5850U laptop.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Sean Christopherson
reviewerQuoted 22 lines · On Tue, Jul 30, 2024, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219112
Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoSteps to reproduce:
1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a VM.
Quoted 7 lines
2) Suspend to RAM
3) Wake from suspend
Expected outcome: Machine should resume from suspend when a KVM VM is running
Observed outcome: Machine will not resume from suspend
As in, suspend/resume the host, and the host never resumes?
Quoted 1 line
I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into
Have you tried older kernels? If so, is bisecting possible? E.g. there was a somewhat related rework in 6.1 (IIRC).
Quoted 10 lines
the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.
This is on an AMD Ryzen 7 5850U laptop.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
--- Comment #1 from Sean Christopherson ([email protected]) ---
Quoted 23 lines · On Tue, Jul 30, 2024, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219112
Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoSteps to reproduce:
1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a
VM.
Quoted 7 lines
2) Suspend to RAM
3) Wake from suspend
Expected outcome: Machine should resume from suspend when a KVM VM is running
Observed outcome: Machine will not resume from suspend
As in, suspend/resume the host, and the host never resumes?
Quoted 1 line
I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into
Have you tried older kernels? If so, is bisecting possible? E.g. there was a somewhat related rework in 6.1 (IIRC).
Quoted 10 lines
the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.
This is on an AMD Ryzen 7 5850U laptop.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
--- Comment #2 from AlexDeLorenzo.dev ([email protected]) --- (In reply to Sean Christopherson from comment #1)
Quoted 6 lines
Quoted 1 line
Expected outcome: Machine should resume from suspend when a KVM VM is
running
Quoted 2 lines
Observed outcome: Machine will not resume from suspend
As in, suspend/resume the host, and the host never resumes?
Yup, the host never resumes. It's also might be possible that the host never suspends correctly in the first place. I've noticed that sometimes the final lock screen image freezes on external monitors, suggesting that a full suspend wasn't successful.
Quoted 6 lines
Quoted 1 line
I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran
into
Have you tried older kernels? If so, is bisecting possible? E.g. there was a
somewhat related rework in 6.1 (IIRC).
Several months ago, this wasn't a problem when I was using older kernels. Unfortunately, I don't remember what versions I was using at that point, just that they were 6.x.
I'll try with some 6.1 era kernels to see if the problem is still there.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
Brandon Nielsen ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #3 from Brandon Nielsen ([email protected]) ---
I have verified this on my systems as well. One with an AMD 5800x3D, another with an Intel i7-8550U. Both have virtualization support enabled in their respective BIOSes. Usermode VMs used on both machines. Kernel 6.11.5 currently, but I haven't been using VMs frequently lately so I don't know when this behavior started.
I think suspend is being entered properly "kernel: PM: suspend entry (deep)" is seen in the journal, and I think exiting suspend at least starts "kernel: PM:
suspend exit" is seen in the journal along with a bunch of other resuming stuff (network interfaces being brought up, clocks being synchronized, etc.).
Unfortunately the displays never wake up and the machine never becomes usable.
I even see my power button press being logged when I eventually need to hard kill the machine so I can reboot.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
--- Comment #4 from Brandon Nielsen ([email protected]) --- I am unable to recreate with non-user mode VMs.
So, at least in my case, the real issue is probably in systemd:
https://bugzilla.redhat.com/show_bug.cgi?id=2321268
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
Sean Christopherson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #5 from Sean Christopherson ([email protected]) --- Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].
Long explanation very short, can you try setting KVM's "nx_huge_pages" module param to "never"? E.g. via
echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages
Note, the module param is writable even while KVM is loaded, but it needs to be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.
AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.
[*] https://lore.kernel.org/all/[email protected]
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
--- Comment #6 from Brandon Nielsen ([email protected]) --- (In reply to Sean Christopherson from comment #5)
Quoted 14 lines
Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].
Long explanation very short, can you try setting KVM's "
nx_huge_pages" module param to "never"? E.g. viaecho "never" | sudo tee
/sys/module/kvm/parameters/nx_huge_pagesNote, the module param is writable even while KVM is loaded, but it needs to be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.
Quoted 5 lines
AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.
Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop that is set to "N", in both cases, setting it to "never" seems to fix suspend. I can't find the documentation for that particular tunable, but that seems odd.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219112] Machine will not wake from suspend if KVM VM is running
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219112
--- Comment #7 from Sean Christopherson ([email protected]) --- (In reply to Brandon Nielsen from comment #6)
Quoted 20 lines
(In reply to Sean Christopherson from comment #5)
Quoted 10 lines
Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].
Long explanation very short, can you try setting KVM's "
nx_huge_pages" module param to "never"? E.g. viaecho "never" | sudo tee
/sys/module/kvm/parameters/nx_huge_pagesNote, the module param is writable even while KVM is loaded, but it needs
to
Quoted 8 lines
be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.
AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.
Quoted 6 lines
Quoted 1 line
Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop that is set to "N", in both cases, setting it to "never" seems to fix suspend. I can't find the documentation for that particular tunable, but that seems odd.
The behavior is expected and working as intended. The parameter controls a mitigation that is enabled by default if and only if the CPU is affected by L1TF. But, KVM creates a kthread related to the mitigation even if the mitigation is disabled, so that KVM doesn't need to try and spawn/destroy the kthread on-demand if the user/admin toggles the mitigation while the VM is running.
The "never" setting is a relatively recent addition that allows the admin to commit to never enabling the mitigation (without reloading KVM), specifically so that KVM can skip creating the kthread, e.g. to reduce VM creation latency.
As for documentation, I'll add that to the todo list, I simply forgot to add it when introducing the "never" option.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.