Skip to contentSkip to conversation

New: Machine will not wake from suspend if KVM VM is running

Bug 219112 [email protected] 2 participants 9 messages Jul 30, 2024 – Nov 5, 2024 KVM

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Steps to reproduce:

1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a VM.

2) Suspend to RAM

3) Wake from suspend

Expected outcome: Machine should resume from suspend when a KVM VM is running

Observed outcome: Machine will not resume from suspend

I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.

This is on an AMD Ryzen 7 5850U laptop.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw
Quoted 22 lines · On Tue, Jul 30, 2024, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Steps to reproduce:

1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a VM.

Quoted 7 lines

2) Suspend to RAM

3) Wake from suspend

Expected outcome: Machine should resume from suspend when a KVM VM is running

Observed outcome: Machine will not resume from suspend

As in, suspend/resume the host, and the host never resumes?

Quoted 1 line

I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into

Have you tried older kernels? If so, is bisecting possible? E.g. there was a somewhat related rework in 6.1 (IIRC).

Quoted 10 lines

the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.

This is on an AMD Ryzen 7 5850U laptop.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #1 from Sean Christopherson ([email protected]) ---

Quoted 23 lines · On Tue, Jul 30, 2024, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Bug ID: 219112
Summary: Machine will not wake from suspend if KVM VM is
running
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Steps to reproduce:

1) Start a KVM virtual machine with QEMU. I used Quickemu to build and start a
VM.

Quoted 7 lines

2) Suspend to RAM

3) Wake from suspend

Expected outcome: Machine should resume from suspend when a KVM VM is running

Observed outcome: Machine will not resume from suspend

As in, suspend/resume the host, and the host never resumes?

Quoted 1 line

I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran into

Have you tried older kernels? If so, is bisecting possible? E.g. there was a somewhat related rework in 6.1 (IIRC).

Quoted 10 lines

the same problem. Unfortunately, I can't find a kernel version where this problem isn't present. Nothing gets printed to the system or kernel logs.

This is on an AMD Ryzen 7 5850U laptop.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #2 from AlexDeLorenzo.dev ([email protected]) --- (In reply to Sean Christopherson from comment #1)

Quoted 6 lines
Quoted 1 line

Expected outcome: Machine should resume from suspend when a KVM VM is

running

Quoted 2 lines

Observed outcome: Machine will not resume from suspend

As in, suspend/resume the host, and the host never resumes?

Yup, the host never resumes. It's also might be possible that the host never suspends correctly in the first place. I've noticed that sometimes the final lock screen image freezes on external monitors, suggesting that a full suspend wasn't successful.

Quoted 6 lines
Quoted 1 line

I tried this on 6.6.42 LTS, 6.9.x, and 6.10-rc1 through 6.10.2, and ran

into

Have you tried older kernels? If so, is bisecting possible? E.g. there was a
somewhat related rework in 6.1 (IIRC).

Several months ago, this wasn't a problem when I was using older kernels. Unfortunately, I don't remember what versions I was using at that point, just that they were 6.x.

I'll try with some 6.1 era kernels to see if the problem is still there.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Brandon Nielsen ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Brandon Nielsen ([email protected]) ---
I have verified this on my systems as well. One with an AMD 5800x3D, another with an Intel i7-8550U. Both have virtualization support enabled in their respective BIOSes. Usermode VMs used on both machines. Kernel 6.11.5 currently, but I haven't been using VMs frequently lately so I don't know when this behavior started.

I think suspend is being entered properly "kernel: PM: suspend entry (deep)" is seen in the journal, and I think exiting suspend at least starts "kernel: PM:
suspend exit" is seen in the journal along with a bunch of other resuming stuff (network interfaces being brought up, clocks being synchronized, etc.).

Unfortunately the displays never wake up and the machine never becomes usable.

I even see my power button press being logged when I eventually need to hard kill the machine so I can reboot.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #4 from Brandon Nielsen ([email protected]) --- I am unable to recreate with non-user mode VMs.

So, at least in my case, the real issue is probably in systemd:
https://bugzilla.redhat.com/show_bug.cgi?id=2321268

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #5 from Sean Christopherson ([email protected]) --- Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].

Long explanation very short, can you try setting KVM's "nx_huge_pages" module param to "never"? E.g. via

echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages

Note, the module param is writable even while KVM is loaded, but it needs to be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.

AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.

[*] https://lore.kernel.org/all/[email protected]

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#7 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #6 from Brandon Nielsen ([email protected]) --- (In reply to Sean Christopherson from comment #5)

Quoted 14 lines

Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].

Long explanation very short, can you try setting KVM's "nx_huge_pages" module param to "never"? E.g. via

echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages

Note, the module param is writable even while KVM is loaded, but it needs to be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.

Quoted 5 lines

AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.

[*] https://lore.kernel.org/all/[email protected]

Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop that is set to "N", in both cases, setting it to "never" seems to fix suspend. I can't find the documentation for that particular tunable, but that seems odd.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#8 lore raw

[Bug 219112] Machine will not wake from suspend if KVM VM is running

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=219112

--- Comment #7 from Sean Christopherson ([email protected]) --- (In reply to Brandon Nielsen from comment #6)

Quoted 20 lines

(In reply to Sean Christopherson from comment #5)

Quoted 10 lines

Heh, that was quite the rabbit hole. Poking around reminded me of a bug report from Tejun about KVM worker threads not freezing in certain scenarios[*].

Long explanation very short, can you try setting KVM's "nx_huge_pages" module param to "never"? E.g. via

echo "never" | sudo tee /sys/module/kvm/parameters/nx_huge_pages

Note, the module param is writable even while KVM is loaded, but it needs

to

Quoted 8 lines

be set to "never' _before_ launching any VMs, in which case KVM won't spawn the associated worker thread. I.e. the "never" setting only affects worker threads for new VMs.

AFAIK, that's the only problematic flavor of worker thread in KVM. If that makes the suspend/resume problem go away, then odds are very good the underlying issue is the same one Tejun reported.

Quoted 6 lines
Quoted 1 line

[*] https://lore.kernel.org/all/[email protected]

Interesting, so on my Intel laptop that is set to "Y", on my AMD desktop that is set to "N", in both cases, setting it to "never" seems to fix suspend. I can't find the documentation for that particular tunable, but that seems odd.

The behavior is expected and working as intended. The parameter controls a mitigation that is enabled by default if and only if the CPU is affected by L1TF. But, KVM creates a kthread related to the mitigation even if the mitigation is disabled, so that KVM doesn't need to try and spawn/destroy the kthread on-demand if the user/admin toggles the mitigation while the VM is running.

The "never" setting is a relatively recent addition that allows the admin to commit to never enabling the mitigation (without reloading KVM), specifically so that KVM can skip creating the kthread, e.g. to reduce VM creation latency.

As for documentation, I'll add that to the todo list, I simply forgot to add it when introducing the "never" option.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.