New: Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
https://bugzilla.kernel.org/show_bug.cgi?id=219009
Bug ID: 219009
Summary: Random host reboots on Ryzen 7000/8000 using nested
VMs (vls suspected)
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Running nested VMs on AMD Ryzen 7000/8000 (ZEN4) CPUs results in random host's reboots.
There is no kernel panic, no log entries, no relevant output to serial console. It is as if platform is simply hard reset. It seems time to reproduce it varies from system to system and can be dependent on workload and even specific CPU model.
I can reproduce it with kernel 6.9.7 and qemu 9.0 on Ryzen 7950X3D under one hour by using KVM -> Windows 10/11 with Hyper-V services on or KVM -> Windows 10/11 with 3 VBox VMs (also Win11) running. Others people had it repeatedly reproduced on Ryzen 7700,7600 and 8700GE, including KVM -> KVM -> Linux.[1] I also have seen Hetzner (company offering Ryzen based dedicated servers) customers complaining about similiar random reboots.
I tried looking up errata for Ryzen 7000/8000, but could not find one published, so I decided to check errata for EPYC 9004 [2], which is also Zen4 arch as Ryzen 7000/8000. It has nesting related bug #1495 (on page 49), which mentions using Virtualized VMLOAD/VMSAVE can result in MCE and/or system reset.
Based on that errata mentioned above, I reconfigured my system with kvm_amd.vls=0 and for me random reboots with nested virtualization stopped. Same was reported by several people from [1].
Somebody from AMD must be asked to confirm if it is really Ryzen 7000/8000 hardware bug, and if there is a better fix than disabling VLS as it has performance hit. If disabling it is the only fix, then kvm_amd.vls=0 must be default for Ryzen 7000/8000.
[1]
https://www.reddit.com/r/Proxmox/comments/1cym3pl/nested_virtualization_crashing_ryzen_7000_series/
[2]
https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/revision-guides/57095-PUB_1_01.pdf
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
https://bugzilla.kernel.org/show_bug.cgi?id=219009
h4ck3r ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from h4ck3r ([email protected]) --- To all Zen4 users!!!
I'm now experimenting with different settings and found way to enable nested virtualization with host cpu type and no performance penalty.
Disabling aspm in bios does fix it for win11 guest. (proxmox, hyperv enabled) Further testing is required, so anyone having this issue can contribute by testing other guest operating systems.
For now it's best solution until fix from AMD arrives. Unless its irreparable with just microcode update...
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Ben Hirlston ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #2 from Ben Hirlston ([email protected]) ---
I can confirm on my Ryzen 9 7900X30 system disabling aspm helps. kind of relieved this is a bug. I was having this issue on my Ryzen 7 5700G system but to a lessor extent and it got worse when I upgraded to Ryzen 7000 series
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #3 from Ben Hirlston ([email protected]) --- (In reply to Ben Hirlston from comment #2)
Quoted 3 lines
I can confirm on my Ryzen 9 7900X30 system disabling aspm helps. kind of relieved this is a bug. I was having this issue on my Ryzen 7 5700G system but to a lessor extent and it got worse when I upgraded to Ryzen 7000 series
there is a typo I meant 7900X3D sorry about that
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #4 from Ben Hirlston ([email protected]) --- (In reply to Ben Hirlston from comment #3)
Quoted 7 lines
(In reply to Ben Hirlston from comment #2)
Quoted 3 lines
I can confirm on my Ryzen 9 7900X30 system disabling aspm helps. kind of relieved this is a bug. I was having this issue on my Ryzen 7 5700G system but to a lessor extent and it got worse when I upgraded to Ryzen 7000
series
there is a typo I meant 7900X3D sorry about that
no I had to do kvm_amd.vls=0 the hit to performance wasn't as bad as I thought but disabling aspm didn't help I was wrong
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Sagnik Sasmal ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #5 from Sagnik Sasmal ([email protected]) ---
I can confirm the same problem occurring as well on 7950X and 7950X3D CPUs.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #6 from Ben Hirlston ([email protected]) ---
do we know if Ryzen 9000 has this issue? I know I had this issue on Ryzen 5000 but to a lessor extent
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #7 from h4ck3r ([email protected]) --- Update - aspm does not work, thou it decreased chance of random reboot. Issue might be related to memory addressing method used in zen4.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #8 from h4ck3r ([email protected]) --- (In reply to Ben Hirlston from comment #6)
Quoted 2 lines
do we know if Ryzen 9000 has this issue? I know I had this issue on Ryzen 5000 but to a lessor extent
Could you elaborate on what was happening with 5000? (reboots, mce, something other)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #9 from Ben Hirlston ([email protected]) --- (In reply to h4ck3r from comment #8)
Quoted 6 lines
(In reply to Ben Hirlston from comment #6)
Quoted 2 lines
do we know if Ryzen 9000 has this issue? I know I had this issue on Ryzen 5000 but to a lessor extent
Could you elaborate on what was happening with 5000? (reboots, mce, something other)
I would be using my Virtual Machine to Windows Windows 11 and would be doing something intensive that was using vls and the machine would reset just like 7000 but it happened way less often
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #10 from Ben Hirlston ([email protected]) --- (In reply to Ben Hirlston from comment #9)
Quoted 11 lines
(In reply to h4ck3r from comment #8)
Quoted 6 lines
(In reply to Ben Hirlston from comment #6)
Quoted 2 lines
do we know if Ryzen 9000 has this issue? I know I had this issue on Ryzen 5000 but to a lessor extent
Could you elaborate on what was happening with 5000? (reboots, mce, something other)
I would be using my Virtual Machine to Windows Windows 11 and would be doing something intensive that was using vls and the machine would reset just like 7000 but it happened way less often
I no longer have that machine anymore so I can't test it anymore but I have memory of it happening on a bi weekly to monthly occurrence
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #11 from h4ck3r ([email protected]) ---
Im afraid it might be overlooked issue that propagated to zen5. If there is someone with zen5 let know if nestes virt also breaks on it.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #12 from h4ck3r ([email protected]) --- Kernel and qemu devs perspective is also appriciated on that topic.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
blake ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #13 from blake ([email protected]) ---
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it pretty easy to reproduce, just takes some time.
Setting kvm_amd.vls=0 seems to have resolved the issue, we had zero node resets today and I was trying to force them.
Kernel is Proxmox's 6.8.12-1-pve.
Thanks,
Blake
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #14 from h4ck3r ([email protected]) --- (In reply to blake from comment #13)
Quoted 7 lines
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it pretty easy to reproduce, just takes some time.
Quoted 7 lines
Setting kvm_amd.vls=0 seems to have resolved the issue, we had zero node resets today and I was trying to force them.
Kernel is Proxmox's 6.8.12-1-pve.
Thanks,
Blake
Disabling vls does help with crashes, but has too much performance penalty. In my case it would lock gpu utilization at 40% max. (proxmox win10/11 hyperv enabled)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #15 from Ben Hirlston ([email protected]) ---
the kernel I am running for reference is 6.10.6-zen1-1-zen and I have a Ryzen 9 7900X3D
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #16 from blake ([email protected]) ---
At least for me, I haven't noticed any performance hit. These systems are all headless though.
Quoted 28 lines · On Sat, Aug 31, 2024, at 6:51 AM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #14 from h4ck3r ([email protected]) --- (In reply to blake from comment #13)
Quoted 7 lines
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it pretty easy to reproduce, just takes some time.
Quoted 7 lines
Setting kvm_amd.vls=0 seems to have resolved the issue, we had zero node resets today and I was trying to force them.
Kernel is Proxmox's 6.8.12-1-pve.
Thanks,
BlakeDisabling vls does help with crashes, but has too much performance penalty. In my case it would lock gpu utilization at 40% max. (proxmox win10/11 hyperv enabled)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are on the CC list for the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #17 from Ben Hirlston ([email protected]) --- (In reply to blake from comment #16)
Quoted 22 lines
Created attachment 306799 [details]
attachment-12508-0.htmlAt least for me, I haven't noticed any performance hit. These systems are all headless though.
Quoted 9 lines · On Sat, Aug 31, 2024, at 6:51 AM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #14 from h4ck3r ([email protected]) --- (In reply to blake from comment #13)
Quoted 5 lines
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it
pretty
Quoted 5 lines
Quoted 5 lines
easy to reproduce, just takes some time.
Setting kvm_amd.vls=0 seems to have resolved the issue, we had zero node resets today and I was trying to force them.
Quoted 15 lines
Quoted 7 lines
Quoted 4 lines
Kernel is Proxmox's 6.8.12-1-pve.
Thanks,
BlakeDisabling vls does help with crashes, but has too much performance penalty. In my case it would lock gpu utilization at 40% max. (proxmox win10/11
hyperv
Quoted 7 lines
enabled)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are on the CC list for the bug.
the performance hit for me hasn't been that bad but disabling vls is the same as disabling svm for the vm its more like telling the VM the CPU is incapable of nested virtualization that is what I observed
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #18 from Ben Hirlston ([email protected]) ---
I was wondering if this issue might be related to this thread on gentoo's bug tracker?
https://bugs.gentoo.org/808990
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #19 from Ben Hirlston ([email protected]) --- my curiousity relates to this new bug being caused by fixing the CVE's CVE-2021-3653, CVE-2021-3656
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #20 from Ben Hirlston ([email protected]) --- I am wondering if the fixes for those CVE's are related to this bug
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #21 from [email protected] ---
Nope, these are not related, but do check if disabling AVIC helps (set enable_avic parameter of kvm_amd to 0)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #22 from [email protected] ---
I mean avic=0
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Quoted 46 lines · On Sat, 2024-07-06 at 11:20 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219009
Bug ID: 219009
Summary: Random host reboots on Ryzen 7000/8000 using nested
VMs (vls suspected)
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoRunning nested VMs on AMD Ryzen 7000/8000 (ZEN4) CPUs results in random host's reboots.
There is no kernel panic, no log entries, no relevant output to serial console. It is as if platform is simply hard reset. It seems time to reproduce it varies from system to system and can be dependent on workload and even specific CPU model.
I can reproduce it with kernel 6.9.7 and qemu 9.0 on Ryzen 7950X3D under one hour by using KVM -> Windows 10/11 with Hyper-V services on or KVM -> Windows 10/11 with 3 VBox VMs (also Win11) running. Others people had it repeatedly reproduced on Ryzen 7700,7600 and 8700GE, including KVM -> KVM -> Linux.[1] I also have seen Hetzner (company offering Ryzen based dedicated servers) customers complaining about similiar random reboots.
I tried looking up errata for Ryzen 7000/8000, but could not find one published, so I decided to check errata for EPYC 9004 [2], which is also Zen4 arch as Ryzen 7000/8000. It has nesting related bug #1495 (on page 49), which mentions using Virtualized VMLOAD/VMSAVE can result in MCE and/or system reset.
Based on that errata mentioned above, I reconfigured my system with kvm_amd.vls=0 and for me random reboots with nested virtualization stopped. Same was reported by several people from [1].
Somebody from AMD must be asked to confirm if it is really Ryzen 7000/8000 hardware bug, and if there is a better fix than disabling VLS as it has performance hit. If disabling it is the only fix, then kvm_amd.vls=0 must be default for Ryzen 7000/8000.
Quoted 5 lines
[1]
https://www.reddit.com/r/Proxmox/comments/1cym3pl/nested_virtualization_crashing_ryzen_7000_series/
[2]
https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/revision-guides/57095-PUB_1_01.pdf
Hi!
Can someone from AMD take a look at this bug:
From the bug report it appears that recent Zen4 CPUs have errata in their virtual VMLOAD/VMSAVE implemenatation, which causes random host reboots (#MC?) when nesting is used, which is IMHO a quite serious issue.
Thanks,
Best regards,
Maxim Levitsky
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #23 from [email protected] ---
Quoted 50 lines · On Sat, 2024-07-06 at 11:20 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219009
Bug ID: 219009
Summary: Random host reboots on Ryzen 7000/8000 using nested
VMs (vls suspected)
Product: Virtualization
Version: unspecified
Hardware: AMD
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: NoRunning nested VMs on AMD Ryzen 7000/8000 (ZEN4) CPUs results in random host's
reboots.There is no kernel panic, no log entries, no relevant output to serial console.
It is as if platform is simply hard reset. It seems time to reproduce it varies
from system to system and can be dependent on workload and even specific CPU model.I can reproduce it with kernel 6.9.7 and qemu 9.0 on Ryzen 7950X3D under one hour by using KVM -> Windows 10/11 with Hyper-V services on or KVM -> Windows 10/11 with 3 VBox VMs (also Win11) running. Others people had it repeatedly reproduced on Ryzen 7700,7600 and 8700GE, including KVM -> KVM -> Linux.[1] I also have seen Hetzner (company offering Ryzen based dedicated servers) customers complaining about similiar random reboots.
I tried looking up errata for Ryzen 7000/8000, but could not find one published, so I decided to check errata for EPYC 9004 [2], which is also Zen4 arch as Ryzen 7000/8000. It has nesting related bug #1495 (on page 49), which mentions using Virtualized VMLOAD/VMSAVE can result in MCE and/or system reset.
Based on that errata mentioned above, I reconfigured my system with kvm_amd.vls=0 and for me random reboots with nested virtualization stopped. Same was reported by several people from [1].
Somebody from AMD must be asked to confirm if it is really Ryzen 7000/8000 hardware bug, and if there is a better fix than disabling VLS as it has performance hit. If disabling it is the only fix, then kvm_amd.vls=0 must be default for Ryzen 7000/8000.
Quoted 7 lines
[1]
https://www.reddit.com/r/Proxmox/comments/1cym3pl/nested_virtualization_crashing_ryzen_7000_series/
[2]
Hi!
Can someone from AMD take a look at this bug:
From the bug report it appears that recent Zen4 CPUs have errata in their virtual VMLOAD/VMSAVE implemenatation,
which causes random host reboots (#MC?) when nesting is used, which is IMHO a quite serious issue.
Thanks,
Best regards,
Maxim Levitsky
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #24 from Žilvinas Žaltiena ([email protected]) --- Disabling avic does not help. I and some other people tried that a few months ago.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #25 from h4ck3r ([email protected]) ---
I've recently talked to person which insisted they never had issues on their guests running on host cpu type without disabling vls.
From what i asked it seems that all guests were linix based and lacked pci passthrought (proxmox newest kernel as of time of this post)
Further testing is required.
I'm gonna spin up linux guest with approx half of host memory (no balooning) without any external device attached to see if stability can be archived that way.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #26 from [email protected] ---
But the question is - did they use nested virtualization on Linux actively and with vls enabled?
The use case which causes the reboots as I understand is Hyperv enabled Windows, in which case pretty much the whole Windows is running as a nested VM, nested to the Hyperv hypervisor.
Once I get my hands on a client Zen4 machine (I only have Zen2 at home), I will also try to reproduce this but not promises when this will happen.
Meanwhile I really hope that someone from AMD can take a look a this, and either confirm that this is or will be fixed with a microcode patch or confirm that we have to disable vls on the affected CPUs.
Best regards,
Maxim Levitsky
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #27 from Žilvinas Žaltiena ([email protected]) --- (In reply to blake from comment #13)
Quoted 6 lines
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it pretty easy to reproduce, just takes some time.
Blake, what OS is used in your VMs ?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #28 from blake ([email protected]) --- All of ours are running Debian Bookworm, stock kernel linux-image-6.1.0-26-amd64. And then hosts are 6.8.12-2-pve
Quoted 18 lines · On Tue, Oct 8, 2024, at 12:53 PM, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #27 from Žilvinas Žaltiena ([email protected]) --- (In reply to blake from comment #13)
Quoted 6 lines
I recently experienced this. I built a proxmox cluster with 7950x. Every node that I tested on would hard reset with no logs when a VM was doing nested virtualization.
Our CI testing uses VMs, and putting the CI in a VM itself makes it pretty easy to reproduce, just takes some time.
Blake, what OS is used in your VMs ?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are on the CC list for the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #29 from h4ck3r ([email protected]) --- (In reply to mlevitsk from comment #26)
Quoted 11 lines
But the question is - did they use nested virtualization on Linux actively and with vls enabled?
The use case which causes the reboots as I understand is Hyperv enabled Windows, in which case pretty much the whole Windows is running as a nested VM, nested to the Hyperv hypervisor.
Once I get my hands on a client Zen4 machine (I only have Zen2 at home), I will also try to reproduce this but not promises when this will happen.
Quoted 6 lines
Meanwhile I really hope that someone from AMD can take a look a this, and either confirm that this is or will be fixed with a microcode patch or confirm that we have to disable vls on the affected CPUs.
Best regards,
Maxim Levitsky
Not really - most of them are microservice type ones.
That would also mean there is less chance of corruption since they occupied less host memory.
And windows uses nested virt even if hyperv is not installed somehow. (installation does not, but freshly booted guest crashed my node)
Im afraid it might be unresolvable issue, even with microcode. At least most things point to similar issue as memory leaks on their igpus.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #30 from h4ck3r ([email protected]) --- I've also saw some memory related issues after overloading guest ram. For example after having to write to swap on windows guest i've seen pcie stutters which persisted even after freeing memory (v4 cpu type) Same thing happened when guest run for too long.
Thou that might be more of an vfio issue than anything else.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #31 from Ben Hirlston ([email protected]) ---
in my case I am on Arch Linux and I am running 6.11.2-zen1-1-zen the Zen Kernel. I need to disable vls for my VM to remain stable if I don't my system will randomly reboot after a random amount of time. disabling vls helps stop that but it makes nested virtualization stop working so I can't run WSL 2 in my Windows 11 guest. or Hyper V correctly
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #32 from [email protected] ---
This is very intersting - disabling VLS should not have any effect other than performance loss - KVM emulates VMLOAD/VMSAVE in this case, instead of hardware doing so.
@Ben Hirlston can you double check that with VLS WSL2 works, and witouth VLS it doesn't?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #33 from Ben Hirlston ([email protected]) ---
I wanna say it doesn't if I ever got it to launch it was probably defaulting to WSL1 as a fallback
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
isdennu ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #34 from isdennu ([email protected]) --- (In reply to h4ck3r from comment #29)
Quoted 19 lines
(In reply to mlevitsk from comment #26)
Quoted 11 lines
But the question is - did they use nested virtualization on Linux actively and with vls enabled?
The use case which causes the reboots as I understand is Hyperv enabled Windows, in which case pretty much the whole Windows is running as a nested VM, nested to the Hyperv hypervisor.
Once I get my hands on a client Zen4 machine (I only have Zen2 at home), I will also try to reproduce this but not promises when this will happen.
Quoted 6 lines
Meanwhile I really hope that someone from AMD can take a look a this, and either confirm that this is or will be fixed with a microcode patch or confirm that we have to disable vls on the affected CPUs.
Best regards,
Maxim Levitsky
Quoted 8 lines
Not really - most of them are microservice type ones.
That would also mean there is less chance of corruption since they occupied less host memory.
And windows uses nested virt even if hyperv is not installed somehow. (installation does not, but freshly booted guest crashed my node)Im afraid it might be unresolvable issue, even with microcode. At least most things point to similar issue as memory leaks on their igpus.
I have a 7900x on which I use igpu and RX7800XT as dgpu (use for AI work) and motherboard ASUS ProArt X670E-CREATOR. That being said, I actively use virtualization.
When disabling VLS and even disabling nested virtualization completely, my host kept rebooting unpredictably. That is, no advice from this thread helped me. As a result, I disabled igpu in BIOS and enabled nested virtualization and VLS. With this configuration the reboots continued. But after disabling VLS the reboots disappeared and now the PC works without fail.
I can assume that the problem with VLS and the use of igpu may be somehow related.
It would be very interesting to know if this problem is present on Ryzen 9000.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #35 from Ben Hirlston ([email protected]) ---
yeah I'm not using my iGPU in my 7900X3D at all I have a 7800 XT and a RTX 3060 that is setup to run with the vfio-pci driver on my MSI MPG X670E CARBON WIFI motherboard. and disabling VLS made my random reboots stop. so to hear that having an igpu enabled caused it to continue is interesting
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #36 from h4ck3r ([email protected]) ---
WSL and hyperv should still work with vls disabled on host cpu type. It will run much slower thou.
Since there is memory leak on am5 igpus they might contribute to reboots. That said i have igpu enabled in bios (not connected nor used) and never saw reboot with vls disabled.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Simon Labrecque ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #37 from Simon Labrecque ([email protected]) ---
Wow, this bug sent me to a weird path :P I've been using a Windows VM for the past 6 months, daily, without problem. Then suddently 2 days ago (didn't update the kernel or anything else), my PC started randomly rebooting when under load. Reading on problems with AMD cpus (and specifically 7950x), I bought a PSU and swapped it. Same problem. Then the cpu (for a 7600), then the motherboard, then the memory... same. I could 100% reproduce the problem by putting the VM under load within 3 minutes. AFAIK, there was no MCE logged.
I finally stumbled on this thread, set kvm_amd.vls=0, and that indeed fully fixed the problem.
I don't know how or why hyper-v was suddently enabled in my Windows VM, but that's obviously what happened. When it did, it triggered this bug quite violently.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Mario Limonciello (AMD) ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |CODE_FIX--- Comment #38 from Mario Limonciello (AMD) ([email protected]) --- Thanks everyone for your feedback and testing.
The following change will go into 6.12 and back to the stable kernels to fix this issue. It is essentially doing the same effect that kvm_amd.vls=0 did.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Sean ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #39 from Sean ([email protected]) ---
I've been encountering random reboots with a 7950X as well, oddly no nested virtualization. I just came across this via a phoronix article, and tried the kvm_amd.vls=0 argument I've hit my third reboot today after trying to amend the flag. I'm going to try bringing in the patch today.
I have two virtual machines I use. Neither is using nested virtualization
- Windows 11 with VFIO and pcie passthrough
- Linux guest with spice and opengl via virgl
Even with just the Linux guest it crashes
My main distribution is NixOs with 6.6. But I've tried Fedora, Arch and Ubuntu with 6.6,6.8.6.10 and 6.11. I've disable power saving, and a number of different tweaks.
I've also swapped memory, motherboard, disks and power supply. Updated bios, etc. The system started showing instability after I bumped to 6.X, and updating to the bios post voltage issues with the zen 4 cpus. I was on 5.1X for a while due to bugs on gpu initialization while using vfio.
There is a note on iGPU which I'm using, and experiencing a number of issues with. I won't go into that, and expect it's due to 6.6, which I'm locked to right now due to zfs.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #40 from Mario Limonciello (AMD) ([email protected]) --- It sounds like you have a separate stability issue, it should be brought into it's own bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #41 from Ben Hirlston ([email protected]) ---
I am testing Linux Zen 6.12 with the vls flag disabled and so far I haven't encountered a crash with WSL2 and Windows 7 in virtualbox in my Windows VM KVM/QEMU VM so I guess we will see if that changes with time
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #42 from Ben Hirlston ([email protected]) ---
to test this I fired up my KVM/QEMU Windows 11 VM and installed virtualbox in it and installed wsl2 to see if running virtual machines in KVM/QEMU would trigger the crash I've been testing for 20 minutes and have yet to see a crash
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #43 from Ben Hirlston ([email protected]) --- the way I obtained Linux Zen 6.12 was by going to the extra-testing repo on archlinux.org and running sudo pacman -U linux-zen-version linux-zen-headers-version to upgrade to those packages
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Ed Tomlinson ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #44 from Ed Tomlinson ([email protected]) ---
I applied the patch for this errata on Saturday, seven days later no resets. I am not using VMs. I do run Arch and their kernel is built with virtualization support and warns "kernel: Booting paravirtualized kernel on bare hardware". Wonder if somewhere in the kernel VMLOAD/VMSAVE gets used. The closest I come to running a VM is using an appimage.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
Christian Haefeli ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #45 from Christian Haefeli ([email protected]) --- (In reply to Mario Limonciello (AMD) from comment #38)
Quoted 7 lines
Thanks everyone for your feedback and testing.
The following change will go into 6.12 and back to the stable kernels to fix this issue. It is essentially doing the same effect that kvm_amd.vls=0 did.
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/
urgent&id=a5ca1dc46a6b610dd4627d8b633d6c84f9724ef0
Hello
I would prefer the more flexible solution via kvm_amd module parameter. I am not having this issue with an Epyc 4244p but am now losing performance due to this hard coded workaround. Does my CPU also qualify as a 'Zen4 Client SoC?
Regards
Christian
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #46 from Mario Limonciello (AMD) ([email protected]) ---
Quoted 2 lines
I would prefer the more flexible solution via
kvm_amdmodule parameter. I am not having this issue
The failure rate is low, but it's statistically significant. Feel free to revert it on your local kernel tree if you're not hitting the issue.
Quoted 1 line
due to this hard coded workaround.
FWIW If you have a fixed BIOS, you will have the same result.
Quoted 1 line
Does my CPU also qualify as a 'Zen4 Client SoC?
Yes it also affects any CPUs leveraged from client CPUs.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #47 from Christian Haefeli ([email protected]) --- (In reply to Mario Limonciello (AMD) from comment #46)
Quoted 6 lines
Quoted 2 lines
I would prefer the more flexible solution via
kvm_amdmodule parameter. I am not having this issueThe failure rate is low, but it's statistically significant. Feel free to revert it on your local kernel tree if you're not hitting the issue.
I do not have the time to tend to a local kernel tree... IMHO an extreme measure
in relation to a low failure rate. It looks like an after sales castration to me.
Quoted 4 lines
Quoted 2 lines
due to this hard coded workaround.
FWIW If you have a fixed BIOS, you will have the same result.
So you are saying that from a specific AGESA version onward, these instructions are being hidden from the OS? If yes, is this the same case for Zen5 based Client SoC SKUs?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #48 from Ben Hirlston ([email protected]) ---
I haven't had issues with this since 6.11 6.12 fixed this for me. for context I have an Ryzen 9 7900X3D
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #49 from Christian Haefeli ([email protected]) --- (In reply to Ben Hirlston from comment #48)
Quoted 2 lines
I haven't had issues with this since 6.11 6.12 fixed this for me. for context I have an Ryzen 9 7900X3D
Yes you most likely own an affected CPU. This in-kernel change fixes the stabiliy issue you have encountered but at the cost of performance. And for people like me, whose CPUs are not affected it is even worse. We lose performance for just nothing. IMHO the proper way would be if AMD would offer a CPU swap for affected customers.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #50 from h4ck3r ([email protected]) --- (In reply to Christian Haefeli from comment #49)
They should at least announce that issue globally.
I got new cpu specifficly for nested virt workloads. Turns out - that "patch" makes it so windows can't use nested virt. Its complaining about missing features.
Bigger issue - windows does not have patch applied afaik. This means that win host with hyperv enabled can crash anytime.
Back to linux - disabling vls does have performance penalty around 10-20% as far as i tested.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #51 from h4ck3r ([email protected]) --- I finally had time to test differences more clearly.
Cpus i used were 7700x and 9700x
Guest os is windows (linux didn't had most issues in the first place)
No nested virt warning with wsl still persists (probably windows being windows cause zen5 only guests don't have it)
Pcie "slowdown" is gone using zen5 (no performance penalty on gpu)
No issues with nvme drive passthrough (it would behave as broken hdd on zen4 having huge latency and increased temps for some reason - probably irq handling)
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #52 from Ben Hirlston ([email protected]) --- this issue seems to be back
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219009] Random host reboots on Ryzen 7000/8000 using nested VMs (vls suspected)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219009
--- Comment #53 from Mario Limonciello (AMD) ([email protected]) ---
Quoted 1 line
this issue seems to be back
If the issue was gone before and you're sure it's the same issue I suggest opening a new ticket and bisecting.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.