New: [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
Subsystems (2)
https://bugzilla.kernel.org/show_bug.cgi?id=219010
Bug ID: 219010
Summary: [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash
because of "Collect hot-reset devices to local buffer"
Product: Virtualization
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No
One of my virtual machines using PCI device passthrough (vfio) stopped working on OpenSuse Tumbleweed since kernel 6.9.7. Qemu 9.0.1 complains:
qemu-system-x86_64: vfio: hot reset info failed: No space left on device
qemu-system-x86_64: GLib: ../glib/gmem.c:177: failed to allocate 18446744068411217972 bytes
and then coredumps. Qemu backtrace shows vfio_pci_get_pci_hot_reset_info() being the last qemu function being called.
Reverting kernel 6.9.7 commit 9313244c26f3792daa86f3a18cc3bd5ad60310e0 (upstream f6944d4a0b87c16bc34ae589169e1ded3d4db08e) - "vfio/pci: Collect hot-reset devices to local buffer" fixes the problem. As I understand, that was backported to 6.9.7 from 6.10 tree.
Upon more throughout analysis I pinpointed that crash is happening because of one specific passed device: sound card of Asus B650 Creator motherboard. VM starts on 6.9.7 if I remove this sound card from it. I think the important bit is this card being VF of device which does not report support for FLR:
15:00.0 | iommu group 28 | Phoenix PCIe Dummy Function <-- not passed to VM, no driver, reset method: pm bus
15:00.2 | iommu group 29 | Encryption controller (PSP/CCP) <-- ccp driver 15:00.3 | iommu group 30 | USB controller <-- xhci_hcd driver 15:00.4 | iommu group 31 | USB controller <-- xhci_hcd driver 15:00.6 | iommu group 32 | HD Audio Controller <-- sound card passed to VM
After reverting the above mentioned commit, qemu complains:
vfio: Cannot reset device 0000:15:00.6, depends on group 28 which is not owned
exactly the same as before 6.9.7 and VM starts with that sound card passed.
This might be an unsupported configuration, but qemu crashing with 6.9.7 also feels like kernel might be breaking userspace by handling/mishandling this differently, especially with minor version change.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #1 from Žilvinas Žaltiena ([email protected]) --- Additional information: passing NVIDIA GPU, Samsung NVMEs works, passing Fresco
FL1100 based USB card does not work. Fresco card is single VF device, but like
that sound card it does not report FLR. Reverting "vfio/pci: Collect hot-reset devices to local buffer" allows to pass every mentioned device.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
The Linux kernel's regression tracker (Thorsten Leemhuis) ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #2 from The Linux kernel's regression tracker (Thorsten Leemhuis) ([email protected]) --- Does the problem happen with 6.10-rc6 or newer as well?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]Quoted 8 lines · On 2024/7/7 01:19, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #1 from Žilvinas Žaltiena ([email protected]) --- Additional information: passing NVIDIA GPU, Samsung NVMEs works, passing Fresco
FL1100 based USB card does not work. Fresco card is single VF device, but like
that sound card it does not report FLR. Reverting "vfio/pci: Collect hot-reset devices to local buffer" allows to pass every mentioned device.
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
From 93618efe933c4fa5ec453bddacdf1ca2ccbf3751 Mon Sep 17 00:00:00 2001
From: Yi Liu <[email protected]>
Date: Tue, 9 Jul 2024 06:41:02 -0700
Subject: [PATCH] vfio/pci: Fix a regresssion
--- drivers/vfio/pci/vfio_pci_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_core.c +1 -1
@@ -1260,7 +1260,7 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(struct vfio_pci_hot_reset_info hdr;struct vfio_pci_fill_info fill = {};bool slot = false; -int ret, count; +int ret, count =0 ;if (copy_from_user (&hdr, arg, minsz))return -EFAULT ;
Signature · 6 lines
2.34.1 -- Regards, Yi Liu
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #3 from Liu, Yi L ([email protected]) ---
Quoted 11 lines · On 2024/7/7 01:19, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #1 from Žilvinas Žaltiena ([email protected]) --- Additional information: passing NVIDIA GPU, Samsung NVMEs works, passing Fresco
FL1100 based USB card does not work. Fresco card is single VF device, but like
that sound card it does not report FLR. Reverting "vfio/pci: Collect hot-reset
devices to local buffer" allows to pass every mentioned device.
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
From 93618efe933c4fa5ec453bddacdf1ca2ccbf3751 Mon Sep 17 00:00:00 2001
From: Yi Liu <[email protected]>
Date: Tue, 9 Jul 2024 06:41:02 -0700
Subject: [PATCH] vfio/pci: Fix a regresssion
--- drivers/vfio/pci/vfio_pci_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0
b/drivers/vfio/pci/vfio_pci_core.c
index 59af22f6f826..0a7bfdd08bc7 100644
drivers/vfio/pci/vfio_pci_core.c +1 -1
@@ -1260,7 +1260,7 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info(struct vfio_pci_hot_reset_info hdr;struct vfio_pci_fill_info fill = {};bool slot = false; -int ret, count; +int ret, count =0 ;if (copy_from_user (&hdr, arg, minsz))return -EFAULT ;
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
Beld Zhang ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #4 from Beld Zhang ([email protected]) --- after manual modify source code:
testing pass, that crash is not occurs again.
nv 3060ti on dell precision T7920
kernel 6.6.38
qemu 8.2.4
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #5 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #3)
Quoted 4 lines
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
Patch fixes the problem on my system.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
Quoted 11 lines · On 2024/7/10 04:49, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #5 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #3)
Quoted 4 lines
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
Patch fixes the problem on my system.
patch submitted to mailing list. Thanks, and feel free to let me know if it is proper to add your reported-by, and add your tested-by.
Signature · 2 lines
Regards, Yi Liu
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #6 from Liu, Yi L ([email protected]) ---
Quoted 11 lines · On 2024/7/10 04:49, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #5 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #3)
Quoted 4 lines
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
Patch fixes the problem on my system.
patch submitted to mailing list. Thanks, and feel free to let me know if it is proper to add your reported-by, and add your tested-by.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to Yi LiuQuoted 10 lines · Yi Liu
Quoted 9 lines · On 2024/7/10 04:49, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #5 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #3)
Quoted 4 lines
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
Quoted 6 lines
Quoted 2 lines
Patch fixes the problem on my system.
patch submitted to mailing list. Thanks, and feel free to let me know if it is proper to add your reported-by, and add your tested-by.
forgot the link. :)
https://lore.kernel.org/kvm/[email protected]/T/#u
Signature · 2 lines
Regards, Yi Liu
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #7 from Liu, Yi L ([email protected]) ---
Quoted 10 lines · Yi Liu
Quoted 9 lines · On 2024/7/10 04:49, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #5 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #3)
Quoted 4 lines
It appears that the count is used without init.. And it does not happen with other devices as they have FLR, hence does not trigger the hotreset info path. Please try below patch to see if it works.
Quoted 6 lines
Quoted 2 lines
Patch fixes the problem on my system.
patch submitted to mailing list. Thanks, and feel free to let me know if it is proper to add your reported-by, and add your tested-by.
forgot the link. :)
https://lore.kernel.org/kvm/[email protected]/T/#u
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
--- Comment #8 from Žilvinas Žaltiena ([email protected]) --- (In reply to Liu, Yi L from comment #6)
Quoted 2 lines
patch submitted to mailing list. Thanks, and feel free to let me know if it is proper to add your reported-by, and add your tested-by.
It is ok to add me.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 219010] [REGRESSION][VFIO] kernel 6.9.7 causing qemu crash because of "Collect hot-reset devices to local buffer"
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=219010
Žilvinas Žaltiena ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |CODE_FIX--- Comment #9 from Žilvinas Žaltiena ([email protected]) --- Fixed in 6.9.10. Closing this.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.