New: USB devices plugged into USB hubs on 6.14.2-300.fc42.x86_64 do not work

https://bugzilla.kernel.org/show_bug.cgi?id=220016

            Bug ID: 220016
           Summary: USB devices plugged into USB hubs on
                    6.14.2-300.fc42.x86_64 do not work
           Product: Drivers
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: USB
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 307967
--> https://bugzilla.kernel.org/attachment.cgi?id=307967&action=edit
6.14.2-300.fc42.x86_64 journalctl -k output

Upgrading to 6.14.2 results in any USB device plugged into a USB hub to not work correctly. Downgrading to 6.13.7-200.fc41.x86_64 allows for all USB devices to work as expected. This issue also seems to be present in 6.15.0 as well.

Here is the downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=2353984
with more details. Another user on the RH BZ with a very similar motherboard is reporting that none of their USB devices are working when upgrading to 6.14.2

I have attached the journalctl -k output for 6.13.7-200.fc41.x86_64 and one thing that seems notable is:

Apr 14 20:06:49 kernel: ahci 0000:26:00.0: version 3.0
Apr 14 20:06:49 kernel: ahci 0000:26:00.0: probe with driver ahci failed with
error -12
Apr 14 20:06:49 kernel: ahci 0000:27:00.0: probe with driver ahci failed with
error -12
Apr 14 20:06:49 kernel: xhci_hcd 0000:23:00.0: init 0000:23:00.0 fail, -16
Apr 14 20:06:49 kernel: xhci_hcd 0000:23:00.0: probe with driver xhci_hcd
failed with error -16
Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.1: init 0000:28:00.1 fail, -16
Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.1: probe with driver xhci_hcd
failed with error -16
Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.3: init 0000:28:00.3 fail, -16
Apr 14 20:06:49 kernel: xhci_hcd 0000:28:00.3: probe with driver xhci_hcd

failed with error -16

and in the 6.13.7-200.fc41.x86_64 output, this is not present. I have attached the 6.13.7-200.fc41.x86_64 journalctl -k output as well and the lspci -v output is below.

# lspci -v|rg -A 15 USB
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0
Host Controller (prog-if 30 [XHCI])
        Subsystem: ASUSTeK Computer Inc. Device 8815
        Flags: bus master, fast devsel, latency 0, IRQ 94, IOMMU group 85
        Memory at f3600000 (64-bit, non-prefetchable) [size=1M]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Endpoint, IntMsgNum 0
        Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010
<?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [2a0] Access Control Services
        Capabilities: [370] Transaction Processing Hints
        Kernel driver in use: xhci_hcd
--
23:00.0 USB controller: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
(prog-if 30 [XHCI])
        Subsystem: ASUSTeK Computer Inc. Device 87c1
        Flags: bus master, fast devsel, latency 0, IRQ 65, IOMMU group 51
        Memory at f0600000 (64-bit, non-prefetchable) [size=32K]
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, IntMsgNum 0
        Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 0201
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [200] Secondary PCI Express
        Capabilities: [300] Latency Tolerance Reporting
        Capabilities: [400] L1 PM Substates
        Kernel driver in use: xhci_hcd

--
28:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI])
Subsystem: ASUSTeK Computer Inc. Device 8815
Flags: bus master, fast devsel, latency 0, IRQ 75, IOMMU group 50 Memory at f0100000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, IntMsgNum 0
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010
<?>
Kernel driver in use: xhci_hcd

28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller (prog-if 30 [XHCI])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device 148c
Flags: bus master, fast devsel, latency 0, IRQ 76, IOMMU group 50 Memory at f0000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, IntMsgNum 0
Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010
<?>
Kernel driver in use: xhci_hcd
--
2c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller (prog-if 30 [XHCI])
Subsystem: ASUSTeK Computer Inc. Device 8815
Flags: bus master, fast devsel, latency 0, IRQ 85, IOMMU group 62 Memory at f0700000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, IntMsgNum 0
Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010
<?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Capabilities: [370] Transaction Processing Hints
Kernel driver in use: xhci_hcd

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #1 from Joe Doss ([email protected]) --- Created attachment 307968
--> https://bugzilla.kernel.org/attachment.cgi?id=307968&action=edit
6.13.7-200.fc41.x86_64 journalctl -k output

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Artem S. Tashkinov ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

     Kernel Version|                            |6.14.2
         Regression|No                          |Yes

--- Comment #2 from Artem S. Tashkinov ([email protected]) --- Would be great if you could bisect:

https://docs.kernel.org/admin-guide/bug-bisect.html

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Michał Pecio ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Michał Pecio ([email protected]) --- What if you disconnect the hub and connect an ordinary device in its place? This log looks like several USB ports on your computer should be completely dead. And some SATA ports too.

This looks like devm_request_mem_region() is failing in usb_hcd_pci_probe(), and maybe pcim_iomap() in ahci (SATA). So something to do with PCI, ACPI or other lower level stuff.

You may try your luck reassigning this bug to drivers/PCI, they seem to be responsive to bugzilla reports.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Joe Doss ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
Component|USB |PCI

--- Comment #4 from Joe Doss ([email protected]) --- (In reply to Michał Pecio from comment #3)

I just swapped my mouse into the USB A 3.0 port that my hub was in and it is dead too.

OK sounds good. I will reassign this to drivers/PCI to see if those folks have any ideas what is going on here.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Steven A. Falco ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #5 from Steven A. Falco ([email protected]) --- Created attachment 307975
--> https://bugzilla.kernel.org/attachment.cgi?id=307975&action=edit
Good boot with 6.13 kernel

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #6 from Steven A. Falco ([email protected]) --- Created attachment 307976
--> https://bugzilla.kernel.org/attachment.cgi?id=307976&action=edit
Bad boot with 6.14 kernel

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #7 from Steven A. Falco ([email protected]) ---
I appear to be having the same problem, and I've attached journal logs from a good boot and a bad boot. In my case I have a mouse and keyboard directly plugged into the motherboard.

The keyboard was not working once I upgraded to kernel 6.14 but the mouse was working. I stepped back to kernel 6.13, and both mouse and keyboard were working.

I tried switching which connectors the mouse and keyboard were plugged into, and at that point the mouse wasn't working but the keyboard was working.

So with kernel 6.14 several of the motherboard jacks were not functional.

I looked at the pci information, and the usb devices are:

usb1/2 0000:23:00.0 ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller usb3/4 0000:28:00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
usb5/6 0000:28:00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
usb7/8 0000:2d:00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
usb9/10 0000:05:00.3 Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

Apparently, with 6.14 one of the Matisse controllers was recognized, but the other was not. And neither of the Starship controllers were recognized.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #8 from Steven A. Falco ([email protected]) ---
I am now able to build a working 6.14 kernel. The only thing I had to change was to turn off CONFIG_PCI_REALLOC_ENABLE_AUTO.

With CONFIG_PCI_REALLOC_ENABLE_AUTO off, all my USB controllers work.

With CONFIG_PCI_REALLOC_ENABLE_AUTO=y, some of my USB controllers don't initialize.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #9 from Joe Doss ([email protected]) --- Setting pci=realloc=off in the kernel cmdline works as well.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #10 from Steven A. Falco ([email protected]) --- The question now is whether this is a kernel bug or a motherboard bug.

If it is a mobo bug then I guess a quirk would be appropriate.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Ilpo Järvinen ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
                   |                            |om

--- Comment #11 from Ilpo Järvinen ([email protected]) --- Does also 6.13 fail if pci=realloc is on the kernel command line? It's not clear to me if PCI resource reallocation got enabled along with the kernel upgrade, that is, if Fedora packaging e.g. decided to enable CONFIG_PCI_REALLOC_ENABLE_AUTO in kernel's .config which is orthogonal to kernel version changes.

Please provide /proc/iomem from both working and non-working configuration. Preferrably with the otherwise identical kernel version, if you can't get it from the non-working configuration at all due to lack of keyboard, I'll manage with the log from working only (things can be inferred from dmesg but it's very very tedious process).

Could you also clarify what you meant with this: "This issue also seems to be present in 6.15.0 as well."? Kernel version 6.15 is not released yet, 6.15 cycle is only at -rc2 at this point.

I'll try to look more into this next week.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #12 from Steven A. Falco ([email protected]) --- Created attachment 307986
--> https://bugzilla.kernel.org/attachment.cgi?id=307986&action=edit
6.13.10 with realloc=on, no keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #13 from Steven A. Falco ([email protected]) --- Created attachment 307987
--> https://bugzilla.kernel.org/attachment.cgi?id=307987&action=edit
6.13.10 with realloc=off, good keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #14 from Steven A. Falco ([email protected]) --- Created attachment 307988
--> https://bugzilla.kernel.org/attachment.cgi?id=307988&action=edit
6.14.2 with realloc=on, no keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #15 from Steven A. Falco ([email protected]) --- Created attachment 307989
--> https://bugzilla.kernel.org/attachment.cgi?id=307989&action=edit
6.14.2 with realloc=off, good keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #16 from Steven A. Falco ([email protected]) --- I ran the tests you requested. I attached /proc/iomem for 6.13.10 both with realloc=off and realloc=on. I repeated the tests with 6.14.2.

For both kernel versions, realloc=on results in a non-functional USB keyboard.

For both kernels, realloc=off works properly.

I had been running Fedora 41 with their 6.13 kernel. It was compiled with realloc=off.

I then upgraded to Fedora 42 with their 6.14 kernel. It was compiled with realloc=on.

Therefore, this is not a new problem with 6.14; it is just that the kernel option was changed in Fedora 42.

I cannot comment about 6.15 but I think Joe was the one who tried it. Perhaps he can comment on that. However, since both 6.13 and 6.14 don't work if realloc=on, it probably doesn't matter.

Please let me know if there is any other data you need.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #17 from Steven A. Falco ([email protected]) --- I just realized that I need to read /proc/iomem as root. I'll recreate the attachments - please give me a few minutes.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Steven A. Falco ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #307986|0                           |1
        is obsolete|                            |
 Attachment #307987|0                           |1
        is obsolete|                            |
 Attachment #307988|0                           |1
        is obsolete|                            |
 Attachment #307989|0                           |1
        is obsolete|                            |

--- Comment #18 from Steven A. Falco ([email protected]) --- Created attachment 307990
--> https://bugzilla.kernel.org/attachment.cgi?id=307990&action=edit
6.13.10 with realloc=on, no keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #19 from Steven A. Falco ([email protected]) --- Created attachment 307991
--> https://bugzilla.kernel.org/attachment.cgi?id=307991&action=edit
6.13.10 with realloc=off, good keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #20 from Steven A. Falco ([email protected]) --- Created attachment 307992
--> https://bugzilla.kernel.org/attachment.cgi?id=307992&action=edit
6.14.2 with realloc=on, no keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #21 from Steven A. Falco ([email protected]) --- Created attachment 307993
--> https://bugzilla.kernel.org/attachment.cgi?id=307993&action=edit
6.14.2 with realloc=off, good keyboard

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #22 from Steven A. Falco ([email protected]) --- I updated the attachments.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #23 from Joe Doss ([email protected]) --- (In reply to Ilpo Järvinen from comment #11)

I was just trying the kernels that were built on Fedora's build system to see if the issues persisted with different versions. Steven is correct that this doesn't really matter at this point.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #24 from Ilpo Järvinen ([email protected]) --- Created attachment 308020
--> https://bugzilla.kernel.org/attachment.cgi?id=308020&action=edit
Add min_bridge_window cmdline parameter

It seems the FW left zero extra tail room in the root iomem windows which makes it a bit hard for pci=realloc to work. It doesn't work well with the default sizing algorithm that is greedy and tries to reserves more space than is really required.

Maybe the attached patch helps more devices to assign their resources:

So please test with pci=realloc,min_bridge_window

It could still leads to a failure or some failures because it seems there simply is not enough space to fit everything even without pci=realloc reserving that extra empty space.

Please also add dyndbg="file drivers/pci/*.c +p" on kernel cmdline to get a bit more information.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #25 from Steven A. Falco ([email protected]) --- Created attachment 308021
--> https://bugzilla.kernel.org/attachment.cgi?id=308021&action=edit
Test with min_bridge_window patch

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #26 from Steven A. Falco ([email protected]) --- I've run the experiment you requested and attached a log file. This did not help with the problem - I still had no keyboard - but hopefully the extra logging will give you more information.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

Justin M. Forbes ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #27 from Justin M. Forbes ([email protected]) --- For some background, CONFIG_PCI_REALLOC_ENABLE_AUTO was turned on for the first time with the 6.14 kernels in Fedora, prior to those it was off. I am turning
it off again because it seems to cause a number of issues. That said, I do
think those issues need to be fixed, so while new kernels (6.14.5+ and 6.15-rc4+) in fedora will not reproduce this problem, it is likely worth debugging with realloc=on

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #28 from Steven A. Falco ([email protected]) --- Thanks Justin. I'll continue to work with Ilpo if there are any proposed fixes or additional tests that I can help with.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #29 from Ilpo Järvinen ([email protected]) --- Created attachment 308058
--> https://bugzilla.kernel.org/attachment.cgi?id=308058&action=edit
Resource reset logging patch (only for 6.15+ kernels)

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #30 from Ilpo Järvinen ([email protected]) --- Justin, sure, my intention is to try to fix these issues now that we've becomes aware of them. I'm just not sure if that can be done for pre-6.15 kernels as touching the resource fitting / assignment algorithm is pretty precarious.

Steven, could you try with 6.15-rc based kernels with that min bridge window patch + pci=realloc,min_bridge_window? Please also include the reset debugging patch I just attached and dyndbg on the command line. There are number of things in the 6.14 log I don't understand and I suspect I might have fixed at least one of those in my recent changes to the resource fitting algorithm.

The failure to shrink bridge window size for 0000:22:02 from 0x500000 to 0x100000 as IOV resources should be optional, is particularly perplexing. Instead, the non-64bit windows vanish from many bridges after bridge windows are released, which could be due to improper reset of a resource after one of the steps which the later steps fail to undo for some reason.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #31 from Steven A. Falco ([email protected]) --- Created attachment 308061
--> https://bugzilla.kernel.org/attachment.cgi?id=308061&action=edit
Test with 6.15 rc plus two patches

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220016

--- Comment #32 from Steven A. Falco ([email protected]) --- I ran the experiment and attached the journal log.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.