New: xhci_hcd: System lockup under CPU load during rapid usbfs polling of SuperSpeed root hubs on AMD Ryzen platforms

https://bugzilla.kernel.org/show_bug.cgi?id=221103

            Bug ID: 221103
           Summary: xhci_hcd: System lockup under CPU load during rapid
                    usbfs polling of SuperSpeed root hubs on AMD Ryzen
                    platforms
           Product: Drivers
           Version: 2.5
    Kernel Version: 6.12 - 6.19, (and mainline)
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: USB
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 309401
--> https://bugzilla.kernel.org/attachment.cgi?id=309401&action=edit
Test program that polls USB nodes and triggers the system freeze

On multiple AMD Ryzen 7000/9000-series systems (X670E/X870 motherboards with recent AGESA firmware) rapidly polling xHCI device nodes via usbfs can cause a complete system freeze.

Reproduction:
1. Compile/run the attached usb_poll.c
2. Create CPU load (stress-ng --cpu 0 works reliably)
3. On affected machines the system freezes within a few minutes without any log

The trigger on my system is consistently one of the SuperSpeed (USB 3.x 10 Gbps) xHCI root hubs (1d6b:0003). I cannot confirm if other affected users freeze on exactly the same device node, but all reports point to the same class bug polling xHCI.

Kernels affected:
6.12 - 6.18 (LTS + mainline) on openSUSE Tumbleweed, Manjaro, Ubuntu, Gentoo, Bazzite, Fedora, etc.

I can provide any additional diagnostics needed:
- Full lsusb -t -v and lsusb -vvv
- lspci -vvv

My MB and CPU, others in the ADB bug report have different AMD systems:
- Motherboard: Asus ProArt X870-E CREATOR
- CPU: AMD Ryzen 9 9950X

Originally exposed by ADB’s USB backend (platform-tools ≥ 36.0.2):
https://issuetracker.google.com/issues/472398009 (full discussion with many affected users)

Tests:

test: Poll all USB devices
method: usb_poll.c
iterations: 600
result: SYSTEM FREEZE opening Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

test: Skip all root hubs (Device 001 on all busses)
method: usb_poll.c
iterations: 2400
result: SUCCESS

test: Only root hubs
method: usb_poll.c
iterations: 159
result: SYSTEM FREEZE opening Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

test: All devices except Bus 010 Device 001
method: usb_poll.c
iterations: 1900
result: SUCCESS

test: Only the specific xHCI root hub
method: usb_poll.c
iterations: 209
result: SYSTEM FREEZE

test: Only the specific xHCI root hub
method: while true; do sudo lsusb -v -s 010:001 >/dev/null; sleep 0.1; done
iterations: 2400
result: SUCCESS

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

Michał Pecio ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Michał Pecio ([email protected]) ---
I tried running this and all ioctl() fail with EPERM because the usbfs file is opened read only. If it really crashes your system, the bug may be not in USB.

Chances are that it's a panic so I'd try to get the kernel log - serial cable, netconsole or something like that. At the very least, connect a PS/2 keyboard and see if the lights begin to blink.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #2 from Paul Alesius ([email protected]) --- Try this command: sudo ./usb_poll

It is definitely in USB.
This program triggers the system freeze reliably. Many of us are affected, as referenced in the ADB bug.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #3 from Paul Alesius ([email protected]) --- Sorry, you may be right that the bug is elsewhere. I just triggered it again without O_RDWR using a program that also logs the exact point of the freeze.

1. It's always on opening the same USB root hub on my machine.
2. The freeze always occurs on the " Opening device..."
3. The ioctl seems to complete on the device where the freeze occurs.

PS/2 - We don't have PS/2 ports on these MBs
SysRq - I don't have that on my Logitech Mechanical keyboard

I just tried with "dmesg -W" (follow mode so it prints messages directly) and it didn't print anything before the system freeze, and there's nothing in the logs, reflecting the experience of the others in the ADB bug report.

Here's the last logged line from the program before the freeze:

Iteration 189 complete — attempted 9 devices
------------------------------------------------
=== Starting iteration 190 ===
Processing bus 011
Device: /dev/bus/usb/011/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 010
Device: /dev/bus/usb/010/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 009
Device: /dev/bus/usb/009/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 008
Device: /dev/bus/usb/008/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 007
Device: /dev/bus/usb/007/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 006
Device: /dev/bus/usb/006/001
Opening device...
Open failed: Invalid argument
Processing bus 005
Device: /dev/bus/usb/005/001
Opening device...
Open failed: Invalid argument
Processing bus 004
Device: /dev/bus/usb/004/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 003
Device: /dev/bus/usb/003/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 002
Device: /dev/bus/usb/002/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 001
Device: /dev/bus/usb/001/004
Opening device...
Skipping non-root hub device
Device: /dev/bus/usb/001/003
Opening device...
Skipping non-root hub device
Device: /dev/bus/usb/001/002
Opening device...
Skipping non-root hub device
Device: /dev/bus/usb/001/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd

Iteration 190 complete — attempted 9 devices
------------------------------------------------
=== Starting iteration 191 ===
Processing bus 011
Device: /dev/bus/usb/011/001
Opening device...
Opened fd=6
Issuing USBDEVFS_CONTROL ioctl (GET_DESCRIPTOR)... ioctl completed
Closed fd
Processing bus 010
Device: /dev/bus/usb/010/001
Opening device...

The C program used that only tries root hubs, while the system is under load with "stress-ng --cpu 0":

#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/usbdevice_fs.h>
#include <string.h>
#include <limits.h>
#include <errno.h>
#include <stdarg.h>

static FILE *logfile = NULL;

void log_and_sync(const char *format, ...)

{
    va_list args;
    va_start(args, format);
    vprintf(format, args);
    va_end(args);
    fflush(stdout);

    if (logfile) {
        va_list args_copy;
        va_start(args, format);
        va_copy(args_copy, args);
        vfprintf(logfile, format, args_copy);
        va_end(args_copy);
        va_end(args);
        fflush(logfile);
        fsync(fileno(logfile));
    }
}

void enumerate_usb() {
    static int iteration = 0;
    int device_count = 0;

    log_and_sync("=== Starting iteration %d ===\n", iteration + 1);

    DIR *usb_dir = opendir("/dev/bus/usb");
    if (!usb_dir) {
        log_and_sync("Error: opendir /dev/bus/usb failed: %s\n",
strerror(errno));
        return;
    }

    struct dirent *bus_entry;
    while ((bus_entry = readdir(usb_dir)) != NULL) {
        if (bus_entry->d_type != DT_DIR || bus_entry->d_name[0] == '.')

continue;

        char bus_path[PATH_MAX];
        snprintf(bus_path, sizeof(bus_path), "/dev/bus/usb/%s",

bus_entry->d_name);

        log_and_sync("Processing bus %s\n", bus_entry->d_name);

        DIR *bus_dir = opendir(bus_path);
        if (!bus_dir) {
            log_and_sync("  Warning: opendir %s failed: %s\n", bus_path,
strerror(errno));
            continue;
        }

        struct dirent *dev_entry;
        while ((dev_entry = readdir(bus_dir)) != NULL) {
            if (dev_entry->d_name[0] == '.') continue;

            char dev_path[PATH_MAX];
            snprintf(dev_path, sizeof(dev_path), "%s/%s", bus_path,

dev_entry->d_name);

            log_and_sync("  Device: %s\n", dev_path);
            log_and_sync("    Opening device...\n");

                        // Skip root hubs (usually /001)
                        if (strcmp(dev_entry->d_name, "001") != 0) {
                        log_and_sync("    Skipping non-root hub device\n");
                        continue;
                        }

            int fd = open(dev_path, O_RDONLY);
            if (fd < 0) {
                log_and_sync("    Open failed: %s\n", strerror(errno));
                continue;
            }

            log_and_sync("    Opened fd=%d\n", fd);

            unsigned char desc[18];
            struct usbdevfs_ctrltransfer ctrl = {
                .bRequestType = 0x80,
                .bRequest     = 6,      // GET_DESCRIPTOR
                .wValue       = 1 << 8, // Device descriptor
                .wIndex       = 0,
                .wLength      = sizeof(desc),
                .data         = desc,
                .timeout      = 1000
            };

            log_and_sync("    Issuing USBDEVFS_CONTROL ioctl

(GET_DESCRIPTOR)...\n");

ioctl(fd, USBDEVFS_CONTROL, &ctrl); // Errors ignored, as in
original

            log_and_sync("    ioctl completed\n");

            close(fd);
            log_and_sync("    Closed fd\n");

            device_count++;
        }
        closedir(bus_dir);
    }
    closedir(usb_dir);

    iteration++;
    log_and_sync("\nIteration %d complete — attempted %d devices\n", iteration,
device_count);
    log_and_sync("------------------------------------------------\n");
}

int main() {
    logfile = fopen("usb_poll6_roothubs.log", "w");
    if (!logfile) {
        perror("Failed to open usb_poll.log for writing (will continue without
file logging)");
    }

    log_and_sync("USB usbfs polling test started (1-second interval)\n");
    log_and_sync("You should now see device paths from all buses.\n");
    log_and_sync("On affected systems, freeze typically occurs within

minutes.\n");
log_and_sync("Detailed logging with fsync() to 'usb_poll.log' for
post-freeze analysis.\n");
log_and_sync("If freeze occurs during ioctl, you should see 'Issuing ...'
but not 'ioctl completed' for that device.\n\n");

    while (1) {
        enumerate_usb();
        //usleep(1000000);  // 1 second
                usleep(500000); // 0.5s
    }

    if (logfile) fclose(logfile);
    return 0;
}

Is there anything else that you can suggest to get further diagnosis?

Thank you

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Works just fine for me here, perhaps this is a bug in your USB device (or hub) that can not handle USB config requests?

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #4 from Greg Kroah-Hartman ([email protected]) ---

Works just fine for me here, perhaps this is a bug in your USB device (or hub) that can not handle USB config requests?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #5 from Michał Pecio ([email protected]) --- Actually, it may still be a problem with xhci_hcd because I found that simply opening the USBFS file causes some xhci_hcd debug logs.

But this ioctl() looks like meaningless noise, the way you call it it's always supposed to fail, either with or without root.

Does your system still crash if you remove the ioctl() and simply open and close the usbfs file?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #6 from Paul Alesius ([email protected]) ---

(or hub) that can not handle USB config requests?

I don't know. It seems to be related to boards with X670 or X870 chipset and Ryzen/Threadripper. From what I can gather from ADB report comments:

Asus Strix X670E-F Gaming BIOS 3402
Asus PRIME X670E-PRO WIFI
Asus ProArt X870-E (My board)
Dell Precision 7875

But since it triggers so reliably on that specific root hub on my machine, it made me suspect that it's in the kernel. The machine has been restarted many times and it's still while opening that specific root hub, which I think are created by the XHCI_hcd if I'm not mistaken.

I will try removing the ioctl now

Thank you

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #7 from Michał Pecio ([email protected]) --- (In reply to Paul Alesius from comment #3)

That's a waste of time, if the system locks up completely or panics then your dmesg will never get a chance to run again and print anything. Neither will Xorg or wayland.

This needs a serial cable or netconsole (never tried it myself but AFAIK people use it for panics too) or DRM_PANIC or something like that.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #8 from Michał Pecio ([email protected]) --- Or perhaps simply this would crash it?

while sleep .1 ; do true </dev/bus/usb/010/001 ; done

Chances are that sleep isn't necessary. This seems equivalent to what you are doing and it triggers some code in xhci_hub.c, but no crash so far on my AM4 system.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #9 from Paul Alesius ([email protected]) ---

Yes it froze the system on iteration 3000

A C program without ioctl also froze it on iteration 887. Doing ioctl seems to trigger it much faster at iteration ~200.

I ordered a USB-to-ethernet adapter in the meantime to try to get more diagnostics through netconsole, instead of serial cable, will that work?

Or do you have enough information about what may be wrong so that I can cancel the USB adapter order? Otherwise, it will arrive in 4-5 days. (USB-to-Ethernet because I don't have a laptop with RJ-45 from my PC.)

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

If the USB stack is locking up, dumping USB data out when that happens would obviously not be possible :)

There is a special USB debug cable that will work instead, search around on the web for one if you are curious about using it.

good luck!

greg k-h

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #10 from Greg Kroah-Hartman ([email protected]) ---

If the USB stack is locking up, dumping USB data out when that happens would obviously not be possible :)

There is a special USB debug cable that will work instead, search around on the web for one if you are curious about using it.

good luck!

greg k-h

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #11 from Paul Alesius ([email protected]) --- Oh worry I meant:

The machine where it locks up is a PC, it has ethernet in the motherboard that locks up

My mobile devices like laptop, only have USB, so I thought I'd try with a cheap ethernet dongle on them instead of getting a specific Asus serial header that cost much more

Thank you

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #12 from Michał Pecio ([email protected]) --- Isn't the required serial cable identical as in 99% of motherboards? ASUS is usually decent enough to show all details, including pinout, in the user manual.

Maybe one thing worth trying is to enable dynamic debug and run an iteration or a few to see what gets logged in dmesg, check if it changes or is always the same and compare the "bad" root hub with good ones, particularly of the same speed.

echo 'module xhci_hcd +p' >/proc/dynamic_debug/control dmesg -W

The most interesting information may be missing if it actually crashes, but maybe something will hint at any unusual patterns in your system.

One thing I see is that this code tries to check status of the ports, so your crashes may (or may not) depend on some particular device being connected.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #13 from Paul Alesius ([email protected]) --- (In reply to Michał Pecio from comment #12)

I just triggered the system freeze three times with dynamic debug and dmesg -W. Each time, the last message shown in the console is:

"Setting command ring address to 0x10e427001"

Dmesg logs "Setting command ring address" twice, and all three freezes occured after the first of the two log records of "Setting command address"

Demsg example below, it did not flush to disk so the "dmesg -W" to the terminal has newer log records but this is what I see in the console:

------ dmesg -W

...

[ 1139.446048] xhci_hcd 0000:7a:00.4: xhci_suspend: stopping usb9 port polling.
[ 1139.446066] xhci_hcd 0000:7a:00.4: Setting command ring address to
0x10e427001
[ 1139.561526] xhci_hcd 0000:7a:00.4: Setting command ring address to
0x10e427001
[ 1139.562304] xhci_hcd 0000:7a:00.4: xhci_resume: starting usb9 port polling.
[ 1139.562306] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb10 port
polling
[ 1139.562308] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb9 port
polling
[ 1139.585922] xhci_hcd 0000:7a:00.4: Get port status 10-1 read: 0x2a0, return
0x2a0
[ 1139.585931] xhci_hcd 0000:7a:00.4: Get port status 10-2 read: 0x2a0, return
0x2a0
[ 1139.792600] xhci_hcd 0000:7a:00.4: set port remote wake mask, actual port
10-1 status  = 0xe0002a0
[ 1139.792619] xhci_hcd 0000:7a:00.4: set port remote wake mask, actual port
10-2 status  = 0xe0002a0
[ 1139.792662] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb10 port
polling
[ 1139.792667] xhci_hcd 0000:7a:00.4: config port 10-1 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1139.792683] xhci_hcd 0000:7a:00.4: config port 10-2 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1139.792705] xhci_hcd 0000:7a:00.4: config port 9-1 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1139.792726] xhci_hcd 0000:7a:00.4: config port 9-2 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1139.792727] xhci_hcd 0000:7a:00.4: xhci_suspend: stopping usb9 port polling.
[ 1139.792746] xhci_hcd 0000:7a:00.4: Setting command ring address to
0x10e427001
[ 1139.821295] xhci_hcd 0000:7a:00.4: Setting command ring address to
0x10e427001
[ 1139.822070] xhci_hcd 0000:7a:00.4: xhci_resume: starting usb9 port polling.
[ 1139.822072] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb10 port
polling
[ 1139.822074] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb9 port
polling
[ 1139.845924] xhci_hcd 0000:7a:00.4: Get port status 10-1 read: 0x2a0, return
0x2a0
[ 1139.845931] xhci_hcd 0000:7a:00.4: Get port status 10-2 read: 0x2a0, return
0x2a0
[ 1140.052603] xhci_hcd 0000:7a:00.4: set port remote wake mask, actual port
10-1 status  = 0xe0002a0
[ 1140.052621] xhci_hcd 0000:7a:00.4: set port remote wake mask, actual port
10-2 status  = 0xe0002a0
[ 1140.052664] xhci_hcd 0000:7a:00.4: xhci_hub_status_data: stopping usb10 port
polling
[ 1140.052670] xhci_hcd 0000:7a:00.4: config port 10-1 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1140.052685] xhci_hcd 0000:7a:00.4: config port 10-2 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1140.052706] xhci_hcd 0000:7a:00.4: config port 9-1 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1140.052727] xhci_hcd 0000:7a:00.4: config port 9-2 wake bits, portsc:
0xa0002a0, write: 0xa0202a0
[ 1140.052728] xhci_hcd 0000:7a:00.4: xhci_suspend: stopping usb9 port polling.
[ 1140.052747] xhci_hcd 0000:7a:00.4: Setting command ring address to

0x10e427001

-- On plugged-in USB devices

There are 3 devices plugged in to USB, two are internal:

noname@devbox ~ $ sudo lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 13d3:3588 IMC Networks Wireless_Device (INTERNAL M.2 WiFi & Bluetooth)
Bus 001 Device 003: ID 0b05:19af ASUSTek Computer, Inc. AURA LED Controller (INTERNAL)
Bus 001 Device 004: ID 046d:c548 Logitech, Inc. Logi Bolt Receiver (I have this one plugged in for mouse and keyboard)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I changed the port of the Logitech Bolt Receiver during these three tests with system freeze, and plugged it into a USB 2.0 port, but the system still freezes on that root hub.

---------

If there's anything else to try to get more diagnostics, I can do that too, otherwise we'll have to wait until I receive the adapter for netconsole.

I am unable to remove the M.2 wifi card at present because I need to disassemble the entire computer and take out the MB because it's behind shielding covers that are not meant to be removed.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #14 from Alan Stern ([email protected]) --- Have you tried disabling runtime PM? Run any of the test scenarios after booting with "usbcore.autosuspend=-1" on the boot command line.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #15 from Paul Alesius ([email protected]) --- (In reply to Alan Stern from comment #14)

Disabling runtime PM for USB has eliminated the freeze, tested with hammering all devices too for 40 minutes.

---- Other logs:

With runtime PM enabled, I see this in dmesg on every boot:

[   17.918387] xhci_hcd 0000:78:00.0: WARN: xHC CMD_RUN timeout
[   17.918508] xhci_hcd 0000:78:00.0: PM: suspend_common(): xhci_pci_suspend
returns -110
[   17.918586] xhci_hcd 0000:78:00.0: can't suspend (hcd_pci_runtime_suspend

returned -110)

But 0000:78:00.0 is a different controller than the root hub that triggers the freeze. Here's lsusb with controller information:

0000:0e:00.0 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub 0000:0e:00.0 Bus 001 Device 002: ID 13d3:3588 IMC Networks Wireless_Device (Internal)
0000:0e:00.0 Bus 001 Device 003: ID 0b05:19af ASUSTek Computer, Inc. AURA LED Controller (Internal)
0000:0e:00.0 Bus 001 Device 004: ID 046d:c548 Logitech, Inc. Logi Bolt Receiver (Plugged in)
0000:0e:00.0 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub 0000:10:00.0 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub 0000:10:00.0 Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub 0000:78:00.0 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub (xhci_pci_suspend -110)
0000:78:00.0 Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub (xhci_pci_suspend -110)
0000:7a:00.3 Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub 0000:7a:00.3 Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub 0000:7a:00.4 Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub 0000:7a:00.4 Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub (Triggers freeze with runtime PM)
0000:7b:00.0 Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #16 from Michał Pecio ([email protected]) --- Would this fix it as well?

echo 'on' > /sys/bus/pci/devices/<pci-id>/power/control

It seems that this must be set to 'auto' to trigger those suspend-resume cycles of the host controller, and it is quite possible that the problem is caused by suspending the HC, not any USB device.

That being said, I toggled it to 'auto' on a few HCs currently plugged into my test system (was 'on' for all of them) and your test program (with failing ioctl() included) still runs just fine. It's at 3000 iterations now.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #17 from Paul Alesius ([email protected]) --- (In reply to Michał Pecio from comment #16)

I am now able to use netconsole to capture the logs during the freeze. This is all that is logged when the bug occurs and the system freezes:

cat netconsole-2026-02-23-1819.log-output-during-bug

[  549.009867] xhci_hcd 0000:7a:00.4: Controller not ready at resume -19
[  549.009891] xhci_hcd 0000:7a:00.4: PCI post-resume error -19!
[  549.009894] xhci_hcd 0000:7a:00.4: HC died; cleaning up

And then I tried with control=on on the affected device (/sys/bus/pci/devices/0000:7a:00.4/power/control), it seems to have eliminated the bug and system freeze. Control defaults to auto on the affected device as you said.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #18 from Michał Pecio ([email protected]) ---
My test got to almost 30k iterations with suspend-resume cycles, no crashes. I loaded the CPUs with simple "while (1);" on each core.

It seems the problem may be caused by a failure to resume the HC. The driver should handle it, but something goes wrong. I hoped it would be a stupid null pointer dereference and netconsole would give us stack trace, but no luck apparently. I tried crashing my system with SysRq-C and netconsole (over a PCIe Ethernet NIC) delivered the panic message just fine, so not sure what happens here.

Could you enable dynamic debug and check if simply toggling power/control between 'on' and 'auto' produces the same xhci_suspend/xhci_resume messages? Would this be enough to hang the system, or is it necessary that somebody waits for the HC to resume to perform some operations on the root hub? Maybe it's just lack of necessary checks for dead HC somewhere.

What's the state of power/control for those HCs which aren't causing problems? Are they also getting resumed and suspended under your test, but without crashing? That would be at least one optimistic result in this whole mess :)

There is another bug 221073 about some AMD HCs dying on resume from system sleep, may be related. So far nobody knows why it happens.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #19 from Paul Alesius ([email protected]) --- (In reply to Michał Pecio from comment #18)

Enabling dynamic debug and changing power/control on/auto rapidly produces the same suspend/resume messages on all devices. Changing control= on and auto rapidly on 0000:7a:00.4 does not trigger the freeze.

About half of them are on and auto, those with control=auto by default do not trigger the freeze (except the known-bad 7a:00.4, and I've not stressed the others as much until arriving at the conclusion that it's 7a:00.4 triggering the freeze). Here's their default values and notes on them:

control=on 0000:0e:00.0 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0
root hub
control=on 0000:0e:00.0 Bus 001 Device 002: ID 13d3:3588 IMC Networks
Wireless_Device (Internal)
control=on 0000:0e:00.0 Bus 001 Device 003: ID 0b05:19af ASUSTek Computer,
Inc. AURA LED Controller (Internal)
control=on 0000:0e:00.0 Bus 001 Device 004: ID 046d:c548 Logitech, Inc. Logi
Bolt Receiver (Plugged in)
control=on 0000:0e:00.0 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0
root hub
control=on 0000:10:00.0 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0
root hub
control=on 0000:10:00.0 Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0
root hub
The 78:00.0 have xhci_pci_suspend -110 errors during boot:

[   17.918387] xhci_hcd 0000:78:00.0: WARN: xHC CMD_RUN timeout
[   17.918508] xhci_hcd 0000:78:00.0: PM: suspend_common(): xhci_pci_suspend
returns -110
[   17.918586] xhci_hcd 0000:78:00.0: can't suspend (hcd_pci_runtime_suspend

returned -110)
control=auto 0000:78:00.0 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation
2.0 root hub
control=auto 0000:78:00.0 Bus 006 Device 001: ID 1d6b:0003 Linux Foundation
3.0 root hub
control=auto 0000:7a:00.3 Bus 007 Device 001: ID 1d6b:0002 Linux Foundation
2.0 root hub
control=auto 0000:7a:00.3 Bus 008 Device 001: ID 1d6b:0003 Linux Foundation
3.0 root hub
control=auto 0000:7a:00.4 Bus 009 Device 001: ID 1d6b:0002 Linux Foundation
2.0 root hub
This is the root hub that freeze during rapid polling, same PCI ID as the line above that is unaffected:
control=auto 0000:7a:00.4 Bus 010 Device 001: ID 1d6b:0003 Linux Foundation
3.0 root hub
control=auto 0000:7b:00.0 Bus 011 Device 001: ID 1d6b:0002 Linux Foundation
2.0 root hub

I then enabled full dynamic debug + netconsole (printk=8):

 $ echo 'module xhci_hcd +p' | sudo tee /proc/dynamic_debug/control
 $ echo 'module usbcore +p' | sudo tee /proc/dynamic_debug/control
 $ echo 'module pci +p' | sudo tee /proc/dynamic_debug/control
 $ echo 8 | sudo tee /proc/sys/kernel/printk

Surprisingly, the system did not freeze for over 20 minutes with 3 instances polling simultaneously and stress-ng --cpu 0. The moment I killed stress-ng first by coincidence, the system froze immediately. Netconsole captured this up until the lockup:
...

[ 1766.915244] xhci_hcd 0000:7a:00.4: PME# disabled
[ 1766.915262] xhci_hcd 0000:7a:00.4: enabling bus mastering
... (normal suspend/resume cycle) ...
[ 1767.170769] xhci_hcd 0000:7a:00.4: PME# disabled
[ 1767.170774] xhci_hcd 0000:7a:00.4: enabling bus mastering
[ 1767.181194] xhci_hcd 0000:7a:00.4: Controller not ready at resume -19
[ 1767.181209] xhci_hcd 0000:7a:00.4: PCI post-resume error -19!
[ 1767.181213] xhci_hcd 0000:7a:00.4: HC died; cleaning up
[ 1767.181222] xhci_hcd 0000:7a:00.4: hcd_pci_runtime_resume: -19
[ 1767.181232] hub 9-0:1.0: state 0 ports 2 chg 0000 evt 0000
[ 1767.181238] hub 10-0:1.0: state 0 ports 2 chg 0000 evt 0000

I don't know enough to say whether they are the same root cause, but both involve an AMD xHC dying on resume, so they may be related.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #20 from Michał Pecio ([email protected]) --- Have you ever seen this "Controller not ready / post-resume error / HC died" sequence without the machine hanging? It would probably be hard not to notice, because the whole bus with all devices becomes unavailable.

This is coming from xhci_resume() calling xhci_handshake() and getting -ENODEV, which means that the hardware malfunctions (or is misconfigured by PCI layer?) and returns 0xffffffff on register read.

I patched this code to pretend that it happened and I got the same log messages, but still no crash or freeze.

Maybe I need to try it a few more times, but it seems that real users are seeing a hang every time the HC dies while opening /dev/bus/usb/ files, because otherwise we would get a separate bug about "HC died" and loss of all devices when running ADB. Or maybe not, because the bus being suspended means that nobody is actively using its devices, so nobody would notice if they disappear...

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #21 from Paul Alesius ([email protected]) ---
(In reply to Michał Pecio from comment #20)
No, I have never seen "Controller not ready at resume -19", "PCI post-resume error -19!", or "HC died; cleaning up" in my weeks of logs without the entire system freezing, after days of hammering the USB devices to diagnose the freeze.

I ran a test holding one usbfs fd open on /dev/bus/usb/010/001 (exec 3< /dev/bus/usb/010/001) while rapidly cycling power/control between 'auto' and 'on' on 0000:7a:00.4. After the initial resume triggered by opening the fd, there were no further suspend/resume messages at all in netconsole (the device stayed powered on and the system remained stable with no further messages over netconsole during hammering of control=on/auto with a 0.2s delay, with full dynamic debug enabled xhci_hcd +p, usbcore +p, pci +p, printk=8)

The "HC died" messages have only ever shown up in netconsole right at the moment of a full system lockup, and only during tests that *repeatedly open and close* the usbfs root-hub file, and during the HC-dead cleanup path.

Let me know if you want me to test a patch or run any other specific test.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

Mathias Nyman ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
                   |                            |om

--- Comment #22 from Mathias Nyman ([email protected]) --- As Michal stated above this is due to reading 0xffffffff from the xHC mmio USBSTS 'status' register in xhci_resume(). This would happen if a PCI xHC controller isn't properly powered up yet.

I'll add a hack that sleeps for a little while and then retries. Lets see if it helps.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #23 from Mathias Nyman ([email protected]) --- Created attachment 309444
--> https://bugzilla.kernel.org/attachment.cgi?id=309444&action=edit
testpatch that retries xhci resume

Testpatch that retries xhci resume 50ms later if xHC status register read returns 0xffffffff

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #24 from Alan Stern ([email protected]) ---
In view of comment #17 plus the fact that the entire system freezes, this looks more likely to be a PCI issue rather than a USB problem. A broken USB host controller isn't the sort of thing you expect to take down the whole computer, but a broken PCI bus or controller could easily do so.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

S. Davar ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #25 from S. Davar ([email protected]) ---
I have been supporting users on an atomic distribution, so no one has tried the test patch.

However, 4+ users who had this freeze behavior on ASUS motherboards with Ryzen 5 / 7 / 9 CPUs seem to have 100% success manually setting all the xHCI controllers to "on" instead of "auto".

All of those users froze very quickly with the usb_poll test, and never froze after changing the xHCI power control behavior. Most have a B650M or similar motherboard.

Was anyone with the freeze able to try the test patch? Also, are there any concerns that a small delay might still be unreliable, e.g. long enough in some cases but not long enough in others? Could it potentially cause a performance hit in some cases, or is it expected to be called rarely enough that it doesn't matter?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

Mario Limonciello (AMD) ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #26 from Mario Limonciello (AMD) ([email protected]) --- Michał Pecio hypothesized that this could be the same root cause as
https://bugzilla.kernel.org/show_bug.cgi?id=221073.

There is a proposed patch series for that bug that could use some testing to confirm.

https://lore.kernel.org/linux-pci/[email protected]/

Can you see if this helps?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

Andrew Skalski ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #27 from Andrew Skalski ([email protected]) --- I believe this same crash is affecting my system as well.

Motherboard: Asus Pro WS WRX90E-SAGE SE
CPU: AMD Ryzen Threadripper PRO 7985WX Kernels I tried: 6.8, 7.0 (Ubuntu 24.04 LTS packages)

Like others have reported, the "adb" command tends to trigger the crash. I also experience intermittent crashes upon plugging in a USB device (either by physically plugging in the cable, or by toggling the USB switch I use to share keyboard/mouse between two systems.) All ports seem to be affected: front panel, rear panel, and also the ports on my Thunderbolt dock.

When the crash is a result of plugging in a device, it generates an MCE (cache error, processor context corrupt, uncorrected.)

This loop alone is enough to cause a crash eventually. My latest attempt ran 12000+ iterations before it crashed:

while sleep 0.01 < /dev/bus/usb/002/001; do echo -n .; done

So far, I have only tried a couple nodes in /dev/bus/usb, but these two both led to a crash:
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

I tested both kernel patches mentioned in this ticket (retry xhci_resume, and the D3cold delay) but neither one resolved the crashes. I tested both using the Ubuntu 6.8 kernel package as a basis. I was not able to successfully build the 7.0 kernel package.

The usbcore.autosuspend=-1 parameter appears to solve both of the crashes.

I tried enabling xhci_hcd debug logging through netconsole, but so far I have not been able to reproduce either crash with debug logging enabled.

In case it's helpful, here's one message I've seen in the "dmesg -W" output (running via IPMI serial console) logged right at the moment of the crash:

xhci_hcd 0000:04:00.4: Refused to change power state from D0 to D3hot

Thanks for your work so far looking into this.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

Lovekesh Solanki ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #28 from Lovekesh Solanki ([email protected]) --- It may be worth getting full acpidump from the affected machine, could it be possible the xhci controller is left inaccessible or in some weird state after power state transition?

and also output of lspci -vvv?

Thanks,
Lovekesh

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #29 from Andrew Skalski ([email protected]) --- Created attachment 310747
--> https://bugzilla.kernel.org/attachment.cgi?id=310747&action=edit
lspci -vvv from affected wrx90 system

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #30 from Andrew Skalski ([email protected]) --- Created attachment 310748
--> https://bugzilla.kernel.org/attachment.cgi?id=310748&action=edit
acpidump from affected wrx90 system

The "lspci -vvv" and "acpidump" outputs are now attached.

In my previous comment where I mentioned the "Refused to change power state from D0 to D3hot" message, I forgot to say that I saw that message only rarely.

By relative frequency, here's what I generally see in "dmesg -W" output on serial console:

Most of the time: nothing
Occasionally: "Long readout interval, skipping watchdog check: cs_nsec:
1528817647 wd_nsec: 1528818397"
Rarely: "Refused to change power state from D0 to D3hot"

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #31 from Andrew Skalski ([email protected]) --- Created attachment 310754
--> https://bugzilla.kernel.org/attachment.cgi?id=310754&action=edit
patch that adds msleep(10) to xhci_resume

The attached patch adds msleep(10) to xhci_resume. This small delay seems to have stopped my machine from crashing in all cases (adb, plugging in devices, cat /dev/bus/usb/*/*)

By adding debug logging to drivers/usb/host/xhci.c, I found the line of code where the lockup appears to occur. Specifically, when this function is called by xhci_resume().

static void xhci_set_cmd_ring_deq(struct xhci_hcd *xhci)

{
        ...
        crcr = xhci_read_64(xhci, &xhci->op_regs->cmd_ring); // this line
        ...
}

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #32 from Michał Pecio ([email protected]) --- For the record, this issue has also been reported on linux-usb, here's the latest case, with links to earlier reports and some patches in later messages:
https://lore.kernel.org/linux-usb/[email protected]/

Your finding is curious. Does it mean it would still work if you add msleep(10) immediately before the failing line and not earlier?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #33 from Andrew Skalski ([email protected]) --- (In reply to Michał Pecio from comment #32)

Yes it works there as well, except you need to use mdelay instead of msleep. For that reason, and because xhci_set_cmd_ring_deq is used elsewhere, I moved the msleep just outside the atomic section in xhci_resume for the proof of concept.

The reason I tried adding the delay was because I noticed enabling debug logging made the problem go away. This made me suspect the lockup was timing related.

Thanks for the link to the other bug reports. I'll have a read through them.

On a separate topic, I did some further investigation into my crashes to tie up some loose ends about the trigger:

1) I'm pretty confident that it's just the two "1022:14c9" controllers that are causing my crashes. I have been running an unpatched kernel with '/sys/bus/pci/devices/<pci-id>/power/control == on' just for those two, and it has been perfectly stable.

2) The crashes that resulted from plugging in USB devices were somewhat of a red herring. A daemon on my system (colord) was reacting to the plug events by scanning everything in /dev/bus/usb. So in all cases the underlying trigger was actually the same: resuming the "1022:14c9" controllers by accessing their device nodes in /dev/bus/usb.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #34 from Michał Pecio ([email protected]) --- Makes sense, I only suggested the alternative placement to confirm that this line is really the trigger and not merely a timing accident.

Crazy idea - does anything change if you replace xhci_read_64() with readq() and drop the 'xhci' argument? Probably not, but that would be a very cheap fix.

If not, it looks like the xHCI FW just isn't ready for action and it somehow manages to bring down the whole CPU.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #35 from Andrew Skalski ([email protected]) --- (In reply to Michał Pecio from comment #34)

I went back and double-checked the effect of different mdelay/msleep placements, and I no longer believe the xhci_read_64 call is involved. So I will need to do more building & testing to find out what's really happening. (Thankfully it will be much less time consuming now that I have my rebuild streamlined.)

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #36 from Lovekesh Solanki ([email protected]) --- I have posted a patch series that addresses
https://lore.kernel.org/linux-usb/[email protected]/

It should fix your issue.

Link to the series:
https://lore.kernel.org/all/[email protected]/T/#u

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #37 from Andrew Skalski ([email protected]) --- (In reply to Lovekesh Solanki from comment #36)

I applied your patch to the 7.0 kernel on Ubuntu 26.04, and here are the results:

OK: I ran "adb" in the background for about two hours without any problems
BAD: The "sleep 0.1; cat /dev/bus/usb/002/001" loop still triggers the lockup

For the "colord" test, I used a patched 6.8 kernel and Ubuntu 24.04 due to differences in colord behavior between versions (the 24.04 version opened the device nodes O_RDONLY, but the 26.04 version only tried O_RDWR and failed due to EACCES.)

OK(?): I tested plugging/unplugging devices for about 10 minutes while colord was running, without issues.
OK(?): I repeated the colord plug/unplug test with adb running in the background. No issues after 10-15 minutes of doing this.

While it doesn't prevent the lockup in all cases, it does improve things.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #38 from Michał Pecio ([email protected]) --- Could it be that your sleep is too short and the loop runs faster than any real world software typically does? What if you increase it to 0.2, 0.5 or 1?

It's theoretically possible that the crash happens when the xHC is resumed too quickly after being suspended. Maybe it doesn't matter how long we keep it awake, until this duration is so long that immediately after suspending some userspace loop enters another iteration and resumes it back.

This *could* explain why the regression happened and why reducing time to suspend from 200 to 120ms helped somewhat. In such case, we only need to get this down enough to stop crashes with real world software. The underlying bug is still there and you will always be able to trigger it with a particularly chosen loop.

The only fully reliable workaround would be to characterize how long the xHC must stay asleep to reliably resume and add in-kernel timeouts to enforce this limit, no matter what userspace (or kernel USB drivers) are trying to do.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #39 from Lovekesh Solanki ([email protected]) --- (In reply to Michał Pecio from comment #38)

Though there's no need for adding this workaround right if no real world situation ever hits this.. may just be extra complexity?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #40 from Andrew Skalski ([email protected]) --- (In reply to Michał Pecio from comment #38)

After experimenting more, I'm starting to think that it's actually the other way around: Suspending the device at the "wrong" time causes the next resume to crash.

I wrote a program that opens "/dev/bus/usb/002/001" and alternately writes "on\n" and "auto\n" on a predetermined schedule. For the timing, I use clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts) to control for drift due to waiting for the IO to complete.

On my system, I found the sweet spot to be around 240700us between initiating write("on\n") and write("auto\n"). Example output leading to a crash on a 7.0 kernel without your patch:

$ sudo ./a.out 
+3000000 us resume, +240722 us autosuspend: ..........R..........S
+3000000 us resume, +240694 us autosuspend: ..........R..........S
+3000000 us resume, +240701 us autosuspend: ..........R..........S
+3000000 us resume, +240703 us autosuspend: ..........R..........S
+3000000 us resume, +240714 us autosuspend: ..........R..........S
+3000000 us resume, +240698 us autosuspend: ..........

The dots are printed in a { print '.', sleep 1/10 of the duration } loop. The "R" and "S" are printed after the write("on\n") and write("auto\n") return.

While I ran the above program, I set these dynamic_debug flags and captured dmesg output. (First column is the time difference from the previous log message.)

module xhci_hcd func xhci_suspend +ptmfl
module xhci_hcd func xhci_resume +ptmfl

30.404398 [ 1135.752405] [7530] xhci_hcd:xhci_resume:1282: xhci_hcd 0000:e4:00.4: xhci_resume: starting usb1 port polling.
0.227738 [ 1135.980143] [6986] xhci_hcd:xhci_suspend:993: xhci_hcd 0000:e4:00.4: xhci_suspend: stopping usb1 port polling.
3.012930 [ 1138.993073] [7530] xhci_hcd:xhci_resume:1282: xhci_hcd 0000:e4:00.4: xhci_resume: starting usb1 port polling.
0.227758 [ 1139.220831] [6986] xhci_hcd:xhci_suspend:993: xhci_hcd 0000:e4:00.4: xhci_suspend: stopping usb1 port polling.
3.013086 [ 1142.233917] [7530] xhci_hcd:xhci_resume:1282: xhci_hcd 0000:e4:00.4: xhci_resume: starting usb1 port polling.
0.227617 [ 1142.461534] [798] xhci_hcd:xhci_suspend:993: xhci_hcd 0000:e4:00.4:
xhci_suspend: stopping usb1 port polling.
3.012995 [ 1145.474529] [7530] xhci_hcd:xhci_resume:1282: xhci_hcd 0000:e4:00.4: xhci_resume: starting usb1 port polling.
0.227667 [ 1145.702196] [798] xhci_hcd:xhci_suspend:993: xhci_hcd 0000:e4:00.4:
xhci_suspend: stopping usb1 port polling.
3.012959 [ 1148.715155] [7530] xhci_hcd:xhci_resume:1282: xhci_hcd 0000:e4:00.4: xhci_resume: starting usb1 port polling.
0.227710 [ 1148.942865] [798] xhci_hcd:xhci_suspend:993: xhci_hcd 0000:e4:00.4:
xhci_suspend: stopping usb1 port polling.

I've found that the amount of time spent suspended doesn't matter at all (so it's not resuming "too quickly".) However it is very sensitive to the "on" time. I haven't tested the full range of wait times, so I don't know if the above test represents the only danger zone. But in my testing, suspending the device sooner is fine. Suspending it later is fine, but suspending it at that exact time is problematic - it "feels like" a race condition to me.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #41 from Andrew Skalski ([email protected]) --- Created attachment 310779
--> https://bugzilla.kernel.org/attachment.cgi?id=310779&action=edit
Test program that crashes by writing on/auto to pci power control

(In reply to Andrew Skalski from comment #40)

That was a typo, I meant to write "/sys/bus/pci/devices/0000:e4:00.4/power/control". I think I'll just attach the program...

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #42 from [email protected] ---
New affected system: ASUS ProArt B850-CREATOR WIFI NEO + Ryzen 7 9700X — total freeze from adb legacy USB backend polling, no Android device connected

I can confirm this bug on my system. Symptoms are identical to the ones described here: complete freeze (keyboard dead including SysRq, mouse dead, no SSH response, display frozen), only a hard reset recovers, and zero kernel logs — no panic, no soft lockup message, nothing. I had kernel.softlockup_panic=1, kernel.watchdog_thresh=5, and hung_task_panic=1 enabled with kdump installed and verified working; the NMI watchdog stayed completely silent and no vmcore was ever produced, so this is a hard halt below what the kernel can report.

System:

Motherboard: ASUS ProArt B850-CREATOR WIFI NEO (BIOS 1103, 2026-09-18; also reproduced on BIOS 0239)
CPU: AMD Ryzen 7 9700X (Raphael/Mendocino iGPU driving the display)
GPU: 2× Radeon RX 9060 XT (Navi 44, headless compute via ROCm/llama.cpp)
RAM: 32 GB DDR5-6000 (EXPO enabled)
OS: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic (also reproduced on 7.0.0-34 and 7.0.0-14)
Trigger: a resident adb server from android-tools-adb (platform-tools 34.0.5-debian, legacy NATIVE USB backend that polls /dev/bus/usb every second).
Important: no Android device was connected during any of the freezes — the adb server was simply running in the background after I had used adb devices/adb logcat, so polling of the root hubs alone was sufficient. The load condition was sustained CPU/PCIe/memory load from local LLM inference (llama.cpp, HIP/ROCm backend, both 9060 XTs active). Freezes happened randomly over several days, always while that workload was running.

USB topology (matches the xHCI superspeed root hub pattern from comment #59):
three 1d6b:0003 Linux Foundation 3.0 root hubs (Bus 2/4/6) plus 2.0 hubs; attached devices are an IMC Bluetooth radio, an ASUS AURA LED controller (0b05:19af), a keyboard and a mouse — nothing unusual.

Ruled out on my side: BIOS update 0239 → 1103 (AGESA 1.3.0.1d) did not help; kernel downgrade to 7.0.0-14 and upgrade to 7.0.0-38 did not help; disabling EXPO (running at JEDEC speed) did not help — freezes continued, and I have since re-enabled EXPO with the adb workaround in place and the system remains stable, so memory speed is not a factor; temperatures normal (37–43 °C); no MCE/EDAC errors, no PCIe AER errors ever logged.

Workaround: setting ADB_LIBUSB=1 system-wide (forces the libusb backend) — no freezes for days since, with the same inference workload running, adb available, and EXPO re-enabled.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=221103

--- Comment #43 from Michał Pecio ([email protected]) --- Did you try the known workaround from comments 14 or 16?

Anyway, this is supposed to be fixed in v7.3-rc6 by commit 29480b05e6e2 usb: hub: use shorter 120ms post resume hold for SS root hubs

Please try this RC or wait for the patch to reach stable kernels.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.