New: tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.
https://bugzilla.kernel.org/show_bug.cgi?id=221822
Bug ID: 221822
Summary: tb-retry.patch (against drivers/thunderbolt/tb.c):
makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.
Product: Drivers
Version: 2.5
Kernel Version: 7.2-rc5
Hardware: AMD
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: USB
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 310555
--> https://bugzilla.kernel.org/attachment.cgi?id=310555&action=edit
tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.
## Summary
CalDigit Element 5 Hub (USB4 v2 / TB5) on AMD Strix Point USB4 host: link training fails CL-states/TMU/tunnel creation on ~5 of 6 plug cycles, then one cycle lands and is fully functional — link never lane-bonds (single lane 20 Gb/s)
## Description
An AMD Strix Point mini PC (Meigao F8BAC, Ryzen AI 9 HX 370) cannot reliably establish a link with a CalDigit Element 5 Hub (vendor 0x3d, device 0x28, TB5/USB4 v2, firmware 64.1 — current per CalDigit). The hub works correctly on a Mac at USB4 40 Gb/s with the same cables.
Most plug cycles fail identically:
``` thunderbolt 1-2: new device found, vendor=0x3d device=0x28 thunderbolt 1-2: CalDigit, Inc. Element 5 Hub thunderbolt 0000:c8:00.6: 2: failed to enable CL states thunderbolt 0000:c8:00.6: 2: failed to enable TMU thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed thunderbolt 0000:c8:00.6: 2:1: hop deactivation failed for hop 1, index 8 thunderbolt 0000:c8:00.6: PCIe Down path activation failed: -107 thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed, aborting thunderbolt 1-2: device disconnected ```
The host router autosuspends and re-scans every ~18 s, so cycles repeat indefinitely — **and after ~5 failed cycles, one attempt succeeds**. Once linked, the device is fully functional and stable:
- thunderbolt 0-2: Element 5 Hub, authorized=1, **20.0 Gb/s rx/tx** (link only ever trains **single lane** — bonding never occurs, even on successful links)
- USB3 tunnel up: Intel USB3 HUB (8087:5787, JHL9480 internal) + CalDigit USB3.2 10G hub enumerate through the tunnel
- PCIe tunnel up: full JHL9480 bridge hierarchy (01:00.0 → 02:00.0/01.0/02.0)
- 5 USB HDDs, card readers, and an RTL8153 GbE adapter all enumerate and pass traffic through the tunnel; sustained reads 208–215 MB/s (platter-limited) with zero disconnects or errors after the link lands
So the failure is not a hard incompatibility — the link *can* come up, but lane training/CL-state enablement is unreliable and only succeeds sporadically (and never with lane bonding).
### Hardware
- Host: Micro Computer (HK) Tech Limited "AI Series", board **F8BAC** (Meigao/miniscloud mini PC), AMD Ryzen AI 9 HX 370 (Strix Point), AMI BIOS 1.04 (2025-04-11, latest for this board)
- USB4 host routers: 0000:c8:00.5 [1022:151c] (domain0 / Port A) and 0000:c8:00.6 [1022:151d] (domain1 / Port B). **Both ports train the device** and both fail tunnel setup on most cycles; the working 40 Gb/s bonded link currently in use is on Port A.
- Device: CalDigit Element 5 Hub, TB5/USB4 v2, FW 64.1
- Cables: the CalDigit-supplied TB5 cable
### Kernels tested (all byte-identical failure)
- Proxmox VE kernels: 6.14.8-2-pve, 6.14.11-9-pve, 7.0.6-2-pve, 7.0.14-8-pve
- **Mainline 7.2-rc5** (Ubuntu mainline build) with thunderbolt.dyndbg=+p
- thunderbolt module params: clx=0, clx=0 bw_alloc_mode=0 (link trains, tunnels still fail)
- Kernel cmdlines: pcie_port_pm=off, pcie_aspm=off iommu=pt — no effect
- Boot-attached vs hotplug — no effect
- Both USB4 ports and direct attach — no effect on the tunnel failure. **Daisy-chaining the hub downstream of an Elgato TB3 Pro Dock was the only working topology** (all drives restored, stable) — it was rolled back as non-viable for this deployment, but it corroborates that the failure is specific to native USB4-v1↔USB4-v2 link negotiation (the TB3 path skips CL-state negotiation)
### Code analysis + retry patch experiment (drivers/thunderbolt @ v7.2-rc5)
- -107 = **-ENOTCONN** ("transport endpoint not connected") — same errno lane bonding returns (silently) when the second lane never comes up. PCIe path activation dies on a port-state race.
- The printed failed to enable CL states is **not** a support check (-EOPNOTSUPP is swallowed without warning); it is a real **sideband config-space transaction failure** against the device router in tb_port_pm_secondary_set()/tb_port_clx_enable() (clx.c). TMU and USB3-tunnel failures are the same class.
- **No-fallback aggravator in tb_enable_clx():** CL2 is attempted first for v2 routers (JHL9480 is v2); only -EOPNOTSUPP falls back to CL0s/CL1. A *transient* CL2 transaction error aborts CL enablement with no fallback and no retry. Same no-retry pattern in tb_enable_tmu() and tb_tunnel_usb3().
- Retimer NVM reads report vendor=0x0 device=0x0 even on successful links — further sideband unreliability.
- Gil Fine's 7.2-rc1 router-readiness patches (Router-Ready verify 062023c, Config-Ready timeout ba2cc38, notification timeout 255 ms e24f3c0) are present in the test kernel and do not cure it.
**Tested a 3-site retry patch** (CL2→CL0s/CL1 fallback on ANY error + 3× retry with 100/200/300 ms backoff on tb_enable_clx/tb_enable_tmu/tb_tunnel_usb3/tb_tunnel_activate), built as 7.2.0-rc5-tbfix1:
- **Boot-attached:** CL-states and TMU failures **eliminated** (fallback + retries fixed both). USB3 tunnel creation still failed every cycle — the 600 ms retry window isn't wide enough for USB3 specifically.
- **Hotplug (unplug 10 s, replug): full success on first cycle** — CL/TMU/USB3/PCIe all succeed, zero errors, PCIe Down/Up path activation complete, 4 JHL9480 bridges enumerated, all USB devices + storage via tunnel, sustained reads 207–222 MB/s (platter-limited), stable.
- **Conclusion:** the CL/TMU failures are provably transient and recoverable in-kernel (retries work). USB3 tunnel creation is the remaining hard point — it needs either a device reset (hotplug) or a substantially longer backoff/readiness wait. This narrows the fix target for upstream considerably.
- **Lane bonding is also timing-flaky:** the host's second-lane port sits in TB_PORT_CONNECTING (state 1) for the full 1 s tb_wait_for_port window on most cycles, then "failed to reach state TB_PORT_UP. Ignoring port." When lane 1 does train late, tb_switch_set_link_width(DUAL) is never re-attempted, so the session stays single-lane (20 G) even though both lanes are up. Both a TB3 dock and this TB5 hub DO bond at 40 Gb/s aggregate on cycles where lane 1 trains in time. Suggested hardening: re-attempt lane bonding when the second lane reaches TB_PORT_UP after the initial attempt fails.
### Notes
- A Thunderbolt 3 dock (Elgato TB3 Pro Dock) links and tunnels reliably on the same host/port — the TB3 path skips USB4 CL-state negotiation, so the host hardware is functional for TB3.
- Blanket auto-authorization udev rule is in place; failure precedes authorization anyway.
- Possibly related: bug 221319 (AMD USB4 host + TB5 peripherals), and the May 2026 linux-usb thread "USB4 v2 TBGAA tunnel creation crash in TMU enhanced uni-directional mode" (AMD Strix Halo + TB5 dock) — same class, but that thread ended unresolved with a suspected device PD-firmware cause. This report adds a distinct data point: here the link *does* come up (single lane) and the failure is specifically at device-switch CL-state enablement and tunnel creation.
### Attachments
1. dmesg-tb-plug.txt — standard dmesg of one plug cycle on 7.0.14-8-pve
2. dmesg-tb-dyndbg.txt — full thunderbolt.dyndbg=+p trace of plug cycles on mainline 7.2-rc5
3. tb-retry.patch - against drivers/thunderbolt/tb.c: makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.
---
## Attachment 1: dmesg-tb-plug.txt (7.0.14-8-pve)
``` [ 342.119755] thunderbolt 1-2: new device found, vendor=0x3d device=0x28 [ 342.119951] thunderbolt 1-2: CalDigit, Inc. Element 5 Hub [ 343.058197] thunderbolt 0000:c8:00.6: 2: failed to enable CL states [ 343.058526] thunderbolt 0000:c8:00.6: 2: failed to enable TMU [ 343.059968] thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed [ 343.061282] thunderbolt 0000:c8:00.6: 2:1: hop deactivation failed for hop 1, index 8 [ 343.061386] thunderbolt 0000:c8:00.6: PCIe Down path activation failed: -107 [ 343.061749] thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed, aborting [ 343.062408] thunderbolt 1-2: device disconnected ```
## Attachment 2: dmesg-tb-dyndbg.txt (mainline 7.2-rc5, thunderbolt.dyndbg=+p)
Key excerpt (full trace on request — multi-hundred-line cycles):
``` [ 267.142724] thunderbolt 0000:c8:00.6: 2: current link width symmetric, single lane [ 267.145051] thunderbolt 1-2: new device found, vendor=0x3d device=0x28 [ 267.145265] thunderbolt 1-2: CalDigit, Inc. Element 5 Hub [ 268.493956] thunderbolt 0000:c8:00.6: 2: failed to enable CL states [ 268.494796] thunderbolt 0000:c8:00.6: 2:20: available bandwidth for new USB3 tunnel 18000/18000 Mb/s [ 268.495368] thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed [ 268.496009] thunderbolt 0000:c8:00.6: 0:5 <-> 2:9 (PCI): activating [ 268.496102] thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed, aborting [ 268.496176] thunderbolt 0000:c8:00.6: 0:2: switch unplugged [ 268.496231] thunderbolt 0000:c8:00.6: 2: link width set to symmetric, single lane [ 268.496469] thunderbolt 1-2: device disconnected ```
Host-router TMU succeeds each resume: 0: TMU: mode set to: bi-directional, HiFi.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 221822] tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.
https://bugzilla.kernel.org/show_bug.cgi?id=221822
--- Comment #1 from j. james johnson ([email protected]) --- The 3-site retry patch (CL2→CL0s/CL1 fallback on *any* CL-enable error; 3× retry with 100/200/300 ms backoff on tb_enable_clx/tb_enable_tmu/tb_tunnel_usb3/tb_tunnel_activate) was ported to the **shipping distro kernel 7.0.14-8-pve** (pve-kernel build with patch applied as patches/kernel/0052-...) and is now the daily driver on this host. Results correct the original report:
- **Boot-attached hub self-recovers after ~2 failed cycles (~252 s) — no hotplug.** This invalidates the report's claim that boot-attached USB3 tunnel creation is a hard failure point requiring a device reset.
- **CL-state and TMU failures eliminated** — the CL2→CL0s/CL1 fallback absorbs them.
- **The link lane-bonds at 40 Gb/s** on the successful cycle (link width set to symmetric, dual lanes) — the original Summary's "never bonds" is wrong; bonding occurs whenever lane 1 trains in time, and the retries keep the session alive long enough for that to happen.
- **New errno data:** the winning cycle logged `PCIe Down path activation
failed: -13 (EACCES) twice before succeeding, alongside the -107` (ENOTCONN) seen on failing cycles. Both are transient sideband/port-state races during device-router enumeration; retry recovers both.
- **BIOS excluded:** flashed board BIOS 1.06 (which adds a retimer GPIO-detect change). Retimer sideband reads unchanged — lane-0 NVM still all-zeros, lane-1 retimer still never responds. Firmware is not the active variable; tracked separately with the board vendor.
- 5 HDDs + PCIe/USB3 devices stable under sustained load (I/O at platter speed)
Open, load-tested asks to restate to upstream when the thread is next touched:
(a) retry-with-backoff on transient CL/TMU/USB3/PCIe-path-activation failures;
(b) CL2→CL0s/CL1 fallback on any CL-enable error, not just -EOPNOTSUPP; (c) re-attempt tb_switch_set_link_width(DUAL) when the second lane reaches TB_PORT_UP late — single-lane sessions still occur on cycles where lane 1 trains after the initial bonding attempt.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.