Add device links between tunneled USB3 devices and USB4 Host

The relationship between a USB4 Host Interface providing USB3 tunnels, and tunneled USB3 devices is not represented in device hierarchy.

This caused issues with power managment as devices may suspend and resume in incorrect order.
A device link between the USB4 Host Interface and the USB3 xHCI was originally added to solve this, preventing the USB4 Host Interface from suspending if the USB3 xHCI Host was still active.
This unfortunately also prevents USB4 Host Interface from suspending even if the USB3 xHCI Host is only serving native non-tunneled USB devices.

Improve the current powermanagement situation by creating device links directly from tunneled USB3 devices to the USB4 Host Interface they depend on instead of a device link between the hosts.
This way USB4 host may suspend when the last tunneled device is disconnected.

Intel xHCI hosts are capable of reporting if connected USB3 devices are tunneled via vendor specific capabilities.
Use this until a standard way is available.

v2:
Create device link by default if xHC host is not capable of tunnel detection but the USB3 ports have tunnel capability reported in ACPI _DSD object

Mathias Nyman (4):
xhci: Add USB4 tunnel detection for USB3 devices on Intel hosts
usb: Add tunnel_mode parameter to usb device structure
usb: acpi: add device link between tunneled USB3 device and USB4 Host
Interface
thunderbolt: Don't create device link from USB4 Host Interface to USB3
xHC host

 drivers/thunderbolt/acpi.c       | 40 ++++++------------------
 drivers/usb/core/usb-acpi.c      | 53 ++++++++++++++++++++++++++++++++
 drivers/usb/host/xhci-ext-caps.h |  5 +++
 drivers/usb/host/xhci-hub.c      | 36 ++++++++++++++++++++++
 drivers/usb/host/xhci.c          | 14 +++++++++
 drivers/usb/host/xhci.h          |  3 +-
 include/linux/usb.h              |  8 +++++
 7 files changed, 128 insertions(+), 31 deletions(-)

2.25.1

Knowledge about tunneled devices is useful in order to correctly describe the relationship between tunneled USB3 device and USB4 Host Interface, ensuring proper suspend and resume order, and to be able to power down Thunderbolt if there is no need for tunneling.

Intel hosts share if a USB3 connection is native or tunneled via vendor specific "SPR eSS PORT" registers.

These vendor registers are available if host supports a vendor specific SPR shadow extended capability with ID 206. Registers are per USB3 port and 0x20 apart.

Knowing the tunneling status of the device connected to roothub is enough as all its children will have the same status.

---
v2:
minor return statement change, open up "return !!x"

 drivers/usb/host/xhci-ext-caps.h |  5 +++++
 drivers/usb/host/xhci-hub.c      | 31 +++++++++++++++++++++++++++++++
 drivers/usb/host/xhci.c          | 11 +++++++++++
 drivers/usb/host/xhci.h          |  1 +
 4 files changed, 48 insertions(+)

index 96eb36a58738..67ecf7320c62 100644
@@ -42,6 +42,7 @@
 #define XHCI_EXT_CAPS_DEBUG	10
 /* Vendor caps */
 #define XHCI_EXT_CAPS_VENDOR_INTEL	192
+#define XHCI_EXT_CAPS_INTEL_SPR_SHADOW	206
 /* USB Legacy Support Capability - section 7.1.1 */
 #define XHCI_HC_BIOS_OWNED	(1 << 16)
 #define XHCI_HC_OS_OWNED	(1 << 24)

@@ -64,6 +65,10 @@
 #define XHCI_HLC               (1 << 19)
 #define XHCI_BLC               (1 << 20)
 
+/* Intel SPR shadow capability */
+#define XHCI_INTEL_SPR_ESS_PORT_OFFSET  0x8ac4	/* SuperSpeed port control */
+#define XHCI_INTEL_SPR_TUNEN	BIT(4)		/* Tunnel mode enabled */
+
 /* command register values to disable interrupts and halt the HC */
 /* start/stop HC execution - do not write unless HC is halted*/
 #define XHCI_CMD_RUN		(1 << 0)

index 61f083de6e19..4ba910eadd3f 100644
@@ -752,6 +752,37 @@ static int xhci_exit_test_mode(struct xhci_hcd *xhci)
 	return xhci_reset(xhci, XHCI_RESET_SHORT_USEC);
 }
 
+/**
+ * xhci_port_is_tunneled() - Check if USB3 connection is tunneled over USB4
+ * @xhci: xhci host controller
+ * @port: USB3 port to be checked.
+ *
+ * Some hosts can detect if a USB3 connection is native USB3 or tunneled over
+ * USB4. Intel hosts expose this via vendor specific extended capability 206
+ * eSS PORT registers TUNEN (tunnel enabled) bit.
+ *
+ * A USB3 device must be connected to the port to detect the tunnel.
+ *
+ * Return: true if USB3 connection is tunneled over USB4
+ */
+bool xhci_port_is_tunneled(struct xhci_hcd *xhci, struct xhci_port *port)
+{
+	void __iomem *base;
+	u32 offset;
+
+	base = &xhci->cap_regs->hc_capbase;
+	offset = xhci_find_next_ext_cap(base, 0, XHCI_EXT_CAPS_INTEL_SPR_SHADOW);
+
+	if (offset && offset <= XHCI_INTEL_SPR_ESS_PORT_OFFSET) {
+		offset = XHCI_INTEL_SPR_ESS_PORT_OFFSET + port->hcd_portnum * 0x20;
+
+		if (readl(base + offset) & XHCI_INTEL_SPR_TUNEN)
+			return true;
+	}
+
+	return false;
+}
+
 void xhci_set_link_state(struct xhci_hcd *xhci, struct xhci_port *port,
 			 u32 link_state)
 {

index efdf4c228b8c..1ea2c91106b7 100644
@@ -4525,6 +4525,17 @@ static int xhci_update_device(struct usb_hcd *hcd, struct usb_device *udev)
 	struct xhci_port *port;
 	u32 capability;
 
+	/* Check if USB3 device at root port is tunneled over USB4 */
+	if (hcd->speed >= HCD_USB3 && !udev->parent->parent) {
+		port = xhci->usb3_rhub.ports[udev->portnum - 1];
+
+		if (xhci_port_is_tunneled(xhci, port))
+			dev_dbg(&udev->dev, "tunneled over USB4 link\n");
+		else
+			dev_dbg(&udev->dev, "native USB 3.x link\n");
+		return 0;
+	}
+
 	if (hcd->speed >= HCD_USB3 || !udev->lpm_capable || !xhci->hw_lpm_support)
 		return 0;
 

index ebd0afd59a60..aa1062a595b6 100644
@@ -1929,6 +1929,7 @@ int xhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue, u16 wIndex,
 int xhci_hub_status_data(struct usb_hcd *hcd, char *buf);
 int xhci_find_raw_port_number(struct usb_hcd *hcd, int port1);
 struct xhci_hub *xhci_get_rhub(struct usb_hcd *hcd);
+bool xhci_port_is_tunneled(struct xhci_hcd *xhci, struct xhci_port *port);
 
 void xhci_hc_died(struct xhci_hcd *xhci);
 

2.25.1

Add 'tunnel_mode' enum to usb device structure to describe if a USB3 link is tunneled over USB4, or connected directly using native USB2/USB3 protocols.

Tunneled devices depend on USB4 NHI host to maintain the tunnel. Knowledge about tunneled devices is important to ensure correct suspend and resume order between USB4 hosts and tunneled devices. i.e. make sure tunnel is up before the USB device using it resumes.

USB hosts such as xHCI may have vendor specific ways to detect tunneled connections. This 'tunnel_mode' parameter can be set by USB3 host driver during hcd->driver->update_device(hcd, udev) callback.

tunnel_mode can be set to:
USB_LINK_UNKNOWN = 0
USB_LINK_NATIVE
USB_LINK_TUNNELED

USB_LINK_UNKNOWN is used in case host is not capable of detecting tunneled links.

---
v2:
Add and use a 3 state enum to describe and store tunnel mode instead of bool.

 drivers/usb/host/xhci-hub.c | 13 +++++++++----
 drivers/usb/host/xhci.c     |  7 +++++--
 drivers/usb/host/xhci.h     |  4 ++--
 include/linux/usb.h         |  8 ++++++++
 4 files changed, 24 insertions(+), 8 deletions(-)

index 4ba910eadd3f..d27c30ac17fd 100644
@@ -763,9 +763,12 @@ static int xhci_exit_test_mode(struct xhci_hcd *xhci)
  *
  * A USB3 device must be connected to the port to detect the tunnel.
  *
- * Return: true if USB3 connection is tunneled over USB4
+ * Return: link tunnel mode enum, USB_LINK_UNKNOWN if host is incapable of
+ * detecting USB3 over USB4 tunnels. USB_LINK_NATIVE or USB_LINK_TUNNELED
+ * otherwise.
  */
-bool xhci_port_is_tunneled(struct xhci_hcd *xhci, struct xhci_port *port)
+enum usb_link_tunnel_mode xhci_port_is_tunneled(struct xhci_hcd *xhci,
+						struct xhci_port *port)
 {
 	void __iomem *base;
 	u32 offset;

@@ -777,10 +780,12 @@ bool xhci_port_is_tunneled(struct xhci_hcd *xhci, struct xhci_port *port)
 		offset = XHCI_INTEL_SPR_ESS_PORT_OFFSET + port->hcd_portnum * 0x20;
 
 		if (readl(base + offset) & XHCI_INTEL_SPR_TUNEN)
-			return true;
+			return USB_LINK_TUNNELED;
+		else
+			return USB_LINK_NATIVE;
 	}
 
-	return false;
+	return USB_LINK_UNKNOWN;
 }
 
 void xhci_set_link_state(struct xhci_hcd *xhci, struct xhci_port *port,

index 1ea2c91106b7..a69245074395 100644
@@ -4529,9 +4529,12 @@ static int xhci_update_device(struct usb_hcd *hcd, struct usb_device *udev)
 	if (hcd->speed >= HCD_USB3 && !udev->parent->parent) {
 		port = xhci->usb3_rhub.ports[udev->portnum - 1];
 
-		if (xhci_port_is_tunneled(xhci, port))
+		udev->tunnel_mode = xhci_port_is_tunneled(xhci, port);
+		if (udev->tunnel_mode == USB_LINK_UNKNOWN)
+			dev_dbg(&udev->dev, "link tunnel state unknown\n");
+		else if (udev->tunnel_mode == USB_LINK_TUNNELED)
 			dev_dbg(&udev->dev, "tunneled over USB4 link\n");
-		else
+		else if (udev->tunnel_mode == USB_LINK_NATIVE)
 			dev_dbg(&udev->dev, "native USB 3.x link\n");
 		return 0;
 	}

index aa1062a595b6..0a9d663cdfff 100644
@@ -1929,8 +1929,8 @@ int xhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue, u16 wIndex,
 int xhci_hub_status_data(struct usb_hcd *hcd, char *buf);
 int xhci_find_raw_port_number(struct usb_hcd *hcd, int port1);
 struct xhci_hub *xhci_get_rhub(struct usb_hcd *hcd);
-bool xhci_port_is_tunneled(struct xhci_hcd *xhci, struct xhci_port *port);
-
+enum usb_link_tunnel_mode xhci_port_is_tunneled(struct xhci_hcd *xhci,
+						struct xhci_port *port);
 void xhci_hc_died(struct xhci_hcd *xhci);
 
 #ifdef CONFIG_PM

index 832997a9da0a..672d8fc2abdb 100644
@@ -495,6 +495,12 @@ struct usb_dev_state;
 
 struct usb_tt;
 
+enum usb_link_tunnel_mode {
+	USB_LINK_UNKNOWN = 0,
+	USB_LINK_NATIVE,
+	USB_LINK_TUNNELED,
+};
+
 enum usb_port_connect_type {
 	USB_PORT_CONNECT_TYPE_UNKNOWN = 0,
 	USB_PORT_CONNECT_TYPE_HOT_PLUG,

@@ -605,6 +611,7 @@ struct usb3_lpm_parameters {
  *	WUSB devices are not, until we authorize them from user space.
  *	FIXME -- complete doc
  * @authenticated: Crypto authentication passed
+ * @tunnel_mode: Connection native or tunneled over USB4
  * @lpm_capable: device supports LPM
  * @lpm_devinit_allow: Allow USB3 device initiated LPM, exit latency is in range
  * @usb2_hw_lpm_capable: device can perform USB2 hardware LPM

@@ -714,6 +721,7 @@ struct usb_device {
 	unsigned do_remote_wakeup:1;
 	unsigned reset_resume:1;
 	unsigned port_is_suspended:1;
+	enum usb_link_tunnel_mode tunnel_mode;
 
 	int slot_id;
 	struct usb2_lpm_parameters l1_params;

2.25.1

Describe the power management relationship between a tunneled USB3 device and the tunnel providing USB4 host with a device link as the relationship between them is not evident from normal device hierarchy.

Tunneling capable ports have an ACPI _DSD object pointing to the USB4 Host Interface that is used to establish USB3 3.x tunnels

Set the link directly between tunneled USB3 devices and USB4 Host Interface to ensure that the USB4 host can runtime suspend if no tunneled USB 3.x devices exist.
Current Thunderbolt code sets a link between USB4 Host Interface and USB3 xHCI host which prevents USB4 Host Interface from runtime suspending even if the USB3 host is only serving native USB devices.

See commit b2be2b05cf3b ("thunderbolt: Create device links from ACPI description") for details.

As the device link is only set for USB3 devices that are already tunneled we know that USB4 Host Interface exists and is bound to its driver.

---
v2:
- Don't check twice if device is connected to roothub port
- also create device link if tunnel mode is unknown and port supports tunneling.

 drivers/usb/core/usb-acpi.c | 53 +++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

index 7f8a912d4fe2..21585ed89ef8 100644
@@ -142,6 +142,53 @@ int usb_acpi_set_power_state(struct usb_device *hdev, int index, bool enable)
 }
 EXPORT_SYMBOL_GPL(usb_acpi_set_power_state);
 
+/**
+ * usb_acpi_add_usb4_devlink - add device link to USB4 Host Interface for tunneled USB3 devices
+ *
+ * @udev: Tunneled USB3 device connected to a roothub.
+ *
+ * Adds a device link between a tunneled USB3 device and the USB4 Host Interface
+ * device to ensure correct runtime PM suspend and resume order. This function
+ * should only be called for tunneled USB3 devices.
+ * The USB4 Host Interface this tunneled device depends on is found from the roothub
+ * port ACPI device specific data _DSD entry.
+ *
+ * Return: negative error code on failure, 0 otherwise
+ */
+static int usb_acpi_add_usb4_devlink(struct usb_device *udev)
+{
+	const struct device_link *link;
+	struct usb_port *port_dev;
+	struct usb_hub *hub;
+
+	if (!udev->parent || udev->parent->parent)
+		return 0;
+
+	hub = usb_hub_to_struct_hub(udev->parent);
+	port_dev = hub->ports[udev->portnum - 1];
+
+	struct fwnode_handle *nhi_fwnode __free(fwnode_handle) =
+		fwnode_find_reference(dev_fwnode(&port_dev->dev), "usb4-host-interface", 0);
+
+	if (IS_ERR(nhi_fwnode))
+		return 0;
+
+	link = device_link_add(&port_dev->child->dev, nhi_fwnode->dev,
+			       DL_FLAG_AUTOREMOVE_CONSUMER |
+			       DL_FLAG_RPM_ACTIVE |
+			       DL_FLAG_PM_RUNTIME);
+	if (!link) {
+		dev_err(&port_dev->dev, "Failed to created device link from %s to %s\n",
+			dev_name(&port_dev->child->dev), dev_name(nhi_fwnode->dev));
+		return -EINVAL;
+	}
+
+	dev_dbg(&port_dev->dev, "Created device link from %s to %s\n",
+		dev_name(&port_dev->child->dev), dev_name(nhi_fwnode->dev));
+
+	return 0;
+}
+
 /*
  * Private to usb-acpi, all the core needs to know is that
  * port_dev->location is non-zero when it has been set by the firmware.

@@ -262,6 +309,12 @@ usb_acpi_find_companion_for_device(struct usb_device *udev)
 	if (!hub)
 		return NULL;
 
+
+	/* Tunneled USB3 devices depend on USB4 Host Interface, set device link to it */
+	if (udev->speed >= USB_SPEED_SUPER &&
+	    udev->tunnel_mode != USB_LINK_NATIVE)
+		usb_acpi_add_usb4_devlink(udev);
+
 	/*
 	 * This is an embedded USB device connected to a port and such
 	 * devices share port's ACPI companion.

2.25.1

USB core will create device links between tunneled USB3 devices and USB4 Host Interface during USB device creation.

Those device links are removed with the tunneled USB3 devices, allowing USB4 Host Interface to runtime suspend if USB3 tunnels are not used.

So remove device link creation between USB4 Host Interface and USB3 xHC during NHI probe

---
v2:
no change

 drivers/thunderbolt/acpi.c | 40 ++++++++++----------------------------
 1 file changed, 10 insertions(+), 30 deletions(-)

index c9b6bb46111c..d2a0054217da 100644
@@ -32,40 +32,20 @@ static acpi_status tb_acpi_add_link(acpi_handle handle, u32 level, void *data,
 		goto out_put;
 
 	/*
-	 * Try to find physical device walking upwards to the hierarcy.
-	 * We need to do this because the xHCI driver might not yet be
-	 * bound so the USB3 SuperSpeed ports are not yet created.
+	 * Ignore USB3 ports here as USB core will set up device links between
+	 * tunneled USB3 devices and NHI host during USB device creation.
+	 * USB3 ports might not even have a physical device yet if xHCI driver
+	 * isn't bound yet.
 	 */
-	do {
-		dev = acpi_get_first_physical_node(adev);
-		if (dev)
-			break;
-
-		adev = acpi_dev_parent(adev);
-	} while (adev);
-
-	/*
-	 * Check that the device is PCIe. This is because USB3
-	 * SuperSpeed ports have this property and they are not power
-	 * managed with the xHCI and the SuperSpeed hub so we create the
-	 * link from xHCI instead.
-	 */
-	while (dev && !dev_is_pci(dev))
-		dev = dev->parent;
-
-	if (!dev)
+	dev = acpi_get_first_physical_node(adev);
+	if (!dev || !dev_is_pci(dev))
 		goto out_put;
 
-	/*
-	 * Check that this actually matches the type of device we
-	 * expect. It should either be xHCI or PCIe root/downstream
-	 * port.
-	 */
+	/* Check that this matches a PCIe root/downstream port. */
 	pdev = to_pci_dev(dev);
-	if (pdev->class == PCI_CLASS_SERIAL_USB_XHCI ||
-	    (pci_is_pcie(pdev) &&
-		(pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ||
-		 pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM))) {
+	if (pci_is_pcie(pdev) &&
+	    (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ||
+	     pci_pcie_type(pdev) == PCI_EXP_TYPE_DOWNSTREAM)) {
 		const struct device_link *link;
 
 		/*

2.25.1

Hello,

I had a try with this version of the series but I'm missing a device link from XHCI controller after applying it.

Before series these are the two links (6.11-rc6):
consumer:pci:0000:00:03.1 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:00:03.1 consumer:pci:0000:c4:00.3 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:c4:00.3

After 6.11-rc6 + v2 series only this link:
consumer:pci:0000:00:03.1 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:00:03.1

Here is the whole topology on my system:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e8]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e9]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f7]
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 1c) 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5261 PCI Express Card Reader [10ec:5261] (rev 01)
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a] c2:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev 03)
c2:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640] c2:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device [1022:15c7] c2:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
c2:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ba]
c2:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63) c2:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
c2:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Device [1022:164a]
c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
c3:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] AMD IPU Device [1022:1502]
c4:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
c4:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c0]
c4:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c1]
c4:00.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #1 [1022:1668] c4:00.6 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #2 [1022:1669]

Hi,

Do you have device connected? The link will now be created only when there is a device router with USB 3.x hub/device connected.

Ah thanks for clarifying.
I connected a TBT4-UDZ and I see another link show up now.

0000:c4:00.6/consumer:usb:8-1 -> ../../../virtual/devlink/pci:0000:c4:00.6--usb:8-1

However, something seems wrong with runtime PM when the device is disconnected. Let me show you:

❯ ls -l /sys/bus/pci/drivers/thunderbolt/*/consumer*
lrwxrwxrwx 1 root root 0 Sep 4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.5/consumer:pci:0000:00:03.1 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:00:03.1/ lrwxrwxrwx 1 root root 0 Sep 4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.6/consumer:pci:0000:00:04.1 -> ../../../virtual/devlink/pci:0000:c4:00.6--pci:0000:00:04.1/ ❯ cat /sys/bus/pci/drivers/thunderbolt/*/power/runtime_status
suspended
suspended

(connect dock)

❯ ls -l /sys/bus/pci/drivers/thunderbolt/*/consumer*

lrwxrwxrwx 1 root root 0 Sep  4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.5/consumer:pci:0000:00:03.1 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:00:03.1/
lrwxrwxrwx 1 root root 0 Sep  4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.6/consumer:pci:0000:00:04.1 -> ../../../virtual/devlink/pci:0000:c4:00.6--pci:0000:00:04.1/
lrwxrwxrwx 1 root root 0 Sep  4 12:34 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.6/consumer:usb:8-1 -> ../../../virtual/devlink/pci:0000:c4:00.6--usb:8-1/

❯ cat /sys/bus/pci/drivers/thunderbolt/*/power/runtime_status suspended
active

(unconnect dock and wait at least autosuspend_delay)

❯ ls -l /sys/bus/pci/drivers/thunderbolt/*/consumer*
lrwxrwxrwx 1 root root 0 Sep 4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.5/consumer:pci:0000:00:03.1 -> ../../../virtual/devlink/pci:0000:c4:00.5--pci:0000:00:03.1/ lrwxrwxrwx 1 root root 0 Sep 4 12:33 /sys/bus/pci/drivers/thunderbolt/0000:c4:00.6/consumer:pci:0000:00:04.1 -> ../../../virtual/devlink/pci:0000:c4:00.6--pci:0000:00:04.1/ ❯ cat /sys/bus/pci/drivers/thunderbolt/*/power/runtime_status
suspended
active

I would have expected the USB4 host router to go back into runtime PM, but it doesn't anymore until I reboot the system.

Yes, it should enter runtime suspend after a while. Would you mind sharing dmesg around this?

I tried on my system and it works as expected (the device-power.sh is internal script that dumps the device power states, I can share if you want).

Devices 0d.2/3 are the host routers. 07.[0-3] are the PCIe tunnels (not really used here but shown for completeness, I don't have PCIe tunnel enabled to the hub). 0d.0 is the xHCI controller.

Plug in USB4 hub
----------------

# device-power.sh -s
All PCI devices (software state)
...
0000:00:07.0 8086:7ec4 Status: [D3cold] Real status: [    D0] Runtime PM: [suspended]
0000:00:07.1 8086:7ec5 Status: [D3cold] Real status: [    D0] Runtime PM: [suspended]
0000:00:07.2 8086:7ec6 Status: [D3cold] Real status: [    D0] Runtime PM: [suspended]
0000:00:07.3 8086:7ec7 Status: [D3cold] Real status: [    D0] Runtime PM: [suspended]
...
0000:00:0d.0 8086:7ec0 Status: [D3cold] Real status: [    D0] Runtime PM: [suspended]
0000:00:0d.2 8086:7ec2 Status: [    D0] Real status: [    D0] Runtime PM: [   active]
0000:00:0d.3 8086:7ec3 Status: [    D0] Real status: [    D0] Runtime PM: [   active]

Unplug and wait for the autosuspend_delay
-----------------------------------------

# device-power.sh -s
All PCI devices (software state)
...
0000:00:07.0 8086:7ec4 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
0000:00:07.1 8086:7ec5 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
0000:00:07.2 8086:7ec6 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
0000:00:07.3 8086:7ec7 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
...
0000:00:0d.0 8086:7ec0 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
0000:00:0d.2 8086:7ec2 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]
0000:00:0d.3 8086:7ec3 Status: [D3cold] Real status: [D3cold] Runtime PM: [suspended]

I was capturing artifacts when all of a sudden it started to work. Then I remembered I updated the BIOS on this system very recently. This is a pre-production BIOS.

I downgraded back to old BIOS and everything works as you expect, so there is some BIOS bug at play. I'll see if others can reproduce my result and drive a BIOS solution.

Sorry for the noise on the patch, all is great!

Thanks for testing it

-Mathias