New: Pluggable UD-4VPD dock appears to continually reset with AMD AI 365

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Bug ID: 219748
Summary: Pluggable UD-4VPD dock appears to continually reset
with AMD AI 365
Product: Drivers
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: USB
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected],
[email protected]
Regression: No

Not sure if this is related to the linux kernel to fix up the Pluggable UD-4VPD dock https://www.spinics.net/lists/linux-usb/msg263475.html.

The kernel running is the latest from here AUR (en) - linux-mainline-um5606
https://aur.archlinux.org/packages/linux-mainline-um5606. It is based on the 6.13 kernel. It adds some patches as this laptop has been a bit unstable. Hence, the additional amdgpu boot parameters.

thunderbolt.dyndbg=+p output shows the issue starting at time 27.027128 in the attached dmesg

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #1 from tom ([email protected]) ---
Created attachment 307570
--> https://bugzilla.kernel.org/attachment.cgi?id=307570&action=edit
dmesg

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #2 from Mika Westerberg ([email protected]) --- Can you enable tracing as well and attach both dmesg and the trace output here?

1. Boot the system nothing connected.
2. Enable tracing as explained in
(https://github.com/intel/tbtools?tab=readme-ov-file#tracing)

# tbtrace enable

3. Plug in the dock
4. Verify that the problem reproduces.
5. Disable tracing

# tbtrace disable

6. Take both trace output and dmesg and attach them:

# tbtrace dump -vv > trace.out
# dmesg > dmesg.out

(We need the dmesg as well to be able to match the trace lines with the driver logs).

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Thomas ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #307570|0                           |1

is obsolete| |

--- Comment #3 from Thomas ([email protected]) ---
Created attachment 307577
--> https://bugzilla.kernel.org/attachment.cgi?id=307577&action=edit
dmesg while tracing

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #4 from Thomas ([email protected]) ---
Created attachment 307578
--> https://bugzilla.kernel.org/attachment.cgi?id=307578&action=edit
trace

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #5 from Thomas ([email protected]) ---
(In reply to Mika Westerberg from comment #2)

Attached trace output. It charged the whole time. The first couple were slower than normal.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #6 from Mika Westerberg ([email protected]) --- Created attachment 307579
--> https://bugzilla.kernel.org/attachment.cgi?id=307579&action=edit
Try to shorten SB access timeout

Thanks for the logs! From the trace it looks like similar what already was seen with Pluggable hub connected to Intel hardware. The sideband accesses take some time and the hub interprets this as "issue" and triggers disconnect (or this is what it looks like). The attached patch shortens the timeouts to half. Can you try with it and see if that makes any difference?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #7 from Thomas ([email protected]) ---
Tried the shorter timeout patch and verified it was running with uname -v. The issue still persisted. Shortened USB4_PORT_SB_TIMEOUT to 200 it is still disconnecting.

Are there other values I should try or logs I can get?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #8 from Mika Westerberg ([email protected]) --- Created attachment 307605
--> https://bugzilla.kernel.org/attachment.cgi?id=307605&action=edit
Skip retimer scanning

Can you try this one instead of the previous one? This completely disables sideband accesses done by the driver. If this makes difference then we need to look into the access patterns whether there is something we need to change in order to make the Pluggable device happy.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #9 from Thomas ([email protected]) ---
Created attachment 307659
--> https://bugzilla.kernel.org/attachment.cgi?id=307659&action=edit
tbtrace with skip retimer patch

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #10 from Thomas ([email protected]) ---
Created attachment 307660
--> https://bugzilla.kernel.org/attachment.cgi?id=307660&action=edit
dmesg with skip retimer patch

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #11 from Thomas ([email protected]) ---
The dock works with the skip retimer patch.
Plugging in the dock makes the device connection sound twice from there:
1. plugged in ethernet
2. plugged in usb
3. bumped the cable and unplugged the dock briefly -- sorry
4. plugged in an HDMI monitor

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #12 from Mario Limonciello (AMD) ([email protected]) --- Mika - what would you think about putting the retimer scan into a delayed work queue for 5 or 10 seconds later or so?

As a guess, maybe these devices don't like the sideband traffic while they're trying to get setup.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #13 from Mika Westerberg ([email protected]) --- That's an option yes but I would still like to understand what is going on. In order to support CLx (with ALPM) we are going to need to access the sideband (and this applies to Windows too) before we setup DP tunnels and that might happen pretty soon after we enumerated the router.

I'll look into the traces next and see if I can identify something. Thanks!

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #14 from Mika Westerberg ([email protected]) --- Created attachment 307705
--> https://bugzilla.kernel.org/attachment.cgi?id=307705&action=edit
Delay router enumeration for a while

Can you try the attached patch (instead of the previous)? It still skips scanning retimers but adds artificial delay before we enumerate the router. If there is some sort of timeout in the pluggable device side it should trigger now. Please attach dmesg (no need for trace).

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #15 from Thomas ([email protected]) ---
Created attachment 307733
--> https://bugzilla.kernel.org/attachment.cgi?id=307733&action=edit
dmesg after delay-router-enumeration patch

The latest patch resulted in the dock detaching again. Dmesg attached.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #16 from Mika Westerberg ([email protected]) --- Thanks! So we have:

[   56.773330] thunderbolt 0000:c6:00.6: 0:3: hotplug: scanning
[   56.773335] thunderbolt 0000:c6:00.6: 0:3: hotplug: no switch found
[   56.773349] thunderbolt 0000:c6:00.6: acking hot plug event on 0:2
[   56.773362] thunderbolt 0000:c6:00.6: 0:2: hotplug: scanning
[   56.773504] thunderbolt 0000:c6:00.6: 0:2: is connected, link is up (state:
2)
[   56.773515] thunderbolt 0000:c6:00.6: 0:2: waiting for a while
[   57.371077] thunderbolt 0000:c6:00.6: acking hot unplug event on 0:3
[   57.371143] thunderbolt 0000:c6:00.6: acking hot unplug event on 0:2
[   57.692761] thunderbolt 0000:c6:00.6: acking hot plug event on 0:3
[   57.692828] thunderbolt 0000:c6:00.6: acking hot plug event on 0:2
[   58.291125] thunderbolt 0000:c6:00.6: acking hot unplug event on 0:2
[   58.291217] thunderbolt 0000:c6:00.6: acking hot unplug event on 0:3
[   59.430462] thunderbolt 0000:c6:00.6: 0:2: .. done, trying to enumerate the

router

In other words there is some ~600ms timeout in the pluggable device that triggers hot-remove and then hot-add if the device is not enumerated. This is naturally the USB4 spec, as far as I can tell the host does not need to enumerate the device at all actually.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #17 from Mika Westerberg ([email protected]) --- Created attachment 307740
--> https://bugzilla.kernel.org/attachment.cgi?id=307740&action=edit
Scan downstream retimers after the router has been enumerated

Can you try this patch? It moves retimer scanning happen after the router has been enumerated so hopefully this is acceptable for the pluggable device.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Sean ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #18 from Sean ([email protected]) ---
Created attachment 307767
--> https://bugzilla.kernel.org/attachment.cgi?id=307767&action=edit
dmesg after Scan downstream retimers after the router patch

While this topic is related to a Plugable dock, the Thunderbolt implementation is having some issues for AMD GPUs where display EDIDs can't be read
(https://gitlab.freedesktop.org/drm/amd/-/issues/3050). This latest patch appears to have helped that issue, as my TB3 Lenovo dock on a Framework 16 with an AMD 7840HS CPU is now able to use its display outputs at the correct resolutions. I'll be testing with addition TB docks I have access to in the near future as well.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #19 from Thomas ([email protected]) ---
Created attachment 307768
--> https://bugzilla.kernel.org/attachment.cgi?id=307768&action=edit
dmesg after move retimers patch

Patch worked. There's two docks tried in the dmesg. A Dell WD19S and then the pluggable UD-4VPD.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #20 from Mika Westerberg ([email protected]) --- (In reply to Sean from comment #18)

I suggest opening separate issue for this. I don't see how this patch could affect GPU reading EDID from monitor. Assuming it is over DisplayPort tunnel, at that point the device router is already enumerated. This patch just speeds up the enumeration to cope with the Pluggable device internal timeout.

Please attach full dmesg with "thunderbolt.dyndbg=+p" in the command line too to that new issue.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #21 from Mika Westerberg ([email protected]) --- (In reply to Thomas from comment #19)

Thanks for testing. However, I don't see Dell dock. Only the Pluggable one and there is one unplug and then plug. Is that on purpose?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #22 from Thomas ([email protected]) ---
Testing two in one dmesg wasn't the best move on my part. The Dell was me plugging in the wrong dock. Thought I'd leave it for the extra test. It does clutter up the dmesg though.

Dell

[   52.334084] usb 7-1: new high-speed USB device number 2 using xhci_hcd
[   52.475560] usb 7-1: New USB device found, idVendor=0bda, idProduct=5487,
bcdDevice= 1.47
[   52.475577] usb 7-1: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[   52.475581] usb 7-1: Product: Dell dock
[   52.475583] usb 7-1: Manufacturer: Dell Inc.

The switch from Dell to Pluggable, I think, happens here

[  105.079211] usb 7-1.5: USB disconnect, device number 4
[  108.745727] thunderbolt 0000:c6:00.5: control channel starting...

UD-4VPD is identified here

[  113.165640] thunderbolt 0-2: Plugable Plugable UD-4VPD

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Thomas ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #23 from Thomas ([email protected]) ---
Created attachment 307771
--> https://bugzilla.kernel.org/attachment.cgi?id=307771&action=edit
dmesg after move retimers patch only the pluggable device

Thanks for the patch. This is the dmesg output with the move retimers patch. Only the pluggable device is tested in this one.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Mika Westerberg ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #307740|0                           |1

is obsolete| |

--- Comment #24 from Mika Westerberg ([email protected]) --- Created attachment 307772
--> https://bugzilla.kernel.org/attachment.cgi?id=307772&action=edit
Scan downstream retimers after the router has been enumerated v2

Can you still try the attached patch too? I missed two cases where we also want to scan the retimers so did that change there. It should not affect your case but better to check. Thanks!

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #25 from Mario Limonciello (AMD) ([email protected]) --- Comment on attachment 307772
--> https://bugzilla.kernel.org/attachment.cgi?id=307772
Scan downstream retimers after the router has been enumerated v2

That patch looks good to me Mika. Feel free to add:

Also since Sean mentions this helps a different issue on his Pluggable dock (no EDID read) I think you can also add:

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #26 from Thomas ([email protected]) ---
Created attachment 307773
--> https://bugzilla.kernel.org/attachment.cgi?id=307773&action=edit
0001-thunderbolt-Scan with display attached that was undetected

Tried the most recent patch. This is a longer dmesg as devices were tried as well. It was a bit odd as the network didn't seem to be detected initially and then, after unplugging and plugging in the dock, it did work. Monitors plugged into the dock would report as not identified and, after plugging them in and being identified, they did not have video output. Not sure how much of this is due to the patch.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #27 from Thomas ([email protected]) ---
Created attachment 307774
--> https://bugzilla.kernel.org/attachment.cgi?id=307774&action=edit
dmesg showing no r8152 network device afer 0001-thunderbolt-Scan patch

This is the dmesg that had the r8152 network adapter missing.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #28 from Mario Limonciello (AMD) ([email protected]) --- Do those same failures happen in the case that the retimer scan was totally commented out?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #29 from Mika Westerberg ([email protected]) --- For the first dmesg with "undetected" display the enumeration goes fine and the DP tunnel is created however:

[   54.496699] [drm] DPIA AUX failed on 0x0(1), error 3
[   54.579780] [drm] DM_MST: starting TM on aconnector: 00000000a96810d6 [id:
133]
[   54.587986] [drm] DM_MST: DP14, 4-lane link detected
[   54.591410] [drm] DPIA AUX failed on 0x1000(5), error 3
[   54.596690] [drm] DMUB HPD RX IRQ callback: link_index=7
[   54.605700] [drm] DMUB HPD RX IRQ callback: link_index=7
[   54.612512] [drm] DPIA AUX failed on 0x2003(1), error 3
[   54.616646] [drm] DMUB HPD RX IRQ callback: link_index=7
[   54.624654] [drm] DMUB HPD RX IRQ callback: link_index=7
[   54.631037] [drm] DPIA AUX failed on 0x2002(4), error 3

Looks to me some sort of link training issue but as far as I can tell the DP tunnel is up just fine. Immediately after this there is unplug:

[   54.632887] thunderbolt 0000:c6:00.6: acking hot unplug event on 0:2
[   54.632987] thunderbolt 1-0:2.1: retimer disconnected

I think this done by you and then replug. If that's the case then from USB4 perspective it looks okay (let me know if this is not the case). I'm leaving the graphics side for Mario to comment on as I'm not qualified. I do see MST and that seem to be problematic in Linux IIRC.

There is one more thing that need to check. Do you have CONFIG_USB4_DEBUGFS_MARGINING=y set in your kernel .config? I ask because the driver exposes the AMD retimer from the dock upstream port side too. It is okay if you have that set but if not something is not right then.

Regarding the missing network adapter, what happened there is that the USB4 link did not come up at all so all you got is the USB 2.x wires and devices. You see them "High Speed" and "Full Speed" in the dmesg. Now, this is something outside of what software can fix. Typically it's the Power Delivery firmware that handles this but most cases simply unplug plug or flipping the cable makes it work.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #30 from Mario Limonciello (AMD) ([email protected]) ---

My main worry is if the retimer scan is the reason for the DPIA issues too. The report that Sean linked in was a failing to read an EDID which got fixed by pushing the scan later.

So if you can reproduce the display issues try with it commented out entirely (maybe we want that for debugging as a module parameter Mika?).

With it commented out if they're cleared up, I think we still need to push it to a delayed work queue.
If they're still happening, then this should be a GPU driver or GPU microcode issue. We can talk about that on a drm/amd Gitlab issue.

Yup agree.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #31 from Mika Westerberg ([email protected]) --- (In reply to Mario Limonciello (AMD) from comment #30)

I'm not convinced that retimer scanning has anything to do with that issue. It does affect timing when the device router is enumerated and that's the issue with Pluggable, they have implemented some sort of non-standard timeout there.

The other issue happens long after device router is already enumerated and the DP tunnels are established. The patch may seem to "solve" the issue but it probably just hides the real one and this is what we need to understand.

Therefore I suggest to open a separate issue for that one with appropriate full dmesg and the repro steps, CC me too I can take a look. We can defer this fix until we understand the real problem but let's keep the two separate.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #32 from Thomas ([email protected]) ---

Yes. Same issues happen when testing the HDMI port of the dock with that patch applied.

No, CONFIG_USB4_DEBUGFS_MARGINING doesn't show up in /proc/config.gz

Tried a few more things to see if it would provide clearer details.

In Linux, the monitor connected to the dock detected correctly, which doesn't happen every time, setting the resolution to below 4k causes the monitor to display output. Before setting the resolution below 4k, the monitor would detect HDMI signal over and over without displaying video output.

Booted to Windows and reproduced the issue of the monitor blinking on an off with the HDMI output on the dock. It did work after unplugging the dock and plugging it back in though. Didn't check the resolution though. If that is an important detail, I can get it.

Overall, both the Skip retimer scanning and 0001-thunderbolt patches cause the dock to be correctly detected.

I'm not sure if I should pursue -- or how to pursue -- the other issues with video. They seems sporadic. Amdgpu doesn't seem entirely stable. So maybe video issues are expected.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #33 from Mario Limonciello (AMD) ([email protected]) --- OK considering that I think this patch is correct, I feel we shouldn't dig any further from a CM perspective at this time.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

--- Comment #34 from Mika Westerberg ([email protected]) --- Just sent the patch upstream:

https://lore.kernel.org/linux-usb/[email protected]/

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=219748

Mario Limonciello (AMD) ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #35 from Mario Limonciello (AMD) ([email protected]) --- It's upstream now, closing the ticket.

https://git.kernel.org/torvalds/c/75749d2c1d8ce

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.