PCI: Avoid FLR for AMD Starship USB 3.0

PATCH Applied Kevin Buettner 5 participants 7 messages May 24, 2020 – Jun 25, 2020 PCI

lore

Share & tools

lore

Kevin Buettner

author

2020-05-24 07:35 UTC #0 lore raw

PATCH 1/1 PCI: Avoid FLR for AMD Starship USB 3.0

This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.

Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.

Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.

Shortly after attempting to start the guest, I see these messages:

May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready 1023ms after FLR; waiting
May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting
May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting
May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting

And then eventually:

May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready 65535ms after FLR; giving up
May 23 23:01:05 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744 > 2500), lowering kernel.perf_event_max_sample_rate to 1000
May 23 23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 100.952 msecs
...
 kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
May 23 23:01:25 mesquite kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]

The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.

With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.

Signed-off-by Kevin Buettner <[email protected]>

drivers/pci/quirks.c +2 -0

index 43a0c2ce635e..b1db58d00d2b 100644
@@ -5133,6 +5133,7 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503

@@ -5143,6 +5144,7 @@ static void quirk_no_flr(struct pci_dev *dev)
 	dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;
 }
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503, quirk_no_flr);

Bjorn Helgaas

Applied

2020-05-27 21:31 UTC #1 lore raw

[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?

https://lore.kernel.org/r/[email protected]
AMD Starship USB 3.0 host controller
https://lore.kernel.org/r/CAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4zmt1RA@mail.gmail.com
AMD Matisse HD Audio & USB 3.0 host controller
]

Quoted 35 lines · Kevin Buettner

This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready 1023ms after FLR; waiting
May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting
May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting
May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready 65535ms after FLR; giving up
May 23 23:01:05 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744 > 2500), lowering kernel.perf_event_max_sample_rate to 1000
May 23 23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 100.952 msecs
...
 kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
May 23 23:01:25 mesquite kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.

Quoted 5 lines

With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>

Applied to pci/virtualization for v5.8, thanks!

Quoted 21 lines

drivers/pci/quirks.c +2 -0

index 43a0c2ce635e..b1db58d00d2b 100644
@@ -5133,6 +5133,7 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503

@@ -5143,6 +5144,7 @@ static void quirk_no_flr(struct pci_dev *dev)
 	dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;
 }
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503, quirk_no_flr);

Deucher, Alexander

2020-05-27 21:42 UTC #2 lore raw

[AMD Official Use Only - Internal Distribution Only]

Quoted 10 lines

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, May 27, 2020 5:32 PM
To: Kevin Buettner <[email protected]>
Cc: [email protected]; Bjorn Helgaas <[email protected]>; Alex Williamson <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>
Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0

Quoted 4 lines

[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?

+Nehal who handles our USB drivers.

Nehal any ideas about FLR or whether it should be advertised?

Alex

Quoted 93 lines

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2F20200524003529.598434ff%40f31- 4.lan&data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0% 7C0%7C637262119015438912&sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi Mk06dLZCmgUOrsGf3kLpY%3D&reserved=0
AMD Starship USB 3.0 host controller
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z mt1RA%40mail.gmail.com&data=02%7C01%7Calexander.deucher%40a md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11 a82d994e183d%7C0%7C0%7C637262119015438912&sdata=69GsHB0HCp 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&reserved=0
AMD Matisse HD Audio & USB 3.0 host controller ]
Quoted 22 lines · Kevin Buettner
This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
Quoted 7 lines
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
Quoted 1 line
2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler
(perf_event_nmi_handler) took too long to run: 100.952 msecs ...
Quoted 2 lines
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
[qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
BUG:
Quoted 7 lines
soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.
Quoted 5 lines
With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>
Applied to pci/virtualization for v5.8, thanks!
drivers/pci/quirks.c
Quoted 5 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0
43a0c2ce635e..b1db58d00d2b 100644
drivers/pci/quirks.c +0 -0
@@ -5133,6 +5133,7 @@
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
Quoted 11 lines
drivers/pci/quirks.c +1 -0
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
@@ static void quirk_no_flr(struct pci_dev *dev)
dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET; }
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
quirk_no_flr);
Quoted 1 line

Marcos Scriven

2020-05-28 08:13 UTC #3 lore raw

Quoted 18 lines · Alexander

[AMD Official Use Only - Internal Distribution Only]
Quoted 10 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, May 27, 2020 5:32 PM
To: Kevin Buettner <[email protected]>
Cc: [email protected]; Bjorn Helgaas <[email protected]>; Alex Williamson <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>
Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0
Quoted 4 lines
[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?

Quoted 6 lines

+Nehal who handles our USB drivers.
Nehal any ideas about FLR or whether it should be advertised?
Alex

I had read somewhere that the IO die in the Ryzen/Threadripper packages are identical to the ones used in the motherboard chipsets.

Since the latter do reset ok, it would seem a BIOS update of the AGESA may potentially fix the issue.

Unfortunately, it's not something motherboard manufacturer's customer support people know how to deal with or pass back up the chain to AMD engineers. Actual use of this feature seems to be fairly niche.

After I added the workaround for the USB and audio controllers on the 3rd-gen Ryzen, I tried contacting Kim Phillips (who I found as a kernel committer to x86/cpu/amd), but haven't heard back.

It would be wonderful to know if this can potentially be fixed in CPU firmware, and whether there's any likelihood of it actually being distributed by motherboard manufacturers.

Marcos

Quoted 93 lines

Quoted 93 lines
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2F20200524003529.598434ff%40f31- 4.lan&data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0% 7C0%7C637262119015438912&sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi Mk06dLZCmgUOrsGf3kLpY%3D&reserved=0
AMD Starship USB 3.0 host controller
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z mt1RA%40mail.gmail.com&data=02%7C01%7Calexander.deucher%40a md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11 a82d994e183d%7C0%7C0%7C637262119015438912&sdata=69GsHB0HCp 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&reserved=0
AMD Matisse HD Audio & USB 3.0 host controller ]
Quoted 22 lines · Kevin Buettner
This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
Quoted 7 lines
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
Quoted 1 line
2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler
(perf_event_nmi_handler) took too long to run: 100.952 msecs ...
Quoted 2 lines
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
[qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
BUG:
Quoted 7 lines
soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.
Quoted 5 lines
With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>
Applied to pci/virtualization for v5.8, thanks!
drivers/pci/quirks.c
Quoted 5 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0
43a0c2ce635e..b1db58d00d2b 100644
drivers/pci/quirks.c +0 -0
@@ -5133,6 +5133,7 @@
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
Quoted 11 lines
drivers/pci/quirks.c +1 -0
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
@@ static void quirk_no_flr(struct pci_dev *dev)
    dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;  }
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
quirk_no_flr);
Quoted 1 line

Marcos Scriven

2020-06-08 17:47 UTC #4 lore raw

Quoted 42 lines · Marcos Scriven

Quoted 18 lines · Alexander
[AMD Official Use Only - Internal Distribution Only]
Quoted 10 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, May 27, 2020 5:32 PM
To: Kevin Buettner <[email protected]>
Cc: [email protected]; Bjorn Helgaas <[email protected]>; Alex Williamson <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>
Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0
Quoted 4 lines
[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?
Quoted 6 lines
+Nehal who handles our USB drivers.
Nehal any ideas about FLR or whether it should be advertised?
Alex
I had read somewhere that the IO die in the Ryzen/Threadripper packages are identical to the ones used in the motherboard chipsets.
Since the latter do reset ok, it would seem a BIOS update of the AGESA may potentially fix the issue.
Unfortunately, it's not something motherboard manufacturer's customer support people know how to deal with or pass back up the chain to AMD engineers. Actual use of this feature seems to be fairly niche.
After I added the workaround for the USB and audio controllers on the 3rd-gen Ryzen, I tried contacting Kim Phillips (who I found as a kernel committer to x86/cpu/amd), but haven't heard back.

Quoted 8 lines

It would be wonderful to know if this can potentially be fixed in CPU firmware, and whether there's any likelihood of it actually being distributed by motherboard manufacturers.
Marcos

Dear Alex/Nehal

I wonder if you're able to comment please on whether FLR should be advertised?

Is there any chance this could be fixed at the bios/AGESA level, and effectively rolled out?

Thanks

Marcos

Quoted 93 lines

Quoted 93 lines
Quoted 93 lines
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2F20200524003529.598434ff%40f31- 4.lan&data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0% 7C0%7C637262119015438912&sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi Mk06dLZCmgUOrsGf3kLpY%3D&reserved=0
AMD Starship USB 3.0 host controller
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z mt1RA%40mail.gmail.com&data=02%7C01%7Calexander.deucher%40a md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11 a82d994e183d%7C0%7C0%7C637262119015438912&sdata=69GsHB0HCp 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&reserved=0
AMD Matisse HD Audio & USB 3.0 host controller ]
Quoted 22 lines · Kevin Buettner
This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
Quoted 7 lines
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
Quoted 1 line
2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler
(perf_event_nmi_handler) took too long to run: 100.952 msecs ...
Quoted 2 lines
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
[qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
BUG:
Quoted 7 lines
soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.
Quoted 5 lines
With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>
Applied to pci/virtualization for v5.8, thanks!
drivers/pci/quirks.c
Quoted 5 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0
43a0c2ce635e..b1db58d00d2b 100644
drivers/pci/quirks.c +0 -0
@@ -5133,6 +5133,7 @@
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
Quoted 11 lines
drivers/pci/quirks.c +1 -0
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
@@ static void quirk_no_flr(struct pci_dev *dev)
    dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;  }
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
quirk_no_flr);
Quoted 1 line

Shah, Nehal-bakulchandra

2020-06-09 11:47 UTC #5 lore raw

Quoted 19 lines · Marcos Scriven

Quoted 18 lines · Marcos Scriven
Quoted 16 lines · Alexander
[AMD Official Use Only - Internal Distribution Only]
Quoted 14 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, May 27, 2020 5:32 PM
To: Kevin Buettner <[email protected]>
Cc: [email protected]; Bjorn Helgaas <[email protected]>; Alex Williamson <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>
Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0
[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
+Nehal who handles our USB drivers.
Nehal any ideas about FLR or whether it should be advertised?
Alex

Sorry for the delay. We are looking into this with BIOS team. I shall revert soon on this.

Quoted 24 lines

Quoted 14 lines
I had read somewhere that the IO die in the Ryzen/Threadripper packages are identical to the ones used in the motherboard chipsets.
Since the latter do reset ok, it would seem a BIOS update of the AGESA may potentially fix the issue.
Unfortunately, it's not something motherboard manufacturer's customer support people know how to deal with or pass back up the chain to AMD engineers. Actual use of this feature seems to be fairly niche.
After I added the workaround for the USB and audio controllers on the 3rd-gen Ryzen, I tried contacting Kim Phillips (who I found as a kernel committer to x86/cpu/amd), but haven't heard back.
Quoted 8 lines
It would be wonderful to know if this can potentially be fixed in CPU firmware, and whether there's any likelihood of it actually being distributed by motherboard manufacturers.
Marcos
Dear Alex/Nehal

Quoted 99 lines

I wonder if you're able to comment please on whether FLR should be advertised?
Is there any chance this could be fixed at the bios/AGESA level, and effectively rolled out?
Thanks
Marcos
Quoted 90 lines
Quoted 90 lines
Quoted 90 lines
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2F20200524003529.598434ff%40f31- 4.lan&data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0% 7C0%7C637262119015438912&sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi Mk06dLZCmgUOrsGf3kLpY%3D&reserved=0
AMD Starship USB 3.0 host controller
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z mt1RA%40mail.gmail.com&data=02%7C01%7Calexander.deucher%40a md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11 a82d994e183d%7C0%7C0%7C637262119015438912&sdata=69GsHB0HCp 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&reserved=0
AMD Matisse HD Audio & USB 3.0 host controller ]
Quoted 22 lines · Kevin Buettner
This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
Quoted 7 lines
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
Quoted 1 line
2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler
(perf_event_nmi_handler) took too long to run: 100.952 msecs ...
Quoted 2 lines
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
[qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
BUG:
Quoted 7 lines
soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.
Quoted 5 lines
With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>
Applied to pci/virtualization for v5.8, thanks!
drivers/pci/quirks.c
Quoted 5 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0
43a0c2ce635e..b1db58d00d2b 100644
drivers/pci/quirks.c +0 -0
@@ -5133,6 +5133,7 @@
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
Quoted 11 lines
drivers/pci/quirks.c +1 -0
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
@@ static void quirk_no_flr(struct pci_dev *dev)
    dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;  }
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
quirk_no_flr);

Regard

Nehal Shah

Marcos Scriven

2020-06-25 10:23 UTC #6 lore raw

Quoted 23 lines · Nehal-bakulchandra

Hi
Quoted 19 lines · Marcos Scriven
Quoted 18 lines · Marcos Scriven
Quoted 16 lines · Alexander
[AMD Official Use Only - Internal Distribution Only]
Quoted 14 lines
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, May 27, 2020 5:32 PM
To: Kevin Buettner <[email protected]>
Cc: [email protected]; Bjorn Helgaas <[email protected]>; Alex Williamson <[email protected]>; Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>
Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0
[+cc Alex D, Christian -- do you guys have any contacts or insight into why we suddenly have three new AMD devices that advertise FLR support but it doesn't work? Are we doing something wrong in Linux, or are these devices defective?

Quoted 9 lines

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
+Nehal who handles our USB drivers.
Nehal any ideas about FLR or whether it should be advertised?
Alex
Sorry for the delay. We are looking into this with BIOS team. I shall revert soon on this.

Hi Nehal

Sorry to keep pestering about this, but wondering if there's any movement on this?

Is it something that's likely to be fixed and actually rolled out by motherboard manufacturers?

There's been some grumblings in the community about adding workarounds rather than fixing, so it would be good to pass on expectations here.

Marcos

Quoted 128 lines

Quoted 24 lines
Quoted 14 lines
I had read somewhere that the IO die in the Ryzen/Threadripper packages are identical to the ones used in the motherboard chipsets.
Since the latter do reset ok, it would seem a BIOS update of the AGESA may potentially fix the issue.
Unfortunately, it's not something motherboard manufacturer's customer support people know how to deal with or pass back up the chain to AMD engineers. Actual use of this feature seems to be fairly niche.
After I added the workaround for the USB and audio controllers on the 3rd-gen Ryzen, I tried contacting Kim Phillips (who I found as a kernel committer to x86/cpu/amd), but haven't heard back.
Quoted 8 lines
It would be wonderful to know if this can potentially be fixed in CPU firmware, and whether there's any likelihood of it actually being distributed by motherboard manufacturers.
Marcos
Dear Alex/Nehal
Quoted 99 lines
I wonder if you're able to comment please on whether FLR should be advertised?
Is there any chance this could be fixed at the bios/AGESA level, and effectively rolled out?
Thanks
Marcos
Quoted 90 lines
Quoted 90 lines
Quoted 90 lines
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2F20200524003529.598434ff%40f31- 4.lan&data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0% 7C0%7C637262119015438912&sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi Mk06dLZCmgUOrsGf3kLpY%3D&reserved=0
AMD Starship USB 3.0 host controller
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z mt1RA%40mail.gmail.com&data=02%7C01%7Calexander.deucher%40a md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11 a82d994e183d%7C0%7C0%7C637262119015438912&sdata=69GsHB0HCp 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&reserved=0
AMD Matisse HD Audio & USB 3.0 host controller ]
Quoted 22 lines · Kevin Buettner
This commit adds an entry to the quirk_no_flr table for the AMD Starship USB 3.0 host controller.
Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 motherboard with an AMD Ryzen Threadripper 3970X.
Without this patch, when attempting to assign (pass through) an AMD Starship USB 3.0 host controller to a guest OS, the system becomes increasingly unresponsive over the course of several minutes, eventually requiring a hard reset.
Shortly after attempting to start the guest, I see these messages:
May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51 mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR; waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not ready 8191ms after FLR; waiting
And then eventually:
Quoted 7 lines
May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
Quoted 1 line
2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
May 23 23:01:08 mesquite kernel: INFO: NMI handler
(perf_event_nmi_handler) took too long to run: 100.952 msecs ...
Quoted 2 lines
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
[qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
BUG:
Quoted 7 lines
soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
The above log snippets were obtained using the aforementioned hardware running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64. My fix was applied to a local copy of the F32 kernel package, then rebuilt, etc.
Quoted 5 lines
With this patch in place, the host kernel doesn't exhibit these problems. The guest OS (also Fedora 32) starts up and works as expected with the passed-through USB host controller.
Signed-off-by Kevin Buettner <[email protected]>
Applied to pci/virtualization for v5.8, thanks!
drivers/pci/quirks.c
Quoted 5 lines
drivers/pci/quirks.c index renamed from drivers/pci/quirks.c +0 -0
43a0c2ce635e..b1db58d00d2b 100644
drivers/pci/quirks.c +0 -0
@@ -5133,6 +5133,7 @@
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443, quirk_intel_qat_vf_cap);
Quoted 11 lines
drivers/pci/quirks.c +1 -0
  * FLR may cause the following to devices to hang:
  *
  * AMD Starship/Matisse HD Audio Controller 0x1487
+ * AMD Starship USB 3.0 Host Controller 0x148c
  * AMD Matisse USB 3.0 Host Controller 0x149c
  * Intel 82579LM Gigabit Ethernet Controller 0x1502
  * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
@@ static void quirk_no_flr(struct pci_dev *dev)
    dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;  }
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
quirk_no_flr);
Quoted 1 line
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
quirk_no_flr);
Regard
Nehal Shah