Skip to contentSkip to conversation

PCI: Fix Intel Xeon 6 x2 quirk collateral damage on adjacent x4 endpoints

PATCH Superseded Rick Warner 5 participants 6 messages Jul 29 – Nov 8

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Fix Intel Xeon 6 x2 quirk collateral damage on adjacent x4 endpoints

commit a22250fe933d ("PCI: Add Extended Tag + MRRS quirk for Xeon 6") introduced a traffic mitigation quirk for Intel Xeon 6 root ports that negotiate down to an x2 lane width. However, that patch manipulated host bridge properties globally via 'bridge->no_ext_tags = 1' and by hooking 'bridge->enable_device'.

Because multiple unrelated root ports can reside under the exact same global pci_host_bridge domain structure, this overly aggressive mitigation causes severe collateral damage. When a low-speed or bifurcated secondary device (such as an onboard ASMedia SATA controller or BMC graphics link) matches the x2 condition, the kernel strips away Extended Tags and forces a 128B MRRS restriction across that ENTIRE host bridge. This instantly breaks or starves adjacent high-performance, unrelated x4 endpoints (such as NVMe drives), resulting in controller timeouts, initialization failures, and missing drives at boot.

Fix this by refactoring the quirk logic to be completely per-device and downstream-isolated. Remove the broad host bridge structure assignments entirely. Instead, convert the hook to an explicit 'DECLARE_PCI_FIXUP_FINAL' sweep. When an x2 bifurcated root port is discovered, leverage 'pci_walk_bus()' to step down only that specific root port's subordinate tree, manually clearing 'PCI_EXP_DEVCTL_EXT_TAG' and 'PCI_EXP_DEVCTL_READRQ' directly in the endpoint Device Control configuration registers. This enforces Intel's stability parameters locally on the bottlenecked lane branches while fully protecting the parallel x4 highway channels.

Fixes a22250fe933d ("PCI: Add Extended Tag + MRRS quirk for Xeon 6")
---
 arch/x86/pci/fixup.c | 41 +++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)
arch/x86/pci/fixup.c +23 -18
index b301c6c8df75..c2bdf580692a 100644
@@ -301,38 +301,43 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	PCI_DEVICE_ID_INTEL_MCH_PC1,	pcie_r
  *
  * https://cdrdv2.intel.com/v1/dl/getContent/837176
  */
-static int limit_mrrs_to_128(struct pci_host_bridge *b, struct pci_dev *pdev)
+static int pci_xeon_x2_bifurc_quirk_perdev(struct pci_dev *pdev, void *data)
 {
-	int readrq = pcie_get_readrq(pdev);
-
-	if (readrq > 128)
-		pcie_set_readrq(pdev, 128);
-
+	/* Only clamp actual endpoints, skip downstream switches/bridges */
+	if (!pci_is_bridge(pdev)) {
+		pci_info(pdev, "Disabling Extended Tags and Clamping MRRS to 128B due to upstream Xeon 6 x2 bifurcation\n");
+		pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_EXT_TAG);
+		pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_READRQ);
+	}
 	return 0;
 }
 
 static void pci_xeon_x2_bifurc_quirk(struct pci_dev *pdev)
 {
-	struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
 	u32 linkcap;
 
 	pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP, &linkcap);
 	if (FIELD_GET(PCI_EXP_LNKCAP_MLW, linkcap) != 0x2)
 		return;
 
-	bridge->no_ext_tags = 1;
-	bridge->enable_device = limit_mrrs_to_128;
-	pci_info(pdev, "Disabling Extended Tags and limiting MRRS to 128B (performance reasons due to x2 PCIe link)\n");
+	pci_info(pdev, "Xeon 6 x2 lane bifurcation detected; restricting subordinate devices\n");
+
+	/*
+	 * Walk down only this specific root port's downstream bus tree
+	 * and clear extended tags specifically for devices on this branch.
+	 */
+	if (pdev->subordinate)
+		pci_walk_bus(pdev->subordinate, pci_xeon_x2_bifurc_quirk_perdev, NULL);
 }
 
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db0, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db1, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db2, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db3, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db6, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db7, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db8, pci_xeon_x2_bifurc_quirk);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db9, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db0, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db1, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db2, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db3, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db6, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db7, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db8, pci_xeon_x2_bifurc_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0db9, pci_xeon_x2_bifurc_quirk);
 
 /*
  * Fixup to mark boot BIOS video selected by BIOS before it changes
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The one-time bus walk completely misses dynamically hotplugged downstream endpoints, breaking the workaround and exposing the system to hardware instability.

Pre-existing issues:
- [High] Endpoint drivers and VFIO guests can trivially overwrite the manual 128B MRRS clamp and Extended Tag restrictions. --

--- Patch [1]: [PATCH] PCI: Fix Intel Xeon 6 x2 quirk collateral damage on adjacent x4 endpoints --- commit 437863bd6a6bd870ba3147da1f484eb59fd2282e
Author: Rick Warner <[email protected]>

PCI: Fix Intel Xeon 6 x2 quirk collateral damage on adjacent x4 endpoints

This commit refactors the Intel Xeon 6 x2 traffic mitigation quirk to apply locally to subordinate devices rather than globally on the host bridge. It removes the host bridge property assignments and instead performs a one-time bus walk during enumeration to manually clear Extended Tags and clamp MRRS directly on endpoints.

Quoted 11 lines
arch/x86/pci/fixup.c +1 -1
index b301c6c8df753..c2bdf580692aa 100644
@@ -301,38 +301,43 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	PCI_DEVICE_ID_INTEL_MCH_PC1,	pcie_r
  *
  * https://cdrdv2.intel.com/v1/dl/getContent/837176
  */
-static int limit_mrrs_to_128(struct pci_host_bridge *b, struct pci_dev *pdev)
+static int pci_xeon_x2_bifurc_quirk_perdev(struct pci_dev *pdev, void *data)
 {

arch/x86/pci/fixup.c DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_MCH_PC1, pcie_r · line 310

Quoted 10 lines
arch/x86/pci/fixup.c +5 -5
-	int readrq = pcie_get_readrq(pdev);
-
-	if (readrq > 128)
-		pcie_set_readrq(pdev, 128);
-
+	/* Only clamp actual endpoints, skip downstream switches/bridges */
+	if (!pci_is_bridge(pdev)) {
+		pci_info(pdev, "Disabling Extended Tags and Clamping MRRS to 128B due to upstream Xeon 6 x2 bifurcation\n");
+		pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_EXT_TAG);
+		pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_READRQ);

[Severity: High]
This is a pre-existing issue, but can endpoint drivers and VFIO guests trivially overwrite these manual register changes?

This code clears the MRRS and Extended Tag bits directly via configuration writes, but it does not set a subsystem constraint such as no_inc_mrrs on the bridge. During device probe or VFIO guest execution, could a subsequent call to pcie_set_readrq() unconditionally overwrite this 128B limit and reintroduce the hardware instability?

Quoted 6 lines
arch/x86/pci/fixup.c +1 -0
+	}
 	return 0;
 }
 
 static void pci_xeon_x2_bifurc_quirk(struct pci_dev *pdev)
 {

[ ... ]

arch/x86/pci/fixup.c DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_MCH_PC1, pcie_r · line 330

Quoted 6 lines
arch/x86/pci/fixup.c +6 -0
+	/*
+	 * Walk down only this specific root port's downstream bus tree
+	 * and clear extended tags specifically for devices on this branch.
+	 */
+	if (pdev->subordinate)
+		pci_walk_bus(pdev->subordinate, pci_xeon_x2_bifurc_quirk_perdev, NULL);

[Severity: High]
Does this one-time bus walk completely miss dynamically hotplugged downstream endpoints?

By removing the dynamic enable_device hook, this mitigation relies entirely on the DECLARE_PCI_FIXUP_FINAL sweep which executes exclusively once at boot during root port enumeration. If a device is dynamically added later via hotplug or if an SR-IOV VF is instantiated, the root port is not re-enumerated and the callback won't be triggered for the new endpoints. Could this leave hotplugged devices unmitigated, allowing them to negotiate full MRRS and break the workaround?

Quoted 1 line

}

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Rick Warner
Quoted 16 lines · Rick Warner

commit a22250fe933d ("PCI: Add Extended Tag + MRRS quirk for Xeon 6") introduced a traffic mitigation quirk for Intel Xeon 6 root ports that negotiate down to an x2 lane width. However, that patch manipulated host bridge properties globally via 'bridge->no_ext_tags = 1' and by hooking 'bridge->enable_device'.

Because multiple unrelated root ports can reside under the exact same global pci_host_bridge domain structure, this overly aggressive mitigation causes severe collateral damage. When a low-speed or bifurcated secondary device (such as an onboard ASMedia SATA controller or BMC graphics link) matches the x2 condition, the kernel strips away Extended Tags and forces a 128B MRRS restriction across that ENTIRE host bridge. This instantly breaks or starves adjacent high-performance, unrelated x4 endpoints (such as NVMe drives), resulting in controller timeouts, initialization failures, and missing drives at boot.

Quoted 8 lines

Fix this by refactoring the quirk logic to be completely per-device and downstream-isolated. Remove the broad host bridge structure assignments entirely. Instead, convert the hook to an explicit 'DECLARE_PCI_FIXUP_FINAL' sweep. When an x2 bifurcated root port is discovered, leverage 'pci_walk_bus()' to step down only that specific root port's subordinate tree, manually clearing 'PCI_EXP_DEVCTL_EXT_TAG' and 'PCI_EXP_DEVCTL_READRQ' directly in the endpoint Device Control configuration registers.

One problem I see with this approach is that it won't work for hotplugged devices below a bifurcated Root Port: You're only adjusting Extended Tags and MRRS once on enumeration of the Root Ports, leaving devices that are hotplugged later at incorrect settings.

But perhaps you could simply amend limit_mrrs_to_128() to walk up to the Root Port, check whether it is bifurcated, and bail out if it's not?

Thanks,

Lukas

#3 lore raw

Hi All,

Here's an updated version of the fix.  Now in the early setup it sets no_inc_mrrs on the entire bridge instead of disabling ext tags and limiting mrrs for the entire root bridge.  Then it uses DECLARE_PCI_FIXUP_HEADER to check every registering pci device to see if it's a descendant of an x2 port, and if so, disables ext tags and limits mrrs for that device.  This should properly handle hotplugging now.

This has been tested on a Gigabyte MS74-HB0 motherboard with a Seagate ZP4000GM30063 M.2 drive. With the stock kernel, that drive fails to initialize and is unavailable once booted. Testing showed that extended tags are required for it to work.  The drive works properly with this patch.  Concerns were mentioned about vfio/VM usage being able to re-enable extended tags if bridge->no_ext_tags is not set.  I'm not sure how to address that without adding additional a flag to pci device structures that individually blocks extended tags per device instead of relying on the bridge flag for it. I'm open to suggestions for that.

Thanks,
Rick Warner

Quoted 30 lines · Lukas Wunner
Quoted 16 lines · Rick Warner

commit a22250fe933d ("PCI: Add Extended Tag + MRRS quirk for Xeon 6") introduced a traffic mitigation quirk for Intel Xeon 6 root ports that negotiate down to an x2 lane width. However, that patch manipulated host bridge properties globally via 'bridge->no_ext_tags = 1' and by hooking 'bridge->enable_device'.

Because multiple unrelated root ports can reside under the exact same global pci_host_bridge domain structure, this overly aggressive mitigation causes severe collateral damage. When a low-speed or bifurcated secondary device (such as an onboard ASMedia SATA controller or BMC graphics link) matches the x2 condition, the kernel strips away Extended Tags and forces a 128B MRRS restriction across that ENTIRE host bridge. This instantly breaks or starves adjacent high-performance, unrelated x4 endpoints (such as NVMe drives), resulting in controller timeouts, initialization failures, and missing drives at boot.

Quoted 8 lines

Fix this by refactoring the quirk logic to be completely per-device and downstream-isolated. Remove the broad host bridge structure assignments entirely. Instead, convert the hook to an explicit 'DECLARE_PCI_FIXUP_FINAL' sweep. When an x2 bifurcated root port is discovered, leverage 'pci_walk_bus()' to step down only that specific root port's subordinate tree, manually clearing 'PCI_EXP_DEVCTL_EXT_TAG' and 'PCI_EXP_DEVCTL_READRQ' directly in the endpoint Device Control configuration registers.

One problem I see with this approach is that it won't work for hotplugged devices below a bifurcated Root Port: You're only adjusting Extended Tags and MRRS once on enumeration of the Root Ports, leaving devices that are hotplugged later at incorrect settings.

Quoted 6 lines

But perhaps you could simply amend limit_mrrs_to_128() to walk up to the Root Port, check whether it is bifurcated, and bail out if it's not?

Thanks,

Lukas

arch/x86/pci/fixup.c +46 -12
index b301c6c8df75..0ab92a5cc0d1 100644
@@ -301,15 +301,6 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	PCI_DEVICE_ID_INTEL_MCH_PC1,	pcie_r
  *
  * https://cdrdv2.intel.com/v1/dl/getContent/837176
  */
-static int limit_mrrs_to_128(struct pci_host_bridge *b, struct pci_dev *pdev)
-{
-	int readrq = pcie_get_readrq(pdev);
-
-	if (readrq > 128)
-		pcie_set_readrq(pdev, 128);
-
-	return 0;
-}
 
 static void pci_xeon_x2_bifurc_quirk(struct pci_dev *pdev)
 {
@@ -320,9 +311,8 @@ static void pci_xeon_x2_bifurc_quirk(struct pci_dev *pdev)
 	if (FIELD_GET(PCI_EXP_LNKCAP_MLW, linkcap) != 0x2)
 		return;
 
-	bridge->no_ext_tags = 1;
-	bridge->enable_device = limit_mrrs_to_128;
-	pci_info(pdev, "Disabling Extended Tags and limiting MRRS to 128B (performance reasons due to x2 PCIe link)\n");
+	bridge->no_inc_mrrs = 1;
+	pci_info(pdev, "Blocking devices on this bridge from increasing MRRS for performance reasons due to x2 PCIe link)\n");
 }
 
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db0, pci_xeon_x2_bifurc_quirk);
@@ -334,6 +324,50 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db7, pci_xeon_x2_bifurc_quirk);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db8, pci_xeon_x2_bifurc_quirk);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0db9, pci_xeon_x2_bifurc_quirk);
 
+/* Helper to check if a device descends from an affected Xeon 6 x2 Root Port */
+static bool is_descendant_of_xeon6_x2_rp(struct pci_dev *pdev)
+{
+	u32 linkcap;
+	struct pci_dev *upstream = pci_upstream_bridge(pdev);
+
+	while (upstream) {
+		if (upstream->vendor == PCI_VENDOR_ID_INTEL &&
+		    upstream->device >= 0x0db0 &&
+		    upstream->device <= 0x0db9 ) {
+			pcie_capability_read_dword(upstream, PCI_EXP_LNKCAP, &linkcap);
+			if (FIELD_GET(PCI_EXP_LNKCAP_MLW, linkcap) != 0x2)
+				return false;
+			else
+				return true;
+		}
+		upstream = pci_upstream_bridge(upstream);
+	}
+	return false;
+}
+
+static void pci_xeon6_x2_local_endpoint_fixup(struct pci_dev *pdev)
+{
+	/* Skip bridges/switches; only target actual endpoints */
+	if (pci_is_bridge(pdev))
+		return;
+
+	/* Only apply to devices under the x2 branch; leaves x4 branches completely untouched */
+	if (!is_descendant_of_xeon6_x2_rp(pdev))
+		return;
+
+	pci_info(pdev, "Applying local Xeon 6 x2 quirk: Disabling Extended Tags and locking MRRS to 128B\n");
+
+	pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_EXT_TAG);
+	pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_READRQ);
+}
+
+/*
+ * HEADER fixups run for EVERY endpoint during its initial discovery phase.
+ * This natively catches boot devices, hotplugged devices, and SR-IOV VFs.
+ */
+DECLARE_PCI_FIXUP_HEADER(PCI_ANY_ID, PCI_ANY_ID, pci_xeon6_x2_local_endpoint_fixup);
+
+
 /*
  * Fixup to mark boot BIOS video selected by BIOS before it changes
  *
#4 lore raw
Quoted 3 lines · Rick Warner

Hi All,

Here's an updated version of the fix.

Can you resend this as a v2 of the patch (not a reply to the original post), with its own commit log, please? Be sure to cc Lukas and anybody else who shows interest in it.

#5 lore raw
In reply to Rick Warner

Hi maintainers,

Quoted 11 lines · Rick Warner

Hi All,

Here's an updated version of the fix. Now in the early setup it sets no_inc_mrrs on the entire bridge instead of disabling ext tags and limiting mrrs for the entire root bridge. Then it uses DECLARE_PCI_FIXUP_HEADER to check every registering pci device to see if it's a descendant of an x2 port, and if so, disables ext tags and limits mrrs for that device. This should properly handle hotplugging now.

This has been tested on a Gigabyte MS74-HB0 motherboard with a Seagate ZP4000GM30063 M.2 drive. With the stock kernel, that drive fails to initialize and is unavailable once booted. Testing showed that extended tags are required for it to work. The drive works properly with this patch. Concerns were mentioned about vfio/VM usage being able to re-enable extended tags if bridge->no_ext_tags is not set. I'm not sure how to address that without adding additional a flag to pci device structures that individually blocks extended tags per device instead of relying on the bridge flag for it. I'm open to suggestions for that.

Thanks,
Rick Warner

Hope it’s OK to reply here rather than start a new thread/new bug - this sets context to the issue I’m having.

Both the original commit and the proposed updated patch causes the two on-board M.2 NVMe drives on my system to fail initialization. It is a Supermicro X14SBI-F motherboard with 2x Kingston DC2000B 240GB drives. Both drives fail to detect under any 7.x kernel (6.13.x works, but the patch wasn’t added there) - I generally see this during boot:

[    9.382875] nvme nvme0: pci function 0000:2c:00.0
[    9.383490] nvme nvme1: pci function 0000:2d:00.0
[    9.899466] nvme nvme0: invalid id 0 completed on queue 0
[    9.904491] nvme nvme1: invalid id 0 completed on queue 0
….skipped for brevity….
[   10.025904] workqueue: local_pci_probe_callback hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[   10.113726] ice 0000:7a:00.1: DDP package already present on device: ICE OS Default Package version 1.3.43.0
[   10.148629] ice 0000:7a:00.1: 126.024 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x8 link)
[   10.152913] ice 0000:7a:00.1: PTP init successful
[   10.183499] ice 0000:7a:00.1: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[   10.183629] ice 0000:7a:00.1: FW LLDP is disabled, DCBx/LLDP in SW mode.
[   10.186568] ice 0000:7a:00.1: Commit DCB Configuration to the hardware
[   71.086147] nvme nvme0: I/O tag 28 (001c) QID 0 timeout, disable controller
[   71.090916] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable controller
[   71.095029] nvme nvme0: Identify Controller failed (-4)
[   71.098861] nvme nvme1: Identify Controller failed (-4)
[   71.114292] nvme 0000:2d:00.0: error -EIO: probe failed
[   71.115011] nvme 0000:2c:00.0: error -EIO: probe failed
[   71.115016] nvme 0000:2d:00.0: probe with driver nvme failed with error -5
[   71.115668] nvme 0000:2c:00.0: probe with driver nvme failed with error -5

As indicated by the timestamps, there is approximately 60 seconds of delay at every boot where I am assuming the kernel is waiting for some kind of timeout. Nothing is going on in this period of time. To test this out in detail, I started out with Proxmox kernels, (the system is going to run Proxmox) but to be able to report an upstream bug report, I switched to compiling my own kernel based on the GIT sources on git.kernel.org. I compiled three different versions:

- Stock 7.3.1 kernel
- Pathced 7.3.1 kernel with the proposed patch
- Patched 7.3.1 kernel where I reversed the original Xeon 6 x2 PCIe quirk

I am having the issue with either version of the quirk enabled but things are working fine with the quirk removed. The only apparent difference between the two versions of the quirk is that dmesg contains log entries stating that it is disabling extended tags and locking MRRS to 128B in the proposed patch.

As mentioned, the system runs on a motherboard Supermicro X14SBI-F with a Xeon 6 CPU (6511P), and I believe there’s an important detail here: Supermicro purposely have wired only 2 PCIe lanes to each of the M.2 slots so both the NVMe drives are being targeted by the quirk. Whether it is because the want extended tags enabled or because of the MRRS clamping is unknown to me, but you mention that you had a drive that expected Extended Tags to be enabled so it is likely to be same case here.

Looking at the motherboard block diagram, Supermicro decided to bifurcate a x8 root port into x4x2x2 (the x4 is attached to an on-board ASMedia SATA controller that I am not using, so can’t say if that one is working).

I tried two methods of getting the drives to initialize correctly. I wrote a couple of scripts that I ran from the initramfs environment. The first attempts to unbind and rebind the driver. Since the controller has been disabled by the kernel, it doesn’t work, it just costs 2x60-ish seconds of waiting time:

[  120.935727] NVMe diagnostic script: Attempting unbind and rebind of PCIe/NVMe drives from NVMe driver.
[  120.936403] NVMe diagnostic script: Unbinding NVMe drive #1 (PCIe address 0000:2c:00.0): echo 0000:2c:00.0 > /sys/bus/pci/drivers/nvme/unbind
[  120.937407] NVMe diagnostic script: Unbinding NVMe drive #2 (PCIe address 0000:2d:00.0): echo 0000:2d:00.0 > /sys/bus/pci/drivers/nvme/unbind
[  122.940614] NVMe diagnostic script: Binding NVMe drive #1 (PCIe address 0000:2c:00.0): echo 0000:2c:00.0 > /sys/bus/pci/drivers/nvme/bind
[  122.941552] nvme nvme0: pci function 0000:2c:00.0
[  123.457755] could not locate request for tag 0x0
[  123.458208] nvme nvme0: invalid id 0 completed on queue 0
[  185.770092] nvme nvme0: I/O tag 12 (000c) QID 0 timeout, disable controller
[  185.773377] nvme nvme0: Device not ready; aborting shutdown, CSTS=0x1
[  185.779037] nvme nvme0: Identify Controller failed (-4)
[  185.790284] nvme 0000:2c:00.0: error -EIO: probe failed
[  185.790840] nvme 0000:2c:00.0: probe with driver nvme failed with error -5
[  185.791991] NVMe diagnostic script: Binding NVMe drive #2 (PCIe address 0000:2d:00.0): echo 0000:2d:00.0 > /sys/bus/pci/drivers/nvme/bind
[  185.807432] nvme nvme0: pci function 0000:2d:00.0
[  186.324847] could not locate request for tag 0x0
[  186.325336] nvme nvme0: invalid id 0 completed on queue 0
[  247.210215] nvme nvme0: I/O tag 20 (0014) QID 0 timeout, disable controller
[  247.213527] nvme nvme0: Device not ready; aborting shutdown, CSTS=0x1
[  247.219036] nvme nvme0: Identify Controller failed (-4)
[  247.230260] nvme 0000:2d:00.0: error -EIO: probe failed
[  247.230860] nvme 0000:2d:00.0: probe with driver nvme failed with error -5

The other script requests a reset of the device and then attempts to bind the device to the NVMe driver, and this works every time I have tried across kernels (I tried pre-compiled versions of the latest version in 7.0, 7.1, 7.2, and 7.3):

[  332.974791] NVMe diagnostic script: Attempting PCIe reset and subsequent NVMe driver binding of PCIe/NVMe drives
[  332.977630] NVMe diagnostic script: Supported reset methods for PCI device 0000:2c:00.0: flr bus cxl_bus
[  332.979836] NVMe diagnostic script: Supported reset methods for PCI device 0000:2d:00.0: flr bus
[  332.980365] NVMe diagnostic script: Requesting reset of device 2c:00.0: echo 1 > /sys/bus/pci/devices/0000:2c:00.0/reset
[  333.086113] NVMe diagnostic script: Requesting reset of device 2d:00.0: echo 1 > /sys/bus/pci/devices/0000:2d:00.0/reset
[  333.190118] NVMe diagnostic script: Binding device 2c:00.0 to NVMe driver: echo 0000:2c:00.0 > /sys/bus/pci/drivers/nvme/bind
[  333.191601] nvme nvme0: pci function 0000:2c:00.0
[  333.195990] nvme nvme0: D3 entry latency set to 10 seconds
[  333.197296] nvme nvme0: passthrough uses implicit buffer lengths
[  333.208479] nvme nvme0: 32/0/0 default/read/poll queues
[  333.220645]  nvme0n1: p2 p3
[  333.222686] NVMe diagnostic script: Binding device 2d:00.0 to NVMe driver: echo 0000:2d:00.0 > /sys/bus/pci/drivers/nvme/bind
[  333.223079] nvme nvme1: pci function 0000:2d:00.0
[  333.225612] nvme nvme1: D3 entry latency set to 10 seconds
[  333.226441] nvme nvme1: passthrough uses implicit buffer lengths
[  333.229506] nvme nvme1: 32/0/0 default/read/poll queues
[  333.233849]  nvme1n1: p2 p3

As the timestamps show the initialization is successful and happens instantly. I do have the full dmesg logs, but I am unsure of what the policy on attaching files to email is. My logs have been captured in a initramfs environment so I am somewhat limited to what is available by default there.

The drives are bootdisks of the machine so it is not in operation yet, making it a perfect target for tests. Just let me know what to do.

Here are relevant sections of lspci -nn and lspci -vvv (the drives are currently limited to 3.0/8GT/s speeds - I tried lowering it in the UEFI to see if that made a difference which it didn’t - they will normally operate at 4.0/16GT/s speeds):

2c:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC] [2646:5024] (rev 01) 2d:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC] [2646:5024] (rev 01) 2b:06.0 PCI bridge [0604]: Intel Corporation Device [8086:0db2]
2b:07.0 PCI bridge [0604]: Intel Corporation Device [8086:0db8]

2c:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC] (rev 01) (prog-if 02 [NVM Express])
Subsystem: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC]
Physical Slot: 4051
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
NUMA node: 0
IOMMU group: 71
Region 0: Memory at a7600000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-
AtomicOpsCtl: ReqEn+
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: Upstream Port
Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [f8] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Latency Tolerance Reporting
Max snoop latency: 40960ns
Max no snoop latency: 40960ns
Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1e0 v1] Data Link Feature <?>
Capabilities: [200 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+
ECRC+ UnsupReq+ ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ CorrIntErr+ HeaderOF-
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04080001 fc00000f 2cf80000 ffc9cab6
Capabilities: [290 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [2a0 v1] Power Budgeting <?>
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [340 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [378 v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady- MargSoftReady-
Kernel driver in use: nvme
Kernel modules: nvme

2d:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC] (rev 01) (prog-if 02 [NVM Express])
Subsystem: Kingston Technology Company, Inc. DC2000B NVMe SSD [E18DC]
Physical Slot: 4050
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
NUMA node: 0
IOMMU group: 72
Region 0: Memory at a7500000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-
AtomicOpsCtl: ReqEn+
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: Upstream Port
Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [f8] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Latency Tolerance Reporting
Max snoop latency: 40960ns
Max no snoop latency: 40960ns
Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1e0 v1] Data Link Feature <?>
Capabilities: [200 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+
ECRC+ UnsupReq+ ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ CorrIntErr+ HeaderOF-
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04080001 fc00000f 2df80000 9aae760e
Capabilities: [290 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [2a0 v1] Power Budgeting <?>
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [340 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [378 v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady- MargSoftReady-
Kernel driver in use: nvme
Kernel modules: nvme

2b:06.0 PCI bridge: Intel Corporation Device 0db2 (prog-if 00 [Normal decode])
Subsystem: Intel Corporation Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 52
NUMA node: 0
IOMMU group: 67
Region 0: Memory at 21fffff30000 (64-bit, non-prefetchable) [size=64K]
Bus: primary=2b, secondary=2c, subordinate=2c, sec-latency=0
I/O behind bridge: f000-0fff [disabled] [16-bit]
Memory behind bridge: a7600000-a76fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), IntMsgNum 0
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+ TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 32GT/s, Width x2, ASPM not supported
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt+ AutBWInt+
LnkSta: Speed 8GT/s, Width x2
TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #4051, PowerLimit 25W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP+ LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 2 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd+
AtomicOpsCtl: ReqEn+ EgressBlck-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [94] Subsystem: Intel Corporation Device 0000
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable+ 64bit-
Address: fee00038 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+
ECRC+ UnsupReq+ ACSViol+ UncorrIntErr+ BlockedTLP- AtomicOpBlocked+ TLPBlockedErr+ PoisonTLPBlocked- DMWrReqBlocked+ IDECheck+ MisIDETLP+ PCRC_CHECK+ TLPXlatBlocked-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ CorrIntErr+ HeaderOF+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap+
HeaderLog: 60000004 2c0047ff 00000001 369bbef0
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [220 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [190 v1] Vendor Specific Information: ID=0003 Rev=0 Len=00a <?>
Capabilities: [1a0 v1] Downstream Port Containment
DpcCap: IntMsgNum 0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
DpcCtl: Trigger:2 Cmpl- INT+ ErrCor+ PoisonedTLP+ SwTrigger- DL_ActiveErr+
DpcSta: Trigger- Reason:00 INT- RPBusy- TriggerExt:00 RP PIO ErrPtr:1f
Source: 0000
Capabilities: [150 v1] Precision Time Measurement
PTMCap: Requester- Responder+ Root+
PTMClockGranularity: 2ns
PTMControl: Enabled+ RootSelected+
PTMEffectiveGranularity: 2ns
Capabilities: [160 v1] Extended Capability ID 0x2f
Capabilities: [280 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable- ID=0 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [200 v1] Native PCIe Enclosure Management <?>
Capabilities: [a30 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [a90 v1] Data Link Feature <?>
Capabilities: [a9c v1] Physical Layer 16.0 GT/s <?>
Capabilities: [ae0 v1] Physical Layer 32.0 GT/s <?>
Capabilities: [edc v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady- MargSoftReady-
Capabilities: [304 v1] Integrity & Data Encryption
IDECap: Lnk=1 Sel=1 FlowThru- PartHdr- Aggr- PCPC+ IDE_KM- Alg='AES-GCM-256-96b' TCs=1 TeeLim-
IDECtl: FTEn-
[310]LinkIDE#0 Ctl: En- NPR- PR- CPL- PCRC- HdrEnc=no Alg='AES-GCM-256-96b' TC0 ID0 [314]LinkIDE#0 Sta: Status=insecure RecvChkFail-
[318]SelectiveIDE#0 Cap: RID#=1
[31c]SelectiveIDE#0 Ctl: En- NPR- PR- CPL- PCRC- CFG- HdrEnc=no Alg='AES-GCM-256-96b' TC0 ID0 [320]SelectiveIDE#0 Sta: insecure RecvChkFail-
[324]SelectiveIDE#0 RID: Valid- Base=0 Limit=0 SegBase=0
[32c]SelectiveIDE#0 RID#0: Valid- Base=0 Limit=fffff
Capabilities: [b20 v1] Extended Capability ID 0x2b
Capabilities: [e40 v1] Designated Vendor-Specific: Vendor=1e98 ID=0008 Rev=0 Len=20: CXL
Register Locator DVSEC
Block1: BIR: bar0, ID: component registers, offset: 0000000000000000
Capabilities: [e60 v1] Designated Vendor-Specific: Vendor=1e98 ID=0004 Rev=0 Len=16: CXL
GPF DVSEC for CXL Ports
GPF Phase 1 Timeout: 0us
GPF Phase 2 Timeout: 0us
Capabilities: [e70 v1] Designated Vendor-Specific: Vendor=1e98 ID=0003 Rev=0 Len=40: CXL
CXL Extensions DVSEC for Ports
CXLPortSta: PMComplete-
CXLPortCtl: UnmaskSBR- UnmaskLinkDisable- AltMem- AltBME- ViralEnable-
AlternateBus: 00-00
AlternateBus: 0000-0000
Capabilities: [ea4 v1] Designated Vendor-Specific: Vendor=1e98 ID=0007 Rev=1 Len=20: CXL
PCIe DVSEC for Flex Bus Port
FBCap: Cache+ IO+ Mem+ 68BFlit+ MltLogDev-
FBCtl: Cache- IO+ Mem- SynHdrByp- DrftBuf- 68BFlit- MltLogDev- RCD+ Retimer1- Retimer2-
FBSta: Cache- IO- Mem- SynHdrByp- DrftBuf- 68BFlit- MltLogDev-
FBModTS: Received FB Data: 000000
Kernel driver in use: pcieport

2b:07.0 PCI bridge: Intel Corporation Device 0db8 (prog-if 00 [Normal decode])
Subsystem: Intel Corporation Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 53
NUMA node: 0
IOMMU group: 68
Region 0: Memory at 21fffff20000 (64-bit, non-prefetchable) [size=64K]
Bus: primary=2b, secondary=2d, subordinate=2d, sec-latency=0
I/O behind bridge: f000-0fff [disabled] [16-bit]
Memory behind bridge: a7500000-a75fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), IntMsgNum 0
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+ TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #6, Speed 32GT/s, Width x2, ASPM not supported
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt+ AutBWInt+
LnkSta: Speed 8GT/s, Width x2
TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #4050, PowerLimit 25W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP+ LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd+
AtomicOpsCtl: ReqEn+ EgressBlck-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [94] Subsystem: Intel Corporation Device 0000
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable+ 64bit-
Address: fee00058 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+
ECRC+ UnsupReq+ ACSViol+ UncorrIntErr+ BlockedTLP- AtomicOpBlocked+ TLPBlockedErr+ PoisonTLPBlocked- DMWrReqBlocked+ IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr- PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+ CorrIntErr+ HeaderOF+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap+
HeaderLog: 4a080001 2d000004 fc080000 00000000
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [220 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [190 v1] Vendor Specific Information: ID=0003 Rev=0 Len=00a <?>
Capabilities: [1a0 v1] Downstream Port Containment
DpcCap: IntMsgNum 0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
DpcCtl: Trigger:2 Cmpl- INT+ ErrCor+ PoisonedTLP+ SwTrigger- DL_ActiveErr+
DpcSta: Trigger- Reason:00 INT- RPBusy- TriggerExt:00 RP PIO ErrPtr:1f
Source: 0000
Capabilities: [150 v1] Precision Time Measurement
PTMCap: Requester- Responder+ Root+
PTMClockGranularity: 2ns
PTMControl: Enabled+ RootSelected+
PTMEffectiveGranularity: 2ns
Capabilities: [160 v1] Extended Capability ID 0x2f
Capabilities: [280 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable- ID=0 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [200 v1] Native PCIe Enclosure Management <?>
Capabilities: [a30 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [a90 v1] Data Link Feature <?>
Capabilities: [a9c v1] Physical Layer 16.0 GT/s <?>
Capabilities: [ae0 v1] Physical Layer 32.0 GT/s <?>
Capabilities: [edc v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady- MargSoftReady-
Kernel driver in use: pcieport

Thanks,
Mark

Quoted 38 lines · Lukas Wunner
Quoted 30 lines · Rick Warner
Quoted 16 lines

commit a22250fe933d ("PCI: Add Extended Tag + MRRS quirk for Xeon 6") introduced a traffic mitigation quirk for Intel Xeon 6 root ports that negotiate down to an x2 lane width. However, that patch manipulated host bridge properties globally via 'bridge->no_ext_tags = 1' and by hooking 'bridge->enable_device'.

Because multiple unrelated root ports can reside under the exact same global pci_host_bridge domain structure, this overly aggressive mitigation causes severe collateral damage. When a low-speed or bifurcated secondary device (such as an onboard ASMedia SATA controller or BMC graphics link) matches the x2 condition, the kernel strips away Extended Tags and forces a 128B MRRS restriction across that ENTIRE host bridge. This instantly breaks or starves adjacent high-performance, unrelated x4 endpoints (such as NVMe drives), resulting in controller timeouts, initialization failures, and missing drives at boot.

Quoted 8 lines

Fix this by refactoring the quirk logic to be completely per-device and downstream-isolated. Remove the broad host bridge structure assignments entirely. Instead, convert the hook to an explicit 'DECLARE_PCI_FIXUP_FINAL' sweep. When an x2 bifurcated root port is discovered, leverage 'pci_walk_bus()' to step down only that specific root port's subordinate tree, manually clearing 'PCI_EXP_DEVCTL_EXT_TAG' and 'PCI_EXP_DEVCTL_READRQ' directly in the endpoint Device Control configuration registers.

One problem I see with this approach is that it won't work for hotplugged devices below a bifurcated Root Port: You're only adjusting Extended Tags and MRRS once on enumeration of the Root Ports, leaving devices that are hotplugged later at incorrect settings.

Quoted 6 lines

But perhaps you could simply amend limit_mrrs_to_128() to walk up to the Root Port, check whether it is bifurcated, and bail out if it's not?

Thanks,

Lukas

<x2-quirk-disable-extended-tags-and-limit-mrrs-per-device.patch>