Skip to contentSkip to conversation

PCI: Remove unsupported or incomplete PCIe Capabilities

PATCH Applied Qiang Yu 7 participants 28 messages Nov 10, 2025 – Jan 9

Subsystems (6)
lore
Share & tools
lore

Qiang Yu

author
#0 lore raw

cover PCI: Remove unsupported or incomplete PCIe Capabilities

This patch series addresses issues where certain PCIe Standard or Extended Capabilities are advertised by the controller, but not fully or at all implemented in hardware. Exposing such capabilities to the PCI framework can lead to unexpected or undefined behavior.

The series consists of two main parts:

1. Infrastructure patches (1-2): Add generic capability removal support
2. Platform-specific patches(3-5): Apply fixes for DWC based PCIe Root Ports and Qualcomm PCIe Root Ports

Patch 1 extends the existing PCI capability finding macros to optionally return the position of the preceding capability. This information is essential for capability removal operations, as it allows updating the "next" pointer of the predecessor to skip over the capability being removed.

Patch 2 introduces two new APIs in the DesignWare PCIe controller driver:
- dw_pcie_remove_capability() - Remove standard PCI capabilities
- dw_pcie_remove_ext_capability() - Remove extended PCI capabilities

Patch 3 implements automatic MSI/MSI-X capability removal for Root Ports when using the iMSI-RX module as MSI controller. This addresses the issue where iMSI-RX cannot generate MSI interrupts for Root Ports, forcing a fallback to INTx interrupts for Root Ports.

Patch 4 removes MSI-X capability from Qualcomm Root Ports unconditionally. On platforms like Glymur, MSI-X capability is exposed but lacks the required Table and PBA structures, leading to SMMU faults when the capability is used.

Patch 5 removes DPC (Downstream Port Containment) Extended Capability from Qualcomm platforms. While the capability registers are present, DPC is not fully supported, and exposing it can cause undefined behavior.

---
Qiang Yu (5):
PCI: Add preceding capability position support and update drivers
PCI: dwc: Add new APIs to remove standard and extended Capability
PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller
PCI: qcom: Remove MSI-X Capability for Root Ports
PCI: qcom: Remove DPC Extended Capability

 drivers/pci/controller/cadence/pcie-cadence.c     |  4 +-
 drivers/pci/controller/dwc/pcie-designware-ep.c   |  2 +-
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++
 drivers/pci/controller/dwc/pcie-designware.c      | 59 +++++++++++++++++++++--
 drivers/pci/controller/dwc/pcie-designware.h      |  2 +
 drivers/pci/controller/dwc/pcie-qcom.c            |  3 ++
 drivers/pci/pci.c                                 |  8 +--
 drivers/pci/pci.h                                 | 23 +++++++--
 8 files changed, 97 insertions(+), 14 deletions(-)

---
base-commit: 3a8660878839faadb4f1a6dd72c3179c1df56787
change-id: 20251109-remove_cap-4b7c0600f766

Best regards,

Signature · 1 line
Qiang Yu <[email protected]>
author
#1 lore raw

PATCH 1/5 PCI: Add preceding capability position support and update drivers

Add support for finding the preceding capability position in PCI capability list by extending the capability finding macros with an additional parameter. This functionality is essential for modifying PCI capability list, as it provides the necessary information to update the "next" pointer of the predecessor capability when removing entries.

Modify two macros to accept a new 'prev_ptr' parameter:
- PCI_FIND_NEXT_CAP - Now accepts 'prev_ptr' parameter for standard capabilities
- PCI_FIND_NEXT_EXT_CAP - Now accepts 'prev_ptr' parameter for extended capabilities

When a capability is found, these macros:
- Store the position of the preceding capability in *prev_ptr (if prev_ptr != NULL)
- Maintain all existing functionality when prev_ptr is NULL

Update current callers to accommodate this API
change:
- Cadence PCIe controller: Pass NULL to cdns_pcie_find_capability() and cdns_pcie_find_ext_capability() calls
- DesignWare PCIe controller: Pass NULL to all capability finding macro calls

The drivers pass NULL as the prev_ptr parameter since they don't require knowledge of the preceding capability position for their current functionality. This ensures backward compatibility while enabling future capability list manipulation features.

No functional changes to driver behavior result from this patch - it maintains the existing capability finding functionality while adding the infrastructure for future capability removal operations.

---
 drivers/pci/controller/cadence/pcie-cadence.c   |  4 ++--
 drivers/pci/controller/dwc/pcie-designware-ep.c |  2 +-
 drivers/pci/controller/dwc/pcie-designware.c    |  6 +++---
 drivers/pci/pci.c                               |  8 ++++----
 drivers/pci/pci.h                               | 23 +++++++++++++++++++----
 5 files changed, 29 insertions(+), 14 deletions(-)
drivers/pci/controller/cadence/pcie-cadence.c +2 -2
index bd683d0fecb225f2134893faa7199d659157b3f1..d614452861f7755108b1220527dc277a3754a76d 100644
@@ -13,13 +13,13 @@
 u8 cdns_pcie_find_capability(struct cdns_pcie *pcie, u8 cap)
 {
 	return PCI_FIND_NEXT_CAP(cdns_pcie_read_cfg, PCI_CAPABILITY_LIST,
-				 cap, pcie);
+				 cap, NULL, pcie);
 }
 EXPORT_SYMBOL_GPL(cdns_pcie_find_capability);
 
 u16 cdns_pcie_find_ext_capability(struct cdns_pcie *pcie, u8 cap)
 {
-	return PCI_FIND_NEXT_EXT_CAP(cdns_pcie_read_cfg, 0, cap, pcie);
+	return PCI_FIND_NEXT_EXT_CAP(cdns_pcie_read_cfg, 0, cap, NULL, pcie);
 }
 EXPORT_SYMBOL_GPL(cdns_pcie_find_ext_capability);
 
drivers/pci/controller/dwc/pcie-designware-ep.c +1 -1
index 7f2112c2fb21543d11e848a4a62d529cc3f4e8d0..e62665e228b9035f35c441bffd2a5759ecf12546 100644
@@ -72,7 +72,7 @@ EXPORT_SYMBOL_GPL(dw_pcie_ep_reset_bar);
 static u8 dw_pcie_ep_find_capability(struct dw_pcie_ep *ep, u8 func_no, u8 cap)
 {
 	return PCI_FIND_NEXT_CAP(dw_pcie_ep_read_cfg, PCI_CAPABILITY_LIST,
-				 cap, ep, func_no);
+				 cap, NULL, ep, func_no);
 }
 
 /**
drivers/pci/controller/dwc/pcie-designware.c +3 -3
index c644216995f69cbf065e61a0392bf1e5e32cf56e..5585d3ed74316bd218572484f6320019db8d6a10 100644
@@ -224,13 +224,13 @@ void dw_pcie_version_detect(struct dw_pcie *pci)
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap)
 {
 	return PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
-				 pci);
+				 NULL, pci);
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_capability);
 
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 {
-	return PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, pci);
+	return PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, NULL, pci);
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
@@ -244,7 +244,7 @@ static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 		return 0;
 
 	while ((vsec = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, vsec,
-					     PCI_EXT_CAP_ID_VNDR, pci))) {
+					     PCI_EXT_CAP_ID_VNDR, NULL, pci))) {
 		header = dw_pcie_readl_dbi(pci, vsec + PCI_VNDR_HEADER);
 		if (PCI_VNDR_HEADER_ID(header) == vsec_id)
 			return vsec;
drivers/pci/pci.c +4 -4
index b14dd064006cca80ec5275e45a35d6dc2b4d0bbc..83e3252f6691f2289ab5cacc3346a37c69a13d59 100644
@@ -426,7 +426,7 @@ static int pci_dev_str_match(struct pci_dev *dev, const char *p,
 static u8 __pci_find_next_cap(struct pci_bus *bus, unsigned int devfn,
 			      u8 pos, int cap)
 {
-	return PCI_FIND_NEXT_CAP(pci_bus_read_config, pos, cap, bus, devfn);
+	return PCI_FIND_NEXT_CAP(pci_bus_read_config, pos, cap, NULL, bus, devfn);
 }
 
 u8 pci_find_next_capability(struct pci_dev *dev, u8 pos, int cap)
@@ -531,7 +531,7 @@ u16 pci_find_next_ext_capability(struct pci_dev *dev, u16 start, int cap)
 		return 0;
 
 	return PCI_FIND_NEXT_EXT_CAP(pci_bus_read_config, start, cap,
-				     dev->bus, dev->devfn);
+				     NULL, dev->bus, dev->devfn);
 }
 EXPORT_SYMBOL_GPL(pci_find_next_ext_capability);
 
@@ -600,7 +600,7 @@ static u8 __pci_find_next_ht_cap(struct pci_dev *dev, u8 pos, int ht_cap)
 		mask = HT_5BIT_CAP_MASK;
 
 	pos = PCI_FIND_NEXT_CAP(pci_bus_read_config, pos,
-				PCI_CAP_ID_HT, dev->bus, dev->devfn);
+				PCI_CAP_ID_HT, NULL, dev->bus, dev->devfn);
 	while (pos) {
 		rc = pci_read_config_byte(dev, pos + 3, &cap);
 		if (rc != PCIBIOS_SUCCESSFUL)
@@ -611,7 +611,7 @@ static u8 __pci_find_next_ht_cap(struct pci_dev *dev, u8 pos, int ht_cap)
 
 		pos = PCI_FIND_NEXT_CAP(pci_bus_read_config,
 					pos + PCI_CAP_LIST_NEXT,
-					PCI_CAP_ID_HT, dev->bus,
+					PCI_CAP_ID_HT, NULL, dev->bus,
 					dev->devfn);
 	}
 
drivers/pci/pci.h +19 -4
index 4492b809094b5794bd94dfbc20102cb208c3fa2f..2a33356acc2e8cfe55801bcb7d0cd6d2a336561b 100644
@@ -103,17 +103,21 @@ bool pcie_cap_has_rtctl(const struct pci_dev *dev);
  * @read_cfg: Function pointer for reading PCI config space
  * @start: Starting position to begin search
  * @cap: Capability ID to find
+ * @prev_ptr: Pointer to store position of preceding capability (optional)
  * @args: Arguments to pass to read_cfg function
  *
- * Search the capability list in PCI config space to find @cap.
+ * Search the capability list in PCI config space to find @cap. If
+ * found, update *prev_ptr with the position of the preceding capability
+ * (if prev_ptr != NULL)
  * Implements TTL (time-to-live) protection against infinite loops.
  *
  * Return: Position of the capability if found, 0 otherwise.
  */
-#define PCI_FIND_NEXT_CAP(read_cfg, start, cap, args...)		\
+#define PCI_FIND_NEXT_CAP(read_cfg, start, cap, prev_ptr, args...)	\
 ({									\
 	int __ttl = PCI_FIND_CAP_TTL;					\
-	u8 __id, __found_pos = 0;					\
+	u8 __id,  __found_pos = 0;					\
+	u8 __prev_pos = (start);					\
 	u8 __pos = (start);						\
 	u16 __ent;							\
 									\
@@ -132,9 +136,12 @@ bool pcie_cap_has_rtctl(const struct pci_dev *dev);
 									\
 		if (__id == (cap)) {					\
 			__found_pos = __pos;				\
+			if (prev_ptr != NULL)				\
+				*(u8 *)prev_ptr = __prev_pos;		\
 			break;						\
 		}							\
 									\
+		__prev_pos = __pos;					\
 		__pos = FIELD_GET(PCI_CAP_LIST_NEXT_MASK, __ent);	\
 	}								\
 	__found_pos;							\
@@ -146,21 +153,26 @@ bool pcie_cap_has_rtctl(const struct pci_dev *dev);
  * @read_cfg: Function pointer for reading PCI config space
  * @start: Starting position to begin search (0 for initial search)
  * @cap: Extended capability ID to find
+ * @prev_ptr: Pointer to store position of preceding capability (optional)
  * @args: Arguments to pass to read_cfg function
  *
  * Search the extended capability list in PCI config space to find @cap.
+ * If found, update *prev_ptr with the position of the preceding capability
+ * (if prev_ptr != NULL)
  * Implements TTL protection against infinite loops using a calculated
  * maximum search count.
  *
  * Return: Position of the capability if found, 0 otherwise.
  */
-#define PCI_FIND_NEXT_EXT_CAP(read_cfg, start, cap, args...)		\
+#define PCI_FIND_NEXT_EXT_CAP(read_cfg, start, cap, prev_ptr, args...)	\
 ({									\
 	u16 __pos = (start) ?: PCI_CFG_SPACE_SIZE;			\
 	u16 __found_pos = 0;						\
+	u16 __prev_pos;							\
 	int __ttl, __ret;						\
 	u32 __header;							\
 									\
+	__prev_pos = __pos;						\
 	__ttl = (PCI_CFG_SPACE_EXP_SIZE - PCI_CFG_SPACE_SIZE) / 8;	\
 	while (__ttl-- > 0 && __pos >= PCI_CFG_SPACE_SIZE) {		\
 		__ret = read_cfg##_dword(args, __pos, &__header);	\
@@ -172,9 +184,12 @@ bool pcie_cap_has_rtctl(const struct pci_dev *dev);
 									\
 		if (PCI_EXT_CAP_ID(__header) == (cap) && __pos != start) {\
 			__found_pos = __pos;				\
+			if (prev_ptr != NULL)				\
+				*(u16 *)prev_ptr = __prev_pos;		\
 			break;						\
 		}							\
 									\
+		__prev_pos = __pos;					\
 		__pos = PCI_EXT_CAP_NEXT(__header);			\
 	}								\
 	__found_pos;							\
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/5 PCI: dwc: Add new APIs to remove standard and extended Capability

In reply to Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
drivers/pci/controller/dwc/pcie-designware.c +53 -0
index 5585d3ed74316bd218572484f6320019db8d6a10..24f8e9959cb81ca41e91d27057cc115d32e8d523 100644
@@ -234,6 +234,59 @@ u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap)
+{
+	u8 cap_pos, pre_pos, next_pos;
+	u16 reg;
+
+	cap_pos = PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
+				 &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	reg = dw_pcie_readw_dbi(pci, cap_pos);
+	next_pos = (reg & 0xff00) >> 8;
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	if (pre_pos == PCI_CAPABILITY_LIST)
+		dw_pcie_writeb_dbi(pci, PCI_CAPABILITY_LIST, next_pos);
+	else
+		dw_pcie_writeb_dbi(pci, pre_pos + 1, next_pos);
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_capability);
+
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap)
+{
+	int cap_pos, next_pos, pre_pos;
+	u32 pre_header, header;
+
+	cap_pos = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	header = dw_pcie_readl_dbi(pci, cap_pos);
+	/*
+	 * If the first cap at offset PCI_CFG_SPACE_SIZE is removed,
+	 * only set it's capid to zero as it cannot be skipped.
+	 */
+	if (cap_pos == PCI_CFG_SPACE_SIZE) {
+		dw_pcie_dbi_ro_wr_en(pci);
+		dw_pcie_writel_dbi(pci, cap_pos, header & 0xffff0000);
+		dw_pcie_dbi_ro_wr_dis(pci);
+		return;
+	}
+
+	pre_header = dw_pcie_readl_dbi(pci, pre_pos);
+	next_pos = PCI_EXT_CAP_NEXT(header);
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	dw_pcie_writel_dbi(pci, pre_pos,
+			  (pre_header & 0xfffff) | (next_pos << 20));
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_ext_capability);
+
 static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 					  u16 vsec_id)
 {
drivers/pci/controller/dwc/pcie-designware.h +2 -0
index e995f692a1ecd10130d3be3358827f801811387f..b68dbc528001b63448db8b1a93bf56a5e53bd33e 100644
@@ -552,6 +552,8 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_rasdes_capability(struct dw_pcie *pci);
 u16 dw_pcie_find_ptm_capability(struct dw_pcie *pci);
 
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

Hello Qiang Yu,

I just saw that this patch was queued up.

dw_pcie_remove_ext_capability() basically seems to be identical to:
dw_pcie_ep_hide_ext_capability(), only that your new function does not require previous cap as an argument (so it seems better).

It seems a bit silly to have two functions that do the same thing, perhaps you can send a patch that removes dw_pcie_ep_hide_ext_capability() and updates the only user of that function to use your new function?

Sorry for not seeing this earlier.

Kind regards,
Niklas

Quoted 17 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
author
#4 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

Quoted 8 lines · Niklas Cassel

Hello Qiang Yu,

I just saw that this patch was queued up.

dw_pcie_remove_ext_capability() basically seems to be identical to:
dw_pcie_ep_hide_ext_capability(), only that your new function does not require previous cap as an argument (so it seems better).

Quoted 6 lines

It seems a bit silly to have two functions that do the same thing, perhaps you can send a patch that removes dw_pcie_ep_hide_ext_capability() and updates the only user of that function to use your new function?

Sorry for not seeing this earlier.

Hi Niklas

Thank you for the comments. I'd be happy to send a follow-up patch to resolve this.

- Qiang Yu

Quoted 22 lines

Kind regards,
Niklas

Quoted 17 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
#5 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

In reply to Qiang Yu
Quoted 9 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

It's super annoying to have to do this, but I suppose from the hardware point of view these things *could* work depending on the design of the platform outside the device. So the designers probably assume platform firmware knows those details and hides things that aren't supported. But in the DT case, there likely *is* no platform firmware, so the native RC driver has to do it instead.

Quoted 38 lines
---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
drivers/pci/controller/dwc/pcie-designware.c +22 -0
index 5585d3ed74316bd218572484f6320019db8d6a10..24f8e9959cb81ca41e91d27057cc115d32e8d523 100644
@@ -234,6 +234,59 @@ u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap)
+{
+	u8 cap_pos, pre_pos, next_pos;
+	u16 reg;
+
+	cap_pos = PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
+				 &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	reg = dw_pcie_readw_dbi(pci, cap_pos);
+	next_pos = (reg & 0xff00) >> 8;
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	if (pre_pos == PCI_CAPABILITY_LIST)
+		dw_pcie_writeb_dbi(pci, PCI_CAPABILITY_LIST, next_pos);
+	else
+		dw_pcie_writeb_dbi(pci, pre_pos + 1, next_pos);
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_capability);
+

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 268

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware.c +10 -0
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap)
+{
+	int cap_pos, next_pos, pre_pos;
+	u32 pre_header, header;
+
+	cap_pos = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	header = dw_pcie_readl_dbi(pci, cap_pos);

Blank line here to match style of other comments.

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 271

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware.c +3 -0
+	/*
+	 * If the first cap at offset PCI_CFG_SPACE_SIZE is removed,
+	 * only set it's capid to zero as it cannot be skipped.

If setting the capid to zero works here, why won't it work for all capabilities? If setting capid to zero is enough, we wouldn't need to mess with keeping track of the previous capability, and we could drop the first patch.

s/it's/its/

Quoted 38 lines
drivers/pci/controller/dwc/pcie-designware.c +18 -0
+	 */
+	if (cap_pos == PCI_CFG_SPACE_SIZE) {
+		dw_pcie_dbi_ro_wr_en(pci);
+		dw_pcie_writel_dbi(pci, cap_pos, header & 0xffff0000);
+		dw_pcie_dbi_ro_wr_dis(pci);
+		return;
+	}
+
+	pre_header = dw_pcie_readl_dbi(pci, pre_pos);
+	next_pos = PCI_EXT_CAP_NEXT(header);
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	dw_pcie_writel_dbi(pci, pre_pos,
+			  (pre_header & 0xfffff) | (next_pos << 20));
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_ext_capability);
+
 static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 					  u16 vsec_id)
 {
drivers/pci/controller/dwc/pcie-designware.h +2 -0
index e995f692a1ecd10130d3be3358827f801811387f..b68dbc528001b63448db8b1a93bf56a5e53bd33e 100644
@@ -552,6 +552,8 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_rasdes_capability(struct dw_pcie *pci);
 u16 dw_pcie_find_ptm_capability(struct dw_pcie *pci);
 
Signature · 2 lines
-- 
2.34.1
#6 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

Quoted 11 lines · Bjorn Helgaas
Quoted 9 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

Quoted 7 lines

It's super annoying to have to do this, but I suppose from the hardware point of view these things *could* work depending on the design of the platform outside the device. So the designers probably assume platform firmware knows those details and hides things that aren't supported. But in the DT case, there likely *is* no platform firmware, so the native RC driver has to do it instead.

This is actually a synopsis IP bug, so I'm not sure what the expectations are from vendor implementations pov. We do have platform firmware like UEFI, but it doesn't remove the capabilities.

- Mani

Quoted 100 lines
Quoted 38 lines
---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
drivers/pci/controller/dwc/pcie-designware.c +22 -0
index 5585d3ed74316bd218572484f6320019db8d6a10..24f8e9959cb81ca41e91d27057cc115d32e8d523 100644
@@ -234,6 +234,59 @@ u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap)
+{
+	u8 cap_pos, pre_pos, next_pos;
+	u16 reg;
+
+	cap_pos = PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
+				 &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	reg = dw_pcie_readw_dbi(pci, cap_pos);
+	next_pos = (reg & 0xff00) >> 8;
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	if (pre_pos == PCI_CAPABILITY_LIST)
+		dw_pcie_writeb_dbi(pci, PCI_CAPABILITY_LIST, next_pos);
+	else
+		dw_pcie_writeb_dbi(pci, pre_pos + 1, next_pos);
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_capability);
+

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 268

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware.c +10 -0
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap)
+{
+	int cap_pos, next_pos, pre_pos;
+	u32 pre_header, header;
+
+	cap_pos = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	header = dw_pcie_readl_dbi(pci, cap_pos);

Blank line here to match style of other comments.

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 271

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware.c +3 -0
+	/*
+	 * If the first cap at offset PCI_CFG_SPACE_SIZE is removed,
+	 * only set it's capid to zero as it cannot be skipped.

If setting the capid to zero works here, why won't it work for all capabilities? If setting capid to zero is enough, we wouldn't need to mess with keeping track of the previous capability, and we could drop the first patch.

s/it's/its/

Quoted 38 lines
drivers/pci/controller/dwc/pcie-designware.c +18 -0
+	 */
+	if (cap_pos == PCI_CFG_SPACE_SIZE) {
+		dw_pcie_dbi_ro_wr_en(pci);
+		dw_pcie_writel_dbi(pci, cap_pos, header & 0xffff0000);
+		dw_pcie_dbi_ro_wr_dis(pci);
+		return;
+	}
+
+	pre_header = dw_pcie_readl_dbi(pci, pre_pos);
+	next_pos = PCI_EXT_CAP_NEXT(header);
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	dw_pcie_writel_dbi(pci, pre_pos,
+			  (pre_header & 0xfffff) | (next_pos << 20));
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_ext_capability);
+
 static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 					  u16 vsec_id)
 {
drivers/pci/controller/dwc/pcie-designware.h +2 -0
index e995f692a1ecd10130d3be3358827f801811387f..b68dbc528001b63448db8b1a93bf56a5e53bd33e 100644
@@ -552,6 +552,8 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_rasdes_capability(struct dw_pcie *pci);
 u16 dw_pcie_find_ptm_capability(struct dw_pcie *pci);
 
Signature · 2 lines
-- 
2.34.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்
author
#7 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

In reply to Bjorn Helgaas
Quoted 69 lines · Bjorn Helgaas
Quoted 9 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

It's super annoying to have to do this, but I suppose from the hardware point of view these things *could* work depending on the design of the platform outside the device. So the designers probably assume platform firmware knows those details and hides things that aren't supported. But in the DT case, there likely *is* no platform firmware, so the native RC driver has to do it instead.

Quoted 38 lines
---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
drivers/pci/controller/dwc/pcie-designware.c +22 -0
index 5585d3ed74316bd218572484f6320019db8d6a10..24f8e9959cb81ca41e91d27057cc115d32e8d523 100644
@@ -234,6 +234,59 @@ u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap)
+{
+	u8 cap_pos, pre_pos, next_pos;
+	u16 reg;
+
+	cap_pos = PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
+				 &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	reg = dw_pcie_readw_dbi(pci, cap_pos);
+	next_pos = (reg & 0xff00) >> 8;
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	if (pre_pos == PCI_CAPABILITY_LIST)
+		dw_pcie_writeb_dbi(pci, PCI_CAPABILITY_LIST, next_pos);
+	else
+		dw_pcie_writeb_dbi(pci, pre_pos + 1, next_pos);
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_capability);
+

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 268

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware.c +10 -0
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap)
+{
+	int cap_pos, next_pos, pre_pos;
+	u32 pre_header, header;
+
+	cap_pos = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	header = dw_pcie_readl_dbi(pci, cap_pos);

Blank line here to match style of other comments.

Quoted 9 lines

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 271

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware.c +3 -0
+	/*
+	 * If the first cap at offset PCI_CFG_SPACE_SIZE is removed,
+	 * only set it's capid to zero as it cannot be skipped.

If setting the capid to zero works here, why won't it work for all capabilities? If setting capid to zero is enough, we wouldn't need to mess with keeping track of the previous capability, and we could drop the first patch.

Setting the capability ID to zero is simpler and works for all capabilities. However, it makes users confused to see a capability with capability ID zero, especially when using debug tools like lspci to list the configuration space. I think it works functionally, but it's not a clean approach. Setting capid of the cap at offset PCI_CFG_SPACE_SIZE to zero is a last resort.

-Qiang Yu

Quoted 40 lines

s/it's/its/

Quoted 38 lines
drivers/pci/controller/dwc/pcie-designware.c +18 -0
+	 */
+	if (cap_pos == PCI_CFG_SPACE_SIZE) {
+		dw_pcie_dbi_ro_wr_en(pci);
+		dw_pcie_writel_dbi(pci, cap_pos, header & 0xffff0000);
+		dw_pcie_dbi_ro_wr_dis(pci);
+		return;
+	}
+
+	pre_header = dw_pcie_readl_dbi(pci, pre_pos);
+	next_pos = PCI_EXT_CAP_NEXT(header);
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	dw_pcie_writel_dbi(pci, pre_pos,
+			  (pre_header & 0xfffff) | (next_pos << 20));
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_ext_capability);
+
 static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 					  u16 vsec_id)
 {
drivers/pci/controller/dwc/pcie-designware.h +2 -0
index e995f692a1ecd10130d3be3358827f801811387f..b68dbc528001b63448db8b1a93bf56a5e53bd33e 100644
@@ -552,6 +552,8 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_rasdes_capability(struct dw_pcie *pci);
 u16 dw_pcie_find_ptm_capability(struct dw_pcie *pci);
 
Signature · 2 lines
-- 
2.34.1
author
#8 lore raw

Re: [PATCH 2/5] PCI: dwc: Add new APIs to remove standard and extended Capability

In reply to Bjorn Helgaas
Quoted 69 lines · Bjorn Helgaas
Quoted 9 lines · Qiang Yu

On some platforms, certain PCIe Capabilities may be present in hardware but are not fully implemented as defined in PCIe spec. These incomplete capabilities should be hidden from the PCI framework to prevent unexpected behavior.

Introduce two APIs to remove a specific PCIe Capability and Extended Capability by updating the previous capability's next offset field to skip over the unwanted capability. These APIs allow RC drivers to easily hide unsupported or partially implemented capabilities from software.

It's super annoying to have to do this, but I suppose from the hardware point of view these things *could* work depending on the design of the platform outside the device. So the designers probably assume platform firmware knows those details and hides things that aren't supported. But in the DT case, there likely *is* no platform firmware, so the native RC driver has to do it instead.

Quoted 38 lines
---
 drivers/pci/controller/dwc/pcie-designware.c | 53 ++++++++++++++++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h |  2 ++
 2 files changed, 55 insertions(+)
drivers/pci/controller/dwc/pcie-designware.c +22 -0
index 5585d3ed74316bd218572484f6320019db8d6a10..24f8e9959cb81ca41e91d27057cc115d32e8d523 100644
@@ -234,6 +234,59 @@ u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_ext_capability);
 
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap)
+{
+	u8 cap_pos, pre_pos, next_pos;
+	u16 reg;
+
+	cap_pos = PCI_FIND_NEXT_CAP(dw_pcie_read_cfg, PCI_CAPABILITY_LIST, cap,
+				 &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	reg = dw_pcie_readw_dbi(pci, cap_pos);
+	next_pos = (reg & 0xff00) >> 8;
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	if (pre_pos == PCI_CAPABILITY_LIST)
+		dw_pcie_writeb_dbi(pci, PCI_CAPABILITY_LIST, next_pos);
+	else
+		dw_pcie_writeb_dbi(pci, pre_pos + 1, next_pos);
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_capability);
+

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 268

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware.c +10 -0
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap)
+{
+	int cap_pos, next_pos, pre_pos;
+	u32 pre_header, header;
+
+	cap_pos = PCI_FIND_NEXT_EXT_CAP(dw_pcie_read_cfg, 0, cap, &pre_pos, pci);
+	if (!cap_pos)
+		return;
+
+	header = dw_pcie_readl_dbi(pci, cap_pos);

Blank line here to match style of other comments.

Quoted 8 lines

drivers/pci/controller/dwc/pcie-designware.c u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap) · line 271

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware.c +3 -0
+	/*
+	 * If the first cap at offset PCI_CFG_SPACE_SIZE is removed,
+	 * only set it's capid to zero as it cannot be skipped.

If setting the capid to zero works here, why won't it work for all capabilities? If setting capid to zero is enough, we wouldn't need to mess with keeping track of the previous capability, and we could drop the first patch.

I tried this. Setting capid to zero only works for extended capabilities, not standard ones. Standard cap id is RO and can not be cleared.

- Qiang Yu

Quoted 41 lines

s/it's/its/

Quoted 38 lines
drivers/pci/controller/dwc/pcie-designware.c +18 -0
+	 */
+	if (cap_pos == PCI_CFG_SPACE_SIZE) {
+		dw_pcie_dbi_ro_wr_en(pci);
+		dw_pcie_writel_dbi(pci, cap_pos, header & 0xffff0000);
+		dw_pcie_dbi_ro_wr_dis(pci);
+		return;
+	}
+
+	pre_header = dw_pcie_readl_dbi(pci, pre_pos);
+	next_pos = PCI_EXT_CAP_NEXT(header);
+
+	dw_pcie_dbi_ro_wr_en(pci);
+	dw_pcie_writel_dbi(pci, pre_pos,
+			  (pre_header & 0xfffff) | (next_pos << 20));
+	dw_pcie_dbi_ro_wr_dis(pci);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_remove_ext_capability);
+
 static u16 __dw_pcie_find_vsec_capability(struct dw_pcie *pci, u16 vendor_id,
 					  u16 vsec_id)
 {
drivers/pci/controller/dwc/pcie-designware.h +2 -0
index e995f692a1ecd10130d3be3358827f801811387f..b68dbc528001b63448db8b1a93bf56a5e53bd33e 100644
@@ -552,6 +552,8 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_capability(struct dw_pcie *pci, u8 cap);
+void dw_pcie_remove_ext_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_rasdes_capability(struct dw_pcie *pci);
 u16 dw_pcie_find_ptm_capability(struct dw_pcie *pci);
 
Signature · 2 lines
-- 
2.34.1
author
#9 lore raw

PATCH 3/5 PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +10 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
Signature · 1 line
2.34.1
#10 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

+ Folks who were part of the previous discussions on this topic.

Quoted 56 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +10 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
Signature · 2 lines
-- 
2.34.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்
#11 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Qiang Yu
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 22 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
  drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1091

Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
  	dw_pcie_dbi_ro_wr_dis(pci);
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1093

Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Quoted 6 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+	}
+
  	return 0;
  }
  EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
#12 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 39 lines · Shawn Lin
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 17 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
  drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
Quoted 16 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
  	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

Quoted 5 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1093

Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Nah, it is just for the Root Port. The MSI/MSI-X from endpoint devices will continue to work as usual.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#13 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 57 lines · Manivannan Sadhasivam
Quoted 39 lines · Shawn Lin
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 17 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
   drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
   1 file changed, 10 insertions(+)
Quoted 16 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
   	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 2 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

Thanks for correcting me. Yeap, how can I make such a mistake. :(

Anyway, this patch works for me:

root@debian:/userdata# ./aer-inject aer.txt
[   17.764272] pcieport 0000:00:00.0: aer_inject: Injecting errors 00000040/00000000 into device 0000:01:00.0
[   17.765178] aer_isr ! #log I added in aer_isr
[   17.765394] pcieport 0000:00:00.0: AER: Correctable error message received from 0000:01:00.0
[   17.766211] nvme 0000:01:00.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID)
root@debian:/userdata# [   17.767045] nvme 0000:01:00.0:   device [144d:a80a] error status/mask=00000040/0000e000
[   17.767980] nvme 0000:01:00.0:    [ 6] BadTLP
root@debian:/userdata# cat /proc/interrupts | grep aerdrv
 60:      0      0      0      0      0      0     0     0     INTx   0 
Edge      PCIe PME, aerdrv, PCIe bwctrl
 63:      0      0      0      1      0      0     0     0     INTx   0 
Edge      PCIe PME, aerdrv
110:      0      0      0      0      0      0     0     0     INTx   0 Edge      PCIe PME, aerdrv
Quoted 9 lines
Quoted 6 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1091

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Nah, it is just for the Root Port. The MSI/MSI-X from endpoint devices will continue to work as usual.

Qiang Yu,

Could you please help your IP version with below patch? It's in hex format, you could convert each pair of hex characters to ASCII, i.g, 0x3437302a is 4.70a. The reason is we asked Synopsys to help check this issue before, then we were informed that they have supported it at least since IP version 6.0x. So we may have to limit the version first.

drivers/pci/controller/dwc/pcie-designware-host.c +4 -0
@@ -1057,6 +1057,10 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
        dw_pcie_msi_init(pp);
 
+#define PORT_LOGIC_PCIE_VERSION_NUMBER_OFF 0x8f8
+       val = dw_pcie_readl_dbi(pci, PORT_LOGIC_PCIE_VERSION_NUMBER_OFF);
+       printk("version = 0x%x\n", val);
+
 
 
 
author
#14 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 95 lines · Shawn Lin
Quoted 57 lines · Manivannan Sadhasivam
Quoted 39 lines · Shawn Lin
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 17 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
   drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
   1 file changed, 10 insertions(+)
Quoted 16 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
   	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 2 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

Thanks for correcting me. Yeap, how can I make such a mistake. :(

Anyway, this patch works for me:

root@debian:/userdata# ./aer-inject aer.txt
[   17.764272] pcieport 0000:00:00.0: aer_inject: Injecting errors
00000040/00000000 into device 0000:01:00.0
[   17.765178] aer_isr ! #log I added in aer_isr
[   17.765394] pcieport 0000:00:00.0: AER: Correctable error message
received from 0000:01:00.0
[   17.766211] nvme 0000:01:00.0: PCIe Bus Error: severity=Correctable,
type=Data Link Layer, (Receiver ID)
root@debian:/userdata# [   17.767045] nvme 0000:01:00.0:   device
[144d:a80a] error status/mask=00000040/0000e000
[   17.767980] nvme 0000:01:00.0:    [ 6] BadTLP
root@debian:/userdata# cat /proc/interrupts | grep aerdrv
 60:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv, PCIe bwctrl
 63:      0      0      0      1      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
110:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
Quoted 9 lines
Quoted 6 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1091

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Nah, it is just for the Root Port. The MSI/MSI-X from endpoint devices will continue to work as usual.

Quoted 9 lines

Qiang Yu,

Could you please help your IP version with below patch? It's in hex format, you could convert each pair of hex characters to ASCII, i.g, 0x3437302a is 4.70a. The reason is we asked Synopsys to help check this issue before, then we were informed that they have supported it at least since IP version 6.0x. So we may have to limit the version first.

Okay, let me check with internal team, will update.

- Qiang Yu

Quoted 13 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4 -0
@@ -1057,6 +1057,10 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
        dw_pcie_msi_init(pp);
 
+#define PORT_LOGIC_PCIE_VERSION_NUMBER_OFF 0x8f8
+       val = dw_pcie_readl_dbi(pci, PORT_LOGIC_PCIE_VERSION_NUMBER_OFF);
+       printk("version = 0x%x\n", val);
+
 
 
 
author
#15 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Shawn Lin
Quoted 95 lines · Shawn Lin
Quoted 57 lines · Manivannan Sadhasivam
Quoted 39 lines · Shawn Lin
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 17 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
   drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
   1 file changed, 10 insertions(+)
Quoted 16 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
   	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 2 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

Thanks for correcting me. Yeap, how can I make such a mistake. :(

Anyway, this patch works for me:

root@debian:/userdata# ./aer-inject aer.txt
[   17.764272] pcieport 0000:00:00.0: aer_inject: Injecting errors
00000040/00000000 into device 0000:01:00.0
[   17.765178] aer_isr ! #log I added in aer_isr
[   17.765394] pcieport 0000:00:00.0: AER: Correctable error message
received from 0000:01:00.0
[   17.766211] nvme 0000:01:00.0: PCIe Bus Error: severity=Correctable,
type=Data Link Layer, (Receiver ID)
root@debian:/userdata# [   17.767045] nvme 0000:01:00.0:   device
[144d:a80a] error status/mask=00000040/0000e000
[   17.767980] nvme 0000:01:00.0:    [ 6] BadTLP
root@debian:/userdata# cat /proc/interrupts | grep aerdrv
 60:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv, PCIe bwctrl
 63:      0      0      0      1      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
110:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
Quoted 9 lines
Quoted 6 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1091

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Nah, it is just for the Root Port. The MSI/MSI-X from endpoint devices will continue to work as usual.

Quoted 9 lines

Qiang Yu,

Could you please help your IP version with below patch? It's in hex format, you could convert each pair of hex characters to ASCII, i.g, 0x3437302a is 4.70a. The reason is we asked Synopsys to help check this issue before, then we were informed that they have supported it at least since IP version 6.0x. So we may have to limit the version first.

Hi Shawn,

I checked the IP version of PCIe core on glymur, it is 6.00a (0x3630302A) and iMSI-RX still can't generate MSI for rootport.

- Qiang Yu

Quoted 13 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4 -0
@@ -1057,6 +1057,10 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
        dw_pcie_msi_init(pp);
 
+#define PORT_LOGIC_PCIE_VERSION_NUMBER_OFF 0x8f8
+       val = dw_pcie_readl_dbi(pci, PORT_LOGIC_PCIE_VERSION_NUMBER_OFF);
+       printk("version = 0x%x\n", val);
+
 
 
 
#16 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 106 lines · Qiang Yu
Quoted 95 lines · Shawn Lin
Quoted 57 lines · Manivannan Sadhasivam
Quoted 39 lines · Shawn Lin
Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 1 line

This leads to interrupts such as PME, AER from the Root Port not received

This's true which also stops Rockchip's dwc IP from working with AER service. But my platform can't work with AER service even with ITS support.

Quoted 2 lines

on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

ack.

Quoted 17 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
    drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
    1 file changed, 10 insertions(+)
Quoted 16 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 12 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
    	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 2 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

Thanks for correcting me. Yeap, how can I make such a mistake. :(

Anyway, this patch works for me:

root@debian:/userdata# ./aer-inject aer.txt
[   17.764272] pcieport 0000:00:00.0: aer_inject: Injecting errors
00000040/00000000 into device 0000:01:00.0
[   17.765178] aer_isr ! #log I added in aer_isr
[   17.765394] pcieport 0000:00:00.0: AER: Correctable error message
received from 0000:01:00.0
[   17.766211] nvme 0000:01:00.0: PCIe Bus Error: severity=Correctable,
type=Data Link Layer, (Receiver ID)
root@debian:/userdata# [   17.767045] nvme 0000:01:00.0:   device
[144d:a80a] error status/mask=00000040/0000e000
[   17.767980] nvme 0000:01:00.0:    [ 6] BadTLP
root@debian:/userdata# cat /proc/interrupts | grep aerdrv
  60:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv, PCIe bwctrl
  63:      0      0      0      1      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
110:      0      0      0      0      0      0     0     0     INTx   0 Edge
PCIe PME, aerdrv
Quoted 9 lines
Quoted 6 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1091

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Will it make all devices connected to use INTx only?

Nah, it is just for the Root Port. The MSI/MSI-X from endpoint devices will continue to work as usual.

Quoted 9 lines

Qiang Yu,

Could you please help your IP version with below patch? It's in hex format, you could convert each pair of hex characters to ASCII, i.g, 0x3437302a is 4.70a. The reason is we asked Synopsys to help check this issue before, then we were informed that they have supported it at least since IP version 6.0x. So we may have to limit the version first.

Quoted 5 lines

Hi Shawn,

I checked the IP version of PCIe core on glymur, it is 6.00a (0x3630302A) and iMSI-RX still can't generate MSI for rootport.

Thanks for let me know. I have no doubt about this patch, it works for me as well.

Quoted 16 lines

- Qiang Yu

Quoted 13 lines
drivers/pci/controller/dwc/pcie-designware-host.c +4 -0
@@ -1057,6 +1057,10 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
         dw_pcie_msi_init(pp);
 
+#define PORT_LOGIC_PCIE_VERSION_NUMBER_OFF 0x8f8
+       val = dw_pcie_readl_dbi(pci, PORT_LOGIC_PCIE_VERSION_NUMBER_OFF);
+       printk("version = 0x%x\n", val);
+
 
 
 
Tested-by
#17 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Qiang Yu
Quoted 10 lines · Qiang Yu
Quoted 8 lines · Shawn Lin

在 2025/11/21 星期五 1:00, Manivannan Sadhasivam 写道:
Could you please help your IP version with below patch? It's in hex format, you could convert each pair of hex characters to ASCII, i.g, 0x3437302a is 4.70a. The reason is we asked Synopsys to help check this issue before, then we were informed that they have supported it at least since IP version 6.0x. So we may have to limit the version first.

Quoted 4 lines

Hi Shawn,

I checked the IP version of PCIe core on glymur, it is 6.00a (0x3630302A) and iMSI-RX still can't generate MSI for rootport.

Same here, I have chips with 0x3630302A / 6.00a, and MSI does not work for the root port. This series tests out fine for me, so:

#18 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Manivannan Sadhasivam
Quoted 26 lines · Manivannan Sadhasivam
Quoted 12 lines · Shawn Lin
Quoted 10 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 12 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 9 lines

+++ b/drivers/pci/controller/dwc/pcie-designware-host.c

drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
  	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 3 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

I didn't see any response to this. Is there a case where we want to preserve these capabilities? To ask another way, is there some platform where DWC-based Root Ports can generate MSIs themselves?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
#19 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 27 lines · Bjorn Helgaas
Quoted 26 lines · Manivannan Sadhasivam
Quoted 24 lines · Shawn Lin
Quoted 13 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

+++ b/drivers/pci/controller/dwc/pcie-designware-host.c

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1089

Quoted 7 lines
drivers/pci/controller/dwc/pcie-designware-host.c +6 -0
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
  	dw_pcie_dbi_ro_wr_dis(pci);
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {

Isn't has_msi_ctrl means you have something like GIC-ITS support instead of iMSI module? Am I missing anything?

Quoted 5 lines
Quoted 3 lines

It is the other way around. Presence of this flag means, iMSI-RX is used. But I think the driver should clear the CAPs irrespective of this flag.

I didn't see any response to this.

Shawn did respond to my comment:
https://lore.kernel.org/linux-pci/[email protected]/

Quoted 4 lines

Is there a case where we want to
preserve these capabilities? To ask another way, is there some platform where DWC-based Root Ports can generate MSIs themselves?

Unless the DWC controller implements a custom internal MSI controller (if it does then dw_pcie_host_ops::msi_init() callback will be populated), Root Port MSIs cannot be received. This series did exclude only one of those platforms, pci-keystone by skipping capability removal if 'dw_pcie_rp::has_msi_ctrl' is set.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
Reviewed-by on v1 3/5
#20 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Qiang Yu

Hi,

Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 7 lines

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts,

But "has_msi_ctrl == false" does not necessarily mean it's using an external MSI controller, does it? It could just mean that there's some per-SoC customization needed via the .msi_init() hook.

In practice, that's only pci-keystone.c though, and it's not really clear if that's some modified version of iMSI-RX, or something else entirely. At any rate, I suppose it's best to only tweak the things we know about -- unmodified DWC iMSI-RX support.

Quoted 18 lines

eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1090

Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c +5 -0
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */

Personally, I'd suggest including more of the "why?" in this comment, as the "why?" is pretty perplexing to an uninitiated reader.

Maybe:

/*
* The iMSI-RX module does not support MSI or MSI-X generated by
* the root port. If iMSI-RX is used as the MSI controller,
* remove the MSI and MSI-X capabilities to fall back to INTx
* instead.
*/

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1093

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -0
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Removing the capability structure is a neat idea. I had prototyped solving this problem by adding a new PCI_MSI_FLAGS_* quirk, but that was a lot more invasive. I like this idea instead!

This looks good to me, although maybe the comment could be updated. Feel free to carry my:

I'd also note, not all devices currently actually define their INTx interrupts (such as ... my current test devices :( ), and so pcie_init_service_irqs() / portdrv.c may fail entirely, since it can't really provide any services if there are no IRQs for those services. That does have at least one bad side effect: that the port won't be configured for runtime PM and won't ever enter D3.

I wonder if we should allow pcie_port_device_register() to succeed even if it ends up with an empty 'capabilities' / no services.

Brian

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +2 -0
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
Signature · 2 lines
-- 
2.34.1
#21 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 18 lines · Brian Norris

Hi,

Quoted 15 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

Quoted 8 lines
Quoted 3 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts,

But "has_msi_ctrl == false" does not necessarily mean it's using an external MSI controller, does it? It could just mean that there's some per-SoC customization needed via the .msi_init() hook.

But we really do not know if that is a per-SoC customization or entirely different MSI controller. So it is safe to ignore them and that's what this patch also does.

Quoted 35 lines

In practice, that's only pci-keystone.c though, and it's not really clear if that's some modified version of iMSI-RX, or something else entirely. At any rate, I suppose it's best to only tweak the things we know about -- unmodified DWC iMSI-RX support.

Quoted 18 lines

eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1090

Quoted 8 lines
drivers/pci/controller/dwc/pcie-designware-host.c +5 -0
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */

Personally, I'd suggest including more of the "why?" in this comment, as the "why?" is pretty perplexing to an uninitiated reader.

Quoted 9 lines

Maybe:

/*
* The iMSI-RX module does not support MSI or MSI-X generated by
* the root port. If iMSI-RX is used as the MSI controller,
* remove the MSI and MSI-X capabilities to fall back to INTx
* instead.
*/

I've ammended the commit with the above comment.

Quoted 20 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1093

Quoted 3 lines
drivers/pci/controller/dwc/pcie-designware-host.c +3 -0
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);

Removing the capability structure is a neat idea. I had prototyped solving this problem by adding a new PCI_MSI_FLAGS_* quirk, but that was a lot more invasive. I like this idea instead!

This looks good to me, although maybe the comment could be updated. Feel free to carry my:

I'd also note, not all devices currently actually define their INTx interrupts (such as ... my current test devices :( ), and so pcie_init_service_irqs() / portdrv.c may fail entirely, since it can't really provide any services if there are no IRQs for those services. That does have at least one bad side effect: that the port won't be configured for runtime PM and won't ever enter D3.

Quoted 3 lines

I wonder if we should allow pcie_port_device_register() to succeed even if it ends up with an empty 'capabilities' / no services.

Sounds logical to me as pcie_port_device_register() already returns 0 if there are no PCI Express port capability.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#22 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

In reply to Qiang Yu

In subject, s/MSIX/MSI-X/ to match spec and other usage.

Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 3 lines

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

This will be great, thanks a lot for working on this. This has been a long-standing irritation with this DWC IP.

Quoted 23 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1094

Quoted 10 lines
drivers/pci/controller/dwc/pcie-designware-host.c +9 -0
 
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
+	}

"has_msi_ctrl" doesn't seem like a good name here because there's no documentation about what it means, and "has_msi_ctrl" is completely generic while "iMSI-RX" is very specific.

And apparently platforms with ITS/MBI *can* generate MSIs from Root Ports, but "has_msi_ctrl" would be false for them? This is really hard to read.

pp->has_msi_ctrl is set by qcom_pcie_ecam_host_init() and IIUC, for any platform that lacks .msi_init() and the "msi-parent" and "msi-map" properties.

The qcom_pcie_ecam_host_init() case is weird because it looks like it abuses the pci_ecam_ops.init() callback to initialize MSI stuff, not ECAM stuff. Maybe that MSI init could be done in qcom_pcie_probe() right after it calls pci_host_common_ecam_create()?

Bjorn

#23 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 45 lines · Bjorn Helgaas

In subject, s/MSIX/MSI-X/ to match spec and other usage.

Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 3 lines

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

This will be great, thanks a lot for working on this. This has been a long-standing irritation with this DWC IP.

Quoted 24 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
Quoted 14 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1094

Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +9 -0
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
+	}

"has_msi_ctrl" doesn't seem like a good name here because there's no documentation about what it means, and "has_msi_ctrl" is completely generic while "iMSI-RX" is very specific.

This predates my involvement with DWC drivers, but I guess it expands to 'has internal MSI controller' and 'internal' probably means iMSI-RX. But I agree that the naming could be improved to something like 'imsi_rx_available' or 'has_imsi_rx'. I'll take a stab at it in a separate patch.

Quoted 4 lines

And apparently platforms with ITS/MBI *can* generate MSIs from Root Ports, but "has_msi_ctrl" would be false for them? This is really hard to read.

Yes.

Quoted 9 lines

pp->has_msi_ctrl is set by qcom_pcie_ecam_host_init() and IIUC, for any platform that lacks .msi_init() and the "msi-parent" and "msi-map" properties.

The qcom_pcie_ecam_host_init() case is weird because it looks like it abuses the pci_ecam_ops.init() callback to initialize MSI stuff, not ECAM stuff. Maybe that MSI init could be done in qcom_pcie_probe() right after it calls pci_host_common_ecam_create()?

I think it should be possible to initialize MSI after pci_host_common_ecam_create(). Let me fix *this* and above in a separate series.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
author
#24 lore raw

Re: [PATCH 3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller

Quoted 77 lines · Manivannan Sadhasivam
Quoted 45 lines · Bjorn Helgaas

In subject, s/MSIX/MSI-X/ to match spec and other usage.

Quoted 11 lines · Qiang Yu

Some platforms may not support ITS (Interrupt Translation Service) and MBI (Message Based Interrupt), or there are not enough available empty SPI lines for MBI, in which case the msi-map and msi-parent property will not be provided in device tree node. For those cases, the DWC PCIe driver defaults to using the iMSI-RX module as MSI controller. However, due to DWC IP design, iMSI-RX cannot generate MSI interrupts for Root Ports even when MSI is properly configured and supported as iMSI-RX will only monitor and intercept incoming MSI TLPs from PCIe link, but the memory write generated by Root Port are internal system bus transactions instead of PCIe TLPs, so they are ignored.

Quoted 3 lines

This leads to interrupts such as PME, AER from the Root Port not received on the host and the users have to resort to workarounds such as passing "pcie_pme=nomsi" cmdline parameter.

This will be great, thanks a lot for working on this. This has been a long-standing irritation with this DWC IP.

Quoted 24 lines

To ensure reliable interrupt handling, remove MSI and MSI-X capabilities from Root Ports when using iMSI-RX as MSI controller, which is indicated by has_msi_ctrl == true. This forces a fallback to INTx interrupts, eliminating the need for manual kernel command line workarounds.

With this behavior:
- Platforms with ITS/MBI support use ITS/MBI MSI for interrupts from all components.
- Platforms without ITS/MBI support fall back to INTx for Root Ports and use iMSI-RX for other PCI devices.

---
 drivers/pci/controller/dwc/pcie-designware-host.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/pci/controller/dwc/pcie-designware-host.c +0 -0
index 20c9333bcb1c4812e2fd96047a49944574df1e6f..3724aa7f9b356bfba33a6515e2c62a3170aef1e9 100644
@@ -1083,6 +1083,16 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
 
 	dw_pcie_dbi_ro_wr_dis(pci);
 
Quoted 14 lines

drivers/pci/controller/dwc/pcie-designware-host.c int dw_pcie_setup_rc(struct dw_pcie_rp *pp) · line 1094

Quoted 9 lines
drivers/pci/controller/dwc/pcie-designware-host.c +9 -0
+	/*
+	 * If iMSI-RX module is used as the MSI controller, remove MSI and
+	 * MSI-X capabilities from PCIe Root Ports to ensure fallback to INTx
+	 * interrupt handling.
+	 */
+	if (pp->has_msi_ctrl) {
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSI);
+		dw_pcie_remove_capability(pci, PCI_CAP_ID_MSIX);
+	}

"has_msi_ctrl" doesn't seem like a good name here because there's no documentation about what it means, and "has_msi_ctrl" is completely generic while "iMSI-RX" is very specific.

This predates my involvement with DWC drivers, but I guess it expands to 'has internal MSI controller' and 'internal' probably means iMSI-RX. But I agree that the naming could be improved to something like 'imsi_rx_available' or 'has_imsi_rx'. I'll take a stab at it in a separate patch.

Quoted 4 lines

And apparently platforms with ITS/MBI *can* generate MSIs from Root Ports, but "has_msi_ctrl" would be false for them? This is really hard to read.

Yes.

Quoted 4 lines

pp->has_msi_ctrl is set by qcom_pcie_ecam_host_init() and IIUC, for any platform that lacks .msi_init() and the "msi-parent" and "msi-map" properties.

Quoted 8 lines
Quoted 5 lines

The qcom_pcie_ecam_host_init() case is weird because it looks like it abuses the pci_ecam_ops.init() callback to initialize MSI stuff, not ECAM stuff. Maybe that MSI init could be done in qcom_pcie_probe() right after it calls pci_host_common_ecam_create()?

I think it should be possible to initialize MSI after pci_host_common_ecam_create(). Let me fix *this* and above in a separate series.

The qcom_pcie_ecam_host_init() is used by firmware-managed targets and the function unconditionally sets has_msi_ctrl = true without checking for "msi-parent" or "msi-map" properties in the device tree. So I think firmware should take care of removing MSI/MSIX cap.

- Qiang Yu

Quoted 5 lines

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
author
#25 lore raw

PATCH 4/5 PCI: qcom: Remove MSI-X Capability for Root Ports

In reply to Qiang Yu

On some platforms like Glymur, the hardware does not support MSI-X in RC mode, yet still exposes the MSI-X capability. However, it omits the required MSI-X Table and PBA structures.

This mismatch can lead to issues where the PCIe port driver requests MSI-X instead of MSI, causing the Root Port to trigger interrupts by writing to an uninitialized address, resulting in SMMU faults.

To address this, remove MSI-X capability unconditionally for Root Ports.

---
 drivers/pci/controller/dwc/pcie-qcom.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/pci/controller/dwc/pcie-qcom.c +2 -0
index 805edbbfe7eba496bc99ca82051dee43d240f359..09443ffbb150e9c91bfd3b2adf15286ef2f00a2a 100644
@@ -1350,6 +1350,8 @@ static int qcom_pcie_host_init(struct dw_pcie_rp *pp)
 			goto err_disable_phy;
 	}
 
+	dw_pcie_remove_capability(pcie->pci, PCI_CAP_ID_MSIX);
+
 	qcom_ep_reset_deassert(pcie);
 
 	if (pcie->cfg->ops->config_sid) {
Signature · 1 line
2.34.1
author
#26 lore raw

PATCH 5/5 PCI: qcom: Remove DPC Extended Capability

In reply to Qiang Yu

Some platforms (e.g., X1E80100) expose Downstream Port Containment (DPC) Extended Capability registers in the PCIe Root Port config space, but do not fully support it. To prevent undefined behavior and ensure DPC cap is not visible to PCI framework and users, remove DPC Extended Capability unconditionally, since there is no qcom platform support DPC till now.

---
 drivers/pci/controller/dwc/pcie-qcom.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/controller/dwc/pcie-qcom.c +1 -0
index 09443ffbb150e9c91bfd3b2adf15286ef2f00a2a..1b0f72bc38d912ab46739aa7f904ceca617c668d 100644
@@ -1351,6 +1351,7 @@ static int qcom_pcie_host_init(struct dw_pcie_rp *pp)
 	}
 
 	dw_pcie_remove_capability(pcie->pci, PCI_CAP_ID_MSIX);
+	dw_pcie_remove_ext_capability(pcie->pci, PCI_EXT_CAP_ID_DPC);
 
 	qcom_ep_reset_deassert(pcie);
 
Signature · 1 line
2.34.1
#27 lore raw
In reply to Qiang Yu
Quoted 8 lines · Qiang Yu

This patch series addresses issues where certain PCIe Standard or Extended Capabilities are advertised by the controller, but not fully or at all implemented in hardware. Exposing such capabilities to the PCI framework can lead to unexpected or undefined behavior.

The series consists of two main parts:

[...]

Applied, thanks!

[1/5] PCI: Add preceding capability position support and update drivers

      commit: a2582e05e39adf9ab82a02561cd6f70738540ae0
[2/5] PCI: dwc: Add new APIs to remove standard and extended Capability
      commit: 0183562f1e824c0ca6c918309a0978e9a269af3e
[3/5] PCI: dwc: Remove MSI/MSIX capability if iMSI-RX is used as MSI controller
      commit: f5cd8a929c825ad4df3972df041ad62ad84ca6c9
[4/5] PCI: qcom: Remove MSI-X Capability for Root Ports
      commit: 7c29cd0fdc07e5e21202fdeed0b63cba2b4f10c6
[5/5] PCI: qcom: Remove DPC Extended Capability
      commit: 6a1394990902f0393706d7f96f58c21d88b65df7

Best regards,

Signature · 1 line
Manivannan Sadhasivam <[email protected]>