Skip to contentSkip to conversation

PCI/CXL: Add RDPAS support for CXL.io

RFC PATCH Superseded Dave Jiang 3 participants 10 messages May 26 – Jun 17

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Dave Jiang

author maintainer
#0 lore raw

cover PCI/CXL: Add RDPAS support for CXL.io

The series add RCEC Downstream Port Assocation Structure (RDPAS) parsing support to CXL.io. RDPAS is an ACPI table that is part of the CXL Early Discovery Table (CEDT) defined in CXL specification r4.0 9.18.1.5. It provides the mapping between RCEC and downstream ports. With RDPAS, the error device can be directly found when an error is reported on RCEC, without walking a number of RCiEP in order to determine which one reported the error. While CXL.cachemem is supported by RDPAS, there is no easy way to discover the source id of the error and therefore finding the Linux PCI object for the RCiEP. The intention here is to accelerate the discovery of the error by directly locating the error device with the given information.

This series is based on top of Terry's CXL error protocol series [1]. Looking for comments on the series WRT if it makes sense to add on top of Terry's error handling for RCH/RCD devices.

[1]: https://lore.kernel.org/linux-cxl/[email protected]/T/#t

Dave Jiang (2):
PCI/CXL: Add RDPAS parsing support
PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

 drivers/cxl/acpi.c             |   5 +
 drivers/pci/pcie/aer_cxl_rch.c | 190 ++++++++++++++++++++++++++++++++-
 2 files changed, 192 insertions(+), 3 deletions(-)

base-commit: 721d08f9e0f07f8f1b93e7767762b565f088cdcd

Signature · 1 line
2.54.0
author maintainer
#1 lore raw

RFC PATCH 1/2 PCI/CXL: Add RDPAS parsing support

Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.

A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.

The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.

A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.

---
 drivers/cxl/acpi.c             |  5 ++
 drivers/pci/pcie/aer_cxl_rch.c | 89 ++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)
drivers/cxl/acpi.c +5 -0
index 127537628817..e33cae64def9 100644
@@ -7,6 +7,7 @@
 #include <linux/acpi.h>
 #include <linux/pci.h>
 #include <linux/node.h>
+#include <cxl/ras.h>
 #include <asm/div64.h>
 #include "cxlpci.h"
 #include "cxl.h"
@@ -933,6 +934,10 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 	if (rc < 0)
 		return -ENXIO;
 
+	rc = acpi_table_parse_cedt(ACPI_CEDT_TYPE_RDPAS, cxl_parse_rdpas, host);
+	if (rc < 0)
+		dev_dbg(host, "No RDPAS entries found or failed to parse\n");
+
 	rc = add_cxl_resources(cxl_res);
 	if (rc)
 		return rc;
drivers/pci/pcie/aer_cxl_rch.c +89 -0

98 lines not on this page · raw message

Signature · 1 line
2.54.0
#2 lore raw

Re: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] RDPAS entries for an RCEC with multiple downstream ports are incorrectly dropped as duplicates.…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] RDPAS entries for an RCEC with multiple downstream ports are incorrectly dropped as duplicates.
- [High] Missing length and boundary validation for ACPI CEDT RDPAS subtable can lead to out-of-bounds memory access.
- [Low] The patch introduces #include <cxl/ras.h> in multiple files, but the header file itself is missing from the commit. --

commit 579799797ed7b991cdee3c9ccb1b1c1f0f8e3e23
Author: Dave Jiang <[email protected]>
Subject: PCI/CXL: Add RDPAS parsing support

This commit adds parsing for the RCEC Downstream Port Association Structure (RDPAS) from the CXL Early Discovery Table (CEDT). It stores the 64-bit base address for CXL.io in an xarray indexed by the RCEC segment and BDF.

drivers/cxl/acpi.c line 10

Quoted 9 lines
drivers/cxl/acpi.c +1 -0
index 127537628817e..e33cae64def9a 100644
@@ -7,6 +7,7 @@
 #include <linux/acpi.h>
 #include <linux/pci.h>
 #include <linux/node.h>
+#include <cxl/ras.h>

[Severity: Low]
Is this missing a new header file? The patch includes <cxl/ras.h> here and in aer_cxl_rch.c, but the file doesn't seem to be added in this commit, which will cause a build failure.

[ ... ]

Quoted 4 lines
drivers/pci/pcie/aer_cxl_rch.c +0 -0
index e471eefec9c40..3f3f6177a828b 100644

[ ... ]

drivers/pci/pcie/aer_cxl_rch.c line 41

Quoted 7 lines
drivers/pci/pcie/aer_cxl_rch.c +7 -0
+static int __cxl_parse_rdpas(struct acpi_cedt_rdpas *rdpas, struct device *dev)
+{
+	unsigned long index;
+	int rc;
+
+	/* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */
+	if (rdpas->protocol != CXL_RDPAS_PROTO_IO)

[Severity: High]
Does this need to validate the subtable length before accessing fields?

If the ACPI table provides a truncated RDPAS entry where header->length is less than sizeof(struct acpi_cedt_rdpas), accessing rdpas->protocol (and later rdpas->address) could result in an out-of-bounds read.

drivers/pci/pcie/aer_cxl_rch.c line 51

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+		return 0;
+
+	u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL);
+	if (!addr)
+		return -ENOMEM;
+
+	*addr = rdpas->address;
+
+	index = __rdpas_index(rdpas->segment, rdpas->bdf);
+	rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL);

[Severity: High]
Will this drop valid RDPAS entries if an RCEC is associated with multiple downstream ports?

According to the CXL specification, an RCEC with multiple downstream ports will have a separate RDPAS structure for each port. Since the xarray index is based only on the RCEC's segment and BDF, won't xa_insert() return -EBUSY for the subsequent downstream ports, causing them to be incorrectly discarded as duplicates?

Quoted 6 lines
drivers/pci/pcie/aer_cxl_rch.c +6 -0
+	if (rc) {
+		if (rc != -EBUSY)
+			return rc;
+
+		dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n",
+			rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#3 lore raw

Re: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support

In reply to Dave Jiang
Quoted 22 lines · Dave Jiang

Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.

A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.

The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.

A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.

[...]

Quoted 19 lines
drivers/pci/pcie/aer_cxl_rch.c +19 -0
+static int __cxl_parse_rdpas(struct acpi_cedt_rdpas *rdpas, struct device *dev)
+{
+	unsigned long index;
+	int rc;
+
+	/* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */
+	if (rdpas->protocol != CXL_RDPAS_PROTO_IO)
+		return 0;
+
+	u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL);
+	if (!addr)
+		return -ENOMEM;
+
+	*addr = rdpas->address;
+
+	index = __rdpas_index(rdpas->segment, rdpas->bdf);
+	rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL);
+	if (rc) {
+		if (rc != -EBUSY)

drivers/pci/pcie/aer_cxl_rch.c line 63

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+			return rc;
+
+		dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n",
+			rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+			FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+			FIELD_GET(GENMASK(2, 0), rdpas->bdf));
+		return 0;
+	}
+
+	retain_and_null_ptr(addr);

Miss releasing implementation for the added entry?

Or should destroy xarray in cxl_acpi_exit()?

Quoted 38 lines
drivers/pci/pcie/aer_cxl_rch.c +35 -0
+	dev_dbg(dev, "RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n",
+		rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+		FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+		FIELD_GET(GENMASK(2, 0), rdpas->bdf),
+		rdpas->address);
+
+	return 0;
+}
+
+int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
+		    const unsigned long end)
+{
+	struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header;
+	struct device *dev = arg;
+
+	return __cxl_parse_rdpas(rdpas, dev);
+}
+EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
+
+static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+{
+	u64 index;
+	u64 *addr;
+
+	index = rdpas_index(pci_domain_nr(rcec->bus), rcec->bus->number,
+			    PCI_SLOT(rcec->devfn), PCI_FUNC(rcec->devfn));
+	addr = xa_load(&cxl_rdpas, index);
+	if (!addr)
+		return -ENOENT;
+
+	*dsp_addr = *addr;
+
+	return 0;
+}
+
  static bool is_cxl_mem_dev(struct pci_dev *dev)
  {
  	/*
author maintainer
#4 lore raw

Re: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support

Quoted 48 lines · Li Ming
Quoted 22 lines · Dave Jiang

Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.

A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.

The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.

A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.

[...]

Quoted 23 lines
drivers/pci/pcie/aer_cxl_rch.c +23 -0
+static int __cxl_parse_rdpas(struct acpi_cedt_rdpas *rdpas, struct device *dev)
+{
+    unsigned long index;
+    int rc;
+
+    /* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */
+    if (rdpas->protocol != CXL_RDPAS_PROTO_IO)
+        return 0;
+
+    u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL);
+    if (!addr)
+        return -ENOMEM;
+
+    *addr = rdpas->address;
+
+    index = __rdpas_index(rdpas->segment, rdpas->bdf);
+    rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL);
+    if (rc) {
+        if (rc != -EBUSY)
+            return rc;
+
+        dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n",
+            rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
Quoted 8 lines
Quoted 6 lines

+            FIELD_GET(GENMASK(7, 3), rdpas->bdf), +            FIELD_GET(GENMASK(2, 0), rdpas->bdf)); +        return 0;
+    }
+
+    retain_and_null_ptr(addr);

Miss releasing implementation for the added entry?

Yup I'll add

DJ

Quoted 41 lines

Or should destroy xarray in cxl_acpi_exit()?

Quoted 38 lines

+    dev_dbg(dev, "RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n", +        rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+        FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+        FIELD_GET(GENMASK(2, 0), rdpas->bdf),
+        rdpas->address);

drivers/pci/pcie/aer_cxl_rch.c +30 -0
+
+    return 0;
+}
+
+int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
+            const unsigned long end)
+{
+    struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header;
+    struct device *dev = arg;
+
+    return __cxl_parse_rdpas(rdpas, dev);
+}
+EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
+
+static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+{
+    u64 index;
+    u64 *addr;
+
+    index = rdpas_index(pci_domain_nr(rcec->bus), rcec->bus->number,
+                PCI_SLOT(rcec->devfn), PCI_FUNC(rcec->devfn));
+    addr = xa_load(&cxl_rdpas, index);
+    if (!addr)
+        return -ENOENT;
+
+    *dsp_addr = *addr;
+
+    return 0;
+}
+

  static bool is_cxl_mem_dev(struct pci_dev *dev)
  {
      /*

author maintainer
#5 lore raw

Re: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support

Quoted 60 lines · Dave Jiang
Quoted 48 lines · Li Ming
Quoted 22 lines · Dave Jiang

Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.

A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.

The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.

A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.

[...]

Quoted 23 lines
drivers/pci/pcie/aer_cxl_rch.c +23 -0
+static int __cxl_parse_rdpas(struct acpi_cedt_rdpas *rdpas, struct device *dev)
+{
+    unsigned long index;
+    int rc;
+
+    /* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */
+    if (rdpas->protocol != CXL_RDPAS_PROTO_IO)
+        return 0;
+
+    u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL);
+    if (!addr)
+        return -ENOMEM;
+
+    *addr = rdpas->address;
+
+    index = __rdpas_index(rdpas->segment, rdpas->bdf);
+    rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL);
+    if (rc) {
+        if (rc != -EBUSY)
+            return rc;
+
+        dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n",
+            rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
Quoted 8 lines
Quoted 6 lines

+            FIELD_GET(GENMASK(7, 3), rdpas->bdf), +            FIELD_GET(GENMASK(2, 0), rdpas->bdf)); +        return 0;
+    }
+
+    retain_and_null_ptr(addr);

Miss releasing implementation for the added entry?

Quoted 1 line

Yup I'll add

Actually after thinking about it I think I'll make the xarray only parse once but have the lifetime of the entire kernel boot. There is no need to clean it up.

DJ

Quoted 45 lines

DJ

Quoted 41 lines

Or should destroy xarray in cxl_acpi_exit()?

Quoted 38 lines

+    dev_dbg(dev, "RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n", +        rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+        FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+        FIELD_GET(GENMASK(2, 0), rdpas->bdf),
+        rdpas->address);

drivers/pci/pcie/aer_cxl_rch.c +30 -0
+
+    return 0;
+}
+
+int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
+            const unsigned long end)
+{
+    struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header;
+    struct device *dev = arg;
+
+    return __cxl_parse_rdpas(rdpas, dev);
+}
+EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
+
+static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+{
+    u64 index;
+    u64 *addr;
+
+    index = rdpas_index(pci_domain_nr(rcec->bus), rcec->bus->number,
+                PCI_SLOT(rcec->devfn), PCI_FUNC(rcec->devfn));
+    addr = xa_load(&cxl_rdpas, index);
+    if (!addr)
+        return -ENOENT;
+
+    *dsp_addr = *addr;
+
+    return 0;
+}
+

  static bool is_cxl_mem_dev(struct pci_dev *dev)
  {
      /*

author maintainer
#6 lore raw

RFC PATCH 2/2 PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

In reply to Dave Jiang

The RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the cxl_rdpas xarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.

Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.

---
 drivers/pci/pcie/aer_cxl_rch.c | 103 +++++++++++++++++++++++++++++++--
 1 file changed, 99 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer_cxl_rch.c +99 -4
index 37bbecf495af..0fd6c4234856 100644
@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
 }
 EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
 
-static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+static int cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
 {
 	u64 index;
 	u64 *addr;
@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
 	return 0;
 }
 
+static u16 rcrb_to_aer(void __iomem *rcrb)
+{
+	u16 offset;
+	u32 cap_hdr;
+
+	/* Start from PCIe extended capabilities at offset 0x100 */
+	offset = PCI_CFG_SPACE_SIZE;
+	cap_hdr = readl(rcrb + offset);
+	if (cap_hdr == 0 || cap_hdr == ~0u)
+		return 0;
+
+	while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) {
+		offset = PCI_EXT_CAP_NEXT(cap_hdr);
+		if (!offset)
+			return 0;
+
+		if (offset >= SZ_4K)
+			return 0;
+
+		cap_hdr = readl(rcrb + offset);
+	}
+
+	return offset;
+}
+
+DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T))
+static u16 cxl_rch_get_err_src_id(u64 rcrb_base)
+{
+	u32 root_status, err_src;
+	void __iomem *aer_base;
+	u16 aer_offset;
+
+	void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K);
+	if (!rcrb)
+		return 0;
+
+	aer_offset = rcrb_to_aer(rcrb);
+	if (!aer_offset)
+		return 0;
+
+	aer_base = rcrb + aer_offset;
+
+	root_status = readl(aer_base + PCI_ERR_ROOT_STATUS);
+	if (!(root_status & (PCI_ERR_ROOT_COR_RCV | PCI_ERR_ROOT_UNCOR_RCV)))
+		return 0;
+
+	err_src = readl(aer_base + PCI_ERR_ROOT_ERR_SRC);
+
+	if (root_status & PCI_ERR_ROOT_UNCOR_RCV)
+		return FIELD_GET(GENMASK(31, 16), err_src);
+
+	if (root_status & PCI_ERR_ROOT_COR_RCV)
+		return FIELD_GET(GENMASK(15, 0), err_src);
+
+	return 0;
+}
+
+static bool cxl_rch_handled_error_by_rdpas(struct pci_dev *rcec,
+					   struct aer_err_info *info)
+{
+	u8 bus, devfn;
+	u64 dsp_addr;
+	u16 segment;
+	u16 src_id;
+	int rc;
+
+	rc = cxl_get_rdpas_by_rcec(rcec, &dsp_addr);
+	if (rc)
+		return false;
+
+	src_id = cxl_rch_get_err_src_id(dsp_addr);
+	if (!src_id)
+		return false;
+
+	/* Try uncorrectable error source first, then correctable */
+	segment = pci_domain_nr(rcec->bus);
+	bus = FIELD_GET(GENMASK(15, 8), src_id);
+	devfn = FIELD_GET(GENMASK(7, 0), src_id);
+
+	struct pci_dev *pdev __free(pci_dev_put) =
+		pci_get_domain_bus_and_slot(segment, bus, devfn);
+	if (!pdev)
+		return false;
+
+	cxl_forward_error(dev, info);
+	return true;
+}
+
 void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	/*
@@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
 	 * RCH's downstream port. Check and handle them in the CXL.mem
 	 * device driver.
 	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
-	    is_aer_internal_error(info))
-		pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)
+		return;
+
+	if (!is_aer_internal_error(info))
+		return;
+
+	if (cxl_rch_handled_error_by_rdpas(dev, info))
+		return;
+
+	pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
 }
 
 static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
Signature · 1 line
2.54.0
#7 lore raw

Re: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The AER capability parsing loop in `rcrb_to_aer()` lacks cycle detection and surprise removal…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The AER capability parsing loop in rcrb_to_aer() lacks cycle detection and surprise removal checks, leading to an infinite loop.
- [High] Reading AER registers in cxl_rch_get_err_src_id() can perform out-of-bounds MMIO reads beyond the mapped boundary.
- [Low] The patch introduces references to an undeclared variable dev and an undefined function cxl_forward_error(), while ignoring the correctly identified pdev. --

commit 268f954c78990287261b8d16aa1b0af35992909d
Author: Dave Jiang <[email protected]>

PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

This commit enables usage of RDPAS to shortcut error device discovery for CXL RCHs. It probes the RCRB directly to read the AER capability block and root status, rather than iterating through RCiEPs.

Quoted 4 lines
drivers/pci/pcie/aer_cxl_rch.c +0 -0
index 3f3f6177a828b..75da53bd762cf 100644

[ ... ]

Quoted 14 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
@@ -149,6 +149,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
 	return 0;
 }
 
+static u16 rcrb_to_aer(void __iomem *rcrb)
+{
+	u16 offset;
+	u32 cap_hdr;
+
+	/* Start from PCIe extended capabilities at offset 0x100 */
+	offset = PCI_CFG_SPACE_SIZE;
+	cap_hdr = readl(rcrb + offset);
+	if (cap_hdr == 0 || cap_hdr == ~0u)
+		return 0;

drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 171

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+
+	while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) {
+		offset = PCI_EXT_CAP_NEXT(cap_hdr);
+		if (!offset)
+			return 0;
+
+		if (offset >= SZ_4K)
+			return 0;
+
+		cap_hdr = readl(rcrb + offset);

[Severity: High]
Does this loop lack cycle detection and surprise removal checks?

If hardware fails or surprise removal occurs, readl() could return ~0u. The macro PCI_EXT_CAP_NEXT(~0u) yields 0xffc, which bypasses the !offset and offset >= SZ_4K checks, since 0xffc is less than 4096.

This could result in an infinite loop constantly reading from rcrb + 0xffc.

Quoted 13 lines
drivers/pci/pcie/aer_cxl_rch.c +13 -0
+	}
+
+	return offset;
+}
+
+DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T))
+static u16 cxl_rch_get_err_src_id(u64 rcrb_base)
+{
+	u32 root_status, err_src;
+	void __iomem *aer_base;
+	u16 aer_offset;
+
+	void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K);

drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 179

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+	if (!rcrb)
+		return 0;
+
+	aer_offset = rcrb_to_aer(rcrb);
+	if (!aer_offset)
+		return 0;
+
+	aer_base = rcrb + aer_offset;
+
+	root_status = readl(aer_base + PCI_ERR_ROOT_STATUS);

[Severity: High]
Can this result in an out-of-bounds MMIO read beyond the mapped boundary?

ioremap(rcrb_base, SZ_4K) requests exactly a 4096-byte MMIO mapping. While rcrb_to_aer() ensures the AER capability header starts within this region, adding register offsets like PCI_ERR_ROOT_STATUS or PCI_ERR_ROOT_ERR_SRC (0x34) to a capability base near the end of the block (e.g., 0xffc) will exceed the allocation boundary and potentially trigger a page fault.

Quoted 32 lines
drivers/pci/pcie/aer_cxl_rch.c +32 -0
+	if (!(root_status & (PCI_ERR_ROOT_COR_RCV | PCI_ERR_ROOT_UNCOR_RCV)))
+		return 0;
+
+	err_src = readl(aer_base + PCI_ERR_ROOT_ERR_SRC);
+
+	if (root_status & PCI_ERR_ROOT_UNCOR_RCV)
+		return FIELD_GET(GENMASK(31, 16), err_src);
+
+	if (root_status & PCI_ERR_ROOT_COR_RCV)
+		return FIELD_GET(GENMASK(15, 0), err_src);
+
+	return 0;
+}
+
+static bool cxl_rch_handled_error_by_rdpas(struct pci_dev *rcec,
+					   struct aer_err_info *info)
+{
+	u8 bus, devfn;
+	u64 dsp_addr;
+	u16 segment;
+	u16 src_id;
+	int rc;
+
+	rc = cxl_get_rdpas_by_rcec(rcec, &dsp_addr);
+	if (rc)
+		return false;
+
+	src_id = cxl_rch_get_err_src_id(dsp_addr);
+	if (!src_id)
+		return false;
+
+	/* Try uncorrectable error source first, then correctable */

drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 221

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+	segment = pci_domain_nr(rcec->bus);
+	bus = FIELD_GET(GENMASK(15, 8), src_id);
+	devfn = FIELD_GET(GENMASK(7, 0), src_id);
+
+	struct pci_dev *pdev __free(pci_dev_put) =
+		pci_get_domain_bus_and_slot(segment, bus, devfn);
+	if (!pdev)
+		return false;
+
+	cxl_forward_error(dev, info);

[Severity: Low]
Does this code introduce a compilation failure due to an undeclared variable and undefined function?

The variable dev is not declared within this scope (the passed device parameter is named rcec), and cxl_forward_error() does not appear to be defined.

Additionally, it seems the correctly identified pdev is completely unused.

Quoted 2 lines
drivers/pci/pcie/aer_cxl_rch.c +2 -0
+	return true;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
reviewer
#8 lore raw

Re: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

In reply to Dave Jiang
Quoted 100 lines · Dave Jiang

The RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the cxl_rdpas xarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.

Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.

---
  drivers/pci/pcie/aer_cxl_rch.c | 103 +++++++++++++++++++++++++++++++--
  1 file changed, 99 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer_cxl_rch.c +66 -1
index 37bbecf495af..0fd6c4234856 100644
@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
  }
  EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
 
-static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+static int cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
  {
  	u64 index;
  	u64 *addr;
@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
  	return 0;
  }
 
+static u16 rcrb_to_aer(void __iomem *rcrb)
+{
+	u16 offset;
+	u32 cap_hdr;
+
+	/* Start from PCIe extended capabilities at offset 0x100 */
+	offset = PCI_CFG_SPACE_SIZE;
+	cap_hdr = readl(rcrb + offset);
+	if (cap_hdr == 0 || cap_hdr == ~0u)
+		return 0;
+
+	while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) {
+		offset = PCI_EXT_CAP_NEXT(cap_hdr);
+		if (!offset)
+			return 0;
+
+		if (offset >= SZ_4K)
+			return 0;
+
+		cap_hdr = readl(rcrb + offset);
+	}
+
+	return offset;
+}
+
+DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T))
+static u16 cxl_rch_get_err_src_id(u64 rcrb_base)
+{
+	u32 root_status, err_src;
+	void __iomem *aer_base;
+	u16 aer_offset;
+
+	void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K);
+	if (!rcrb)
+		return 0;
+
+	aer_offset = rcrb_to_aer(rcrb);
+	if (!aer_offset)
+		return 0;
+
+	aer_base = rcrb + aer_offset;
+
+	root_status = readl(aer_base + PCI_ERR_ROOT_STATUS);
+	if (!(root_status & (PCI_ERR_ROOT_COR_RCV | PCI_ERR_ROOT_UNCOR_RCV)))
+		return 0;
+
+	err_src = readl(aer_base + PCI_ERR_ROOT_ERR_SRC);
+
+	if (root_status & PCI_ERR_ROOT_UNCOR_RCV)
+		return FIELD_GET(GENMASK(31, 16), err_src);
+
+	if (root_status & PCI_ERR_ROOT_COR_RCV)
+		return FIELD_GET(GENMASK(15, 0), err_src);
+
+	return 0;
+}
+
+static bool cxl_rch_handled_error_by_rdpas(struct pci_dev *rcec,
+					   struct aer_err_info *info)
+{
+	u8 bus, devfn;
+	u64 dsp_addr;
+	u16 segment;
+	u16 src_id;
+	int rc;

drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 211

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+
+	rc = cxl_get_rdpas_by_rcec(rcec, &dsp_addr);
+	if (rc)
+		return false;
+
+	src_id = cxl_rch_get_err_src_id(dsp_addr);
+	if (!src_id)
+		return false;
+
+	/* Try uncorrectable error source first, then correctable */

I am a little bit confused about this comment and cxl_rch_get_err_src_id(). I think aer_err_info->severify idendifies current ERR is a correctable error or not.

In cxl_rch_get_err_src_id() always handles UNCOR_ERR before COR_ERR, is it possible that getting an UNCOR_ERR src id for a COR_ERR aer_err_info?

E.g. aer_err_info is created for a COR_ERR, but the RCH DSP masks PCI_ERR_ROOT_UNCOR_RCV after the aer_err_info created.

Maybe I misunderstand something, please correct me if I am wrong.

Ming

Quoted 36 lines
drivers/pci/pcie/aer_cxl_rch.c +23 -3
+	segment = pci_domain_nr(rcec->bus);
+	bus = FIELD_GET(GENMASK(15, 8), src_id);
+	devfn = FIELD_GET(GENMASK(7, 0), src_id);
+
+	struct pci_dev *pdev __free(pci_dev_put) =
+		pci_get_domain_bus_and_slot(segment, bus, devfn);
+	if (!pdev)
+		return false;
+
+	cxl_forward_error(dev, info);
+	return true;
+}
+
  void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
  {
  	/*
@@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
  	 * RCH's downstream port. Check and handle them in the CXL.mem
  	 * device driver.
  	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
-	    is_aer_internal_error(info))
-		pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
+	if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)
+		return;
+
+	if (!is_aer_internal_error(info))
+		return;
+
+	if (cxl_rch_handled_error_by_rdpas(dev, info))
+		return;
+
+	pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
  }
 
  static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
author maintainer
#9 lore raw

Re: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery

Quoted 113 lines · Li Ming
Quoted 98 lines · Dave Jiang

The RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the cxl_rdpas xarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.

Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.

---
  drivers/pci/pcie/aer_cxl_rch.c | 103 +++++++++++++++++++++++++++++++--

  1 file changed, 99 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer_cxl_rch.c +65 -0
index 37bbecf495af..0fd6c4234856 100644
@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg,
   }
   EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi");
   -static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
+static int cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr)
   {
       u64 index;
       u64 *addr;
@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
       return 0;
   }
   +static u16 rcrb_to_aer(void __iomem *rcrb)
+{
+    u16 offset;
+    u32 cap_hdr;
+
+    /* Start from PCIe extended capabilities at offset 0x100 */
+    offset = PCI_CFG_SPACE_SIZE;
+    cap_hdr = readl(rcrb + offset);
+    if (cap_hdr == 0 || cap_hdr == ~0u)
+        return 0;
+
+    while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) {
+        offset = PCI_EXT_CAP_NEXT(cap_hdr);
+        if (!offset)
+            return 0;
+
+        if (offset >= SZ_4K)
+            return 0;
+
+        cap_hdr = readl(rcrb + offset);
+    }
+
+    return offset;
+}
+
+DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T))
+static u16 cxl_rch_get_err_src_id(u64 rcrb_base)
+{
+    u32 root_status, err_src;
+    void __iomem *aer_base;
+    u16 aer_offset;
+
+    void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K);
+    if (!rcrb)
+        return 0;
+
+    aer_offset = rcrb_to_aer(rcrb);
+    if (!aer_offset)
+        return 0;
+
+    aer_base = rcrb + aer_offset;
+
+    root_status = readl(aer_base + PCI_ERR_ROOT_STATUS);
+    if (!(root_status & (PCI_ERR_ROOT_COR_RCV | PCI_ERR_ROOT_UNCOR_RCV)))
+        return 0;
+
+    err_src = readl(aer_base + PCI_ERR_ROOT_ERR_SRC);
+
+    if (root_status & PCI_ERR_ROOT_UNCOR_RCV)
+        return FIELD_GET(GENMASK(31, 16), err_src);
+
+    if (root_status & PCI_ERR_ROOT_COR_RCV)
+        return FIELD_GET(GENMASK(15, 0), err_src);
+
+    return 0;
+}
+
+static bool cxl_rch_handled_error_by_rdpas(struct pci_dev *rcec,
+                       struct aer_err_info *info)
+{
+    u8 bus, devfn;
+    u64 dsp_addr;
+    u16 segment;
+    u16 src_id;
+    int rc;

drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 210

Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10 -0
+
+    rc = cxl_get_rdpas_by_rcec(rcec, &dsp_addr);
+    if (rc)
+        return false;
+
+    src_id = cxl_rch_get_err_src_id(dsp_addr);
+    if (!src_id)
+        return false;
+
+    /* Try uncorrectable error source first, then correctable */

I am a little bit confused about this comment and cxl_rch_get_err_src_id(). I think aer_err_info->severify idendifies current ERR is a correctable error or not.

Quoted 5 lines

In cxl_rch_get_err_src_id() always handles UNCOR_ERR before COR_ERR, is it possible that getting an UNCOR_ERR src id for a COR_ERR aer_err_info?

E.g. aer_err_info is created for a COR_ERR, but the RCH DSP masks PCI_ERR_ROOT_UNCOR_RCV after the aer_err_info created.

Maybe I misunderstand something, please correct me if I am wrong.

No. I was blindly just looking at the registers. I need to take into account what came from the aer_err_info.

DJ

Quoted 38 lines

Ming

Quoted 35 lines
drivers/pci/pcie/aer_cxl_rch.c +23 -3
+    segment = pci_domain_nr(rcec->bus);
+    bus = FIELD_GET(GENMASK(15, 8), src_id);
+    devfn = FIELD_GET(GENMASK(7, 0), src_id);
+
+    struct pci_dev *pdev __free(pci_dev_put) =
+        pci_get_domain_bus_and_slot(segment, bus, devfn);
+    if (!pdev)
+        return false;
+
+    cxl_forward_error(dev, info);
+    return true;
+}
+
   void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
   {
       /*
@@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
        * RCH's downstream port. Check and handle them in the CXL.mem
        * device driver.
        */
-    if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
-        is_aer_internal_error(info))
-        pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
+    if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)
+        return;
+
+    if (!is_aer_internal_error(info))
+        return;
+
+    if (cxl_rch_handled_error_by_rdpas(dev, info))
+        return;
+
+    pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
   }
     static int handles_cxl_error_iter(struct pci_dev *dev, void *data)