PCI/CXL: Add RDPAS support for CXL.io
Dave Jiang
maintainercover PCI/CXL: Add RDPAS support for CXL.io
The series add RCEC Downstream Port Assocation Structure (RDPAS) parsing support to CXL.io. RDPAS is an ACPI table that is part of the CXL Early Discovery Table (CEDT) defined in CXL specification r4.0 9.18.1.5. It provides the mapping between RCEC and downstream ports. With RDPAS, the error device can be directly found when an error is reported on RCEC, without walking a number of RCiEP in order to determine which one reported the error. While CXL.cachemem is supported by RDPAS, there is no easy way to discover the source id of the error and therefore finding the Linux PCI object for the RCiEP. The intention here is to accelerate the discovery of the error by directly locating the error device with the given information.
This series is based on top of Terry's CXL error protocol series [1]. Looking for comments on the series WRT if it makes sense to add on top of Terry's error handling for RCH/RCD devices.
[1]: https://lore.kernel.org/linux-cxl/[email protected]/T/#t
Dave Jiang (2):
PCI/CXL: Add RDPAS parsing support
PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
drivers/cxl/acpi.c | 5 + drivers/pci/pcie/aer_cxl_rch.c | 190 ++++++++++++++++++++++++++++++++- 2 files changed, 192 insertions(+), 3 deletions(-)
base-commit: 721d08f9e0f07f8f1b93e7767762b565f088cdcd
Signature · 1 line
2.54.0
Dave Jiang
maintainerRFC PATCH 1/2 PCI/CXL: Add RDPAS parsing support
Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.
A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.
The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.
A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.
--- drivers/cxl/acpi.c | 5 ++ drivers/pci/pcie/aer_cxl_rch.c | 89 ++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+)
drivers/cxl/acpi.c +5 -0
@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include <linux/node.h> +#include <cxl/ras.h> #include <asm/div64.h> #include "cxlpci.h" #include "cxl.h"
@@ -933,6 +934,10 @@ static int cxl_acpi_probe(struct platform_device *pdev)if (rc <0 )return -ENXIO ; + rc =acpi_table_parse_cedt (ACPI_CEDT_TYPE_RDPAS , cxl_parse_rdpas, host); +if (rc <0 ) +dev_dbg (host,"No RDPAS entries found or failed to parse\n" ); + rc =add_cxl_resources (cxl_res);if (rc)return rc;
drivers/pci/pcie/aer_cxl_rch.c +89 -0
98 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] RDPAS entries for an RCEC with multiple downstream ports are incorrectly dropped as duplicates.…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] RDPAS entries for an RCEC with multiple downstream ports are incorrectly dropped as duplicates.
- [High] Missing length and boundary validation for ACPI CEDT RDPAS subtable can lead to out-of-bounds memory access.
- [Low] The patch introduces #include <cxl/ras.h> in multiple files, but the header file itself is missing from the commit. --
commit 579799797ed7b991cdee3c9ccb1b1c1f0f8e3e23
Author: Dave Jiang <[email protected]>
Subject: PCI/CXL: Add RDPAS parsing support
This commit adds parsing for the RCEC Downstream Port Association Structure (RDPAS) from the CXL Early Discovery Table (CEDT). It stores the 64-bit base address for CXL.io in an xarray indexed by the RCEC segment and BDF.
drivers/cxl/acpi.c line 10
Quoted 9 lines
drivers/cxl/acpi.c +1
-0@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include <linux/node.h> +#include <cxl/ras.h>
[Severity: Low]
Is this missing a new header file? The patch includes <cxl/ras.h> here and in aer_cxl_rch.c, but the file doesn't seem to be added in this commit, which will cause a build failure.
[ ... ]
Quoted 4 lines
drivers/pci/pcie/aer_cxl_rch.c +0
-0
[ ... ]
drivers/pci/pcie/aer_cxl_rch.c line 41
Quoted 7 lines
drivers/pci/pcie/aer_cxl_rch.c +7
-0+static int __cxl_parse_rdpas (struct acpi_cedt_rdpas *rdpas,struct device *dev) +{ +unsigned long index; +int rc; + +/* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */ +if (rdpas->protocol !=CXL_RDPAS_PROTO_IO )
[Severity: High]
Does this need to validate the subtable length before accessing fields?
If the ACPI table provides a truncated RDPAS entry where header->length is less than sizeof(struct acpi_cedt_rdpas), accessing rdpas->protocol (and later rdpas->address) could result in an out-of-bounds read.
drivers/pci/pcie/aer_cxl_rch.c line 51
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+return 0 ; + + u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL); +if (!addr) +return -ENOMEM ; + + *addr = rdpas->address; + + index =__rdpas_index (rdpas->segment, rdpas->bdf); + rc =xa_insert (&cxl_rdpas, index, addr,GFP_KERNEL );
[Severity: High]
Will this drop valid RDPAS entries if an RCEC is associated with multiple downstream ports?
According to the CXL specification, an RCEC with multiple downstream ports will have a separate RDPAS structure for each port. Since the xarray index is based only on the RCEC's segment and BDF, won't xa_insert() return -EBUSY for the subsequent downstream ports, causing them to be incorrectly discarded as duplicates?
Quoted 6 lines
drivers/pci/pcie/aer_cxl_rch.c +6
-0+if (rc) { +if (rc != -EBUSY ) +return rc; + + dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n", + rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Li Ming
reviewerRe: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support
In reply to Dave JiangQuoted 22 lines · Dave Jiang
Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.
A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.
The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.
A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.
Signed-off-by Dave Jiang <[email protected]>
[...]
Quoted 19 lines
drivers/pci/pcie/aer_cxl_rch.c +19
-0+static int __cxl_parse_rdpas (struct acpi_cedt_rdpas *rdpas,struct device *dev) +{ +unsigned long index; +int rc; + +/* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */ +if (rdpas->protocol !=CXL_RDPAS_PROTO_IO ) +return 0 ; + + u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL); +if (!addr) +return -ENOMEM ; + + *addr = rdpas->address; + + index =__rdpas_index (rdpas->segment, rdpas->bdf); + rc =xa_insert (&cxl_rdpas, index, addr,GFP_KERNEL ); +if (rc) { +if (rc != -EBUSY )
drivers/pci/pcie/aer_cxl_rch.c line 63
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+return rc; + +dev_dbg (dev,"Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n" , + rdpas->segment,FIELD_GET (GENMASK (15 ,8 ), rdpas->bdf), +FIELD_GET (GENMASK (7 ,3 ), rdpas->bdf), +FIELD_GET (GENMASK (2 ,0 ), rdpas->bdf)); +return 0 ; + } + +retain_and_null_ptr (addr);
Miss releasing implementation for the added entry?
Or should destroy xarray in cxl_acpi_exit()?
Quoted 38 lines
drivers/pci/pcie/aer_cxl_rch.c +35
-0+dev_dbg (dev,"RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n" , + rdpas->segment,FIELD_GET (GENMASK (15 ,8 ), rdpas->bdf), +FIELD_GET (GENMASK (7 ,3 ), rdpas->bdf), +FIELD_GET (GENMASK (2 ,0 ), rdpas->bdf), + rdpas->address); + +return 0 ; +} + +int cxl_parse_rdpas (union acpi_subtable_headers *header,void *arg, +const unsigned long end) +{ +struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header; +struct device *dev = arg; + +return __cxl_parse_rdpas (rdpas, dev); +} +EXPORT_SYMBOL_FOR_MODULES (cxl_parse_rdpas,"cxl_acpi" ); + +static int __maybe_unused cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) +{ +u64 index; +u64 *addr; + + index =rdpas_index (pci_domain_nr (rcec->bus), rcec->bus->number, +PCI_SLOT (rcec->devfn),PCI_FUNC (rcec->devfn)); + addr =xa_load (&cxl_rdpas, index); +if (!addr) +return -ENOENT ; + + *dsp_addr = *addr; + +return 0 ; +} +static bool is_cxl_mem_dev (struct pci_dev *dev) {/*
Dave Jiang
maintainerRe: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support
Quoted 48 lines · Li Ming
Quoted 22 lines · Dave Jiang
Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.
A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.
The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.
A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.
Signed-off-by Dave Jiang <[email protected]>[...]
Quoted 23 lines
drivers/pci/pcie/aer_cxl_rch.c +23
-0+static int __cxl_parse_rdpas (struct acpi_cedt_rdpas *rdpas,struct device *dev) +{ + unsigned long index; + int rc; + + /* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */ + if (rdpas->protocol != CXL_RDPAS_PROTO_IO) + return 0; + + u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL); + if (!addr) + return -ENOMEM; + + *addr = rdpas->address; + + index = __rdpas_index(rdpas->segment, rdpas->bdf); + rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL); + if (rc) { + if (rc != -EBUSY) + return rc; + + dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n", + rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),
Quoted 8 lines
Quoted 6 lines
+
FIELD_GET(GENMASK(7, 3), rdpas->bdf), +FIELD_GET(GENMASK(2, 0), rdpas->bdf)); + return 0;
+ }
+
+retain_and_null_ptr(addr);Miss releasing implementation for the added entry?
Yup I'll add
DJ
Quoted 41 lines
Or should destroy xarray in
cxl_acpi_exit()?Quoted 38 lines
+ dev_dbg(dev, "RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n", +
rdpas->segment,FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+FIELD_GET(GENMASK(2, 0), rdpas->bdf),
+rdpas->address);drivers/pci/pcie/aer_cxl_rch.c +30
-0+ + return 0; +} + +int cxl_parse_rdpas (union acpi_subtable_headers *header,void *arg, + const unsigned long end) +{ + struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header; + struct device *dev = arg; + + return __cxl_parse_rdpas(rdpas, dev); +} +EXPORT_SYMBOL_FOR_MODULES (cxl_parse_rdpas,"cxl_acpi" ); + +static int __maybe_unused cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) +{ + u64 index; + u64 *addr; + + index = rdpas_index(pci_domain_nr(rcec->bus), rcec->bus->number, + PCI_SLOT(rcec->devfn), PCI_FUNC(rcec->devfn)); + addr = xa_load(&cxl_rdpas, index); + if (!addr) + return -ENOENT; + + *dsp_addr = *addr; + + return 0; +} +static bool
is_cxl_mem_dev(struct pci_dev *dev)
{
/*
Dave Jiang
maintainerRe: [RFC PATCH 1/2] PCI/CXL: Add RDPAS parsing support
Quoted 60 lines · Dave Jiang
Quoted 48 lines · Li Ming
Quoted 22 lines · Dave Jiang
Add parsing of the RCEC Downstream Port Association Structure (RDPAS), which is a structure defined in the CXL spec r4.0 9.18.1.5. This structure allows error handler to locate the downstream port(s) that report errors to a given Root Complex Event Collector (RCEC). The structure is part of CXL Early Discovery Table (CEDT) and can be parsed like other CEDT tables.
A base address is provided in the RDPAS structure where depending on the protocol field, it is the RCRB base associated with the downstream port for CXL.io or the Component Base Register base associated with the downstream port for CXL.cachemem.
The 64bit address of the base address field will be stored in an xarray indexed by the combination of the RCEC segment plus the BDF in a 32bit field. Currently only the CXL.io base address will be stored. If the protocol is CXL.cachemem, there is no way to retrieve a source id and therefore identifying the BDF of the error device.
A helper is also added to retrieve the base address based on the segment and BDF of the RCEC.
Signed-off-by Dave Jiang <[email protected]>[...]
Quoted 23 lines
drivers/pci/pcie/aer_cxl_rch.c +23
-0+static int __cxl_parse_rdpas (struct acpi_cedt_rdpas *rdpas,struct device *dev) +{ + unsigned long index; + int rc; + + /* Skip CXL.cachemem as the current RDPAS scheme only supports CXL.io */ + if (rdpas->protocol != CXL_RDPAS_PROTO_IO) + return 0; + + u64 *addr __free(kfree) = kzalloc(sizeof(*addr), GFP_KERNEL); + if (!addr) + return -ENOMEM; + + *addr = rdpas->address; + + index = __rdpas_index(rdpas->segment, rdpas->bdf); + rc = xa_insert(&cxl_rdpas, index, addr, GFP_KERNEL); + if (rc) { + if (rc != -EBUSY) + return rc; + + dev_dbg(dev, "Duplicate RDPAS: PCI %04x:%02lx:%02lx.%ld\n", + rdpas->segment, FIELD_GET(GENMASK(15, 8), rdpas->bdf),Quoted 8 lines
Quoted 6 lines
+
FIELD_GET(GENMASK(7, 3), rdpas->bdf), +FIELD_GET(GENMASK(2, 0), rdpas->bdf)); + return 0;
+ }
+
+retain_and_null_ptr(addr);Miss releasing implementation for the added entry?
Quoted 1 line
Yup I'll add
Actually after thinking about it I think I'll make the xarray only parse once but have the lifetime of the entire kernel boot. There is no need to clean it up.
DJ
Quoted 45 lines
DJ
Quoted 41 lines
Or should destroy xarray in
cxl_acpi_exit()?Quoted 38 lines
+ dev_dbg(dev, "RDPAS entry: PCI %04x:%02lx:%02lx.%ld -> CXL addr %016llx\n", +
rdpas->segment,FIELD_GET(GENMASK(15, 8), rdpas->bdf),
+FIELD_GET(GENMASK(7, 3), rdpas->bdf),
+FIELD_GET(GENMASK(2, 0), rdpas->bdf),
+rdpas->address);drivers/pci/pcie/aer_cxl_rch.c +30
-0+ + return 0; +} + +int cxl_parse_rdpas (union acpi_subtable_headers *header,void *arg, + const unsigned long end) +{ + struct acpi_cedt_rdpas *rdpas = (struct acpi_cedt_rdpas *)header; + struct device *dev = arg; + + return __cxl_parse_rdpas(rdpas, dev); +} +EXPORT_SYMBOL_FOR_MODULES (cxl_parse_rdpas,"cxl_acpi" ); + +static int __maybe_unused cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) +{ + u64 index; + u64 *addr; + + index = rdpas_index(pci_domain_nr(rcec->bus), rcec->bus->number, + PCI_SLOT(rcec->devfn), PCI_FUNC(rcec->devfn)); + addr = xa_load(&cxl_rdpas, index); + if (!addr) + return -ENOENT; + + *dsp_addr = *addr; + + return 0; +} +static bool
is_cxl_mem_dev(struct pci_dev *dev)
{
/*
Dave Jiang
maintainerRFC PATCH 2/2 PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
In reply to Dave JiangThe RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the cxl_rdpas xarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.
Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.
--- drivers/pci/pcie/aer_cxl_rch.c | 103 +++++++++++++++++++++++++++++++-- 1 file changed, 99 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer_cxl_rch.c +99 -4
@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg, }EXPORT_SYMBOL_FOR_MODULES (cxl_parse_rdpas,"cxl_acpi" ); -static int __maybe_unused cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) +static int cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) {u64 index;u64 *addr;
@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)return 0 ; } +static u16 rcrb_to_aer (void __iomem *rcrb) +{ +u16 offset; +u32 cap_hdr; + +/* Start from PCIe extended capabilities at offset 0x100 */ + offset =PCI_CFG_SPACE_SIZE ; + cap_hdr =readl (rcrb + offset); +if (cap_hdr ==0 || cap_hdr == ~0u ) +return 0 ; + +while (PCI_EXT_CAP_ID (cap_hdr) !=PCI_EXT_CAP_ID_ERR ) { + offset =PCI_EXT_CAP_NEXT (cap_hdr); +if (!offset) +return 0 ; + +if (offset >=SZ_4K ) +return 0 ; + + cap_hdr =readl (rcrb + offset); + } + +return offset; +} + +DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T)) +static u16 cxl_rch_get_err_src_id (u64 rcrb_base) +{ +u32 root_status, err_src; +void __iomem *aer_base; +u16 aer_offset; + + void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K); +if (!rcrb) +return 0 ; + + aer_offset =rcrb_to_aer (rcrb); +if (!aer_offset) +return 0 ; + + aer_base = rcrb + aer_offset; + + root_status =readl (aer_base +PCI_ERR_ROOT_STATUS ); +if (!(root_status & (PCI_ERR_ROOT_COR_RCV |PCI_ERR_ROOT_UNCOR_RCV ))) +return 0 ; + + err_src =readl (aer_base +PCI_ERR_ROOT_ERR_SRC ); + +if (root_status &PCI_ERR_ROOT_UNCOR_RCV ) +return FIELD_GET (GENMASK (31 ,16 ), err_src); + +if (root_status &PCI_ERR_ROOT_COR_RCV ) +return FIELD_GET (GENMASK (15 ,0 ), err_src); + +return 0 ; +} + +static bool cxl_rch_handled_error_by_rdpas (struct pci_dev *rcec, +struct aer_err_info *info) +{ +u8 bus, devfn; +u64 dsp_addr; +u16 segment; +u16 src_id; +int rc; + + rc =cxl_get_rdpas_by_rcec (rcec, &dsp_addr); +if (rc) +return false; + + src_id =cxl_rch_get_err_src_id (dsp_addr); +if (!src_id) +return false; + +/* Try uncorrectable error source first, then correctable */ + segment =pci_domain_nr (rcec->bus); + bus =FIELD_GET (GENMASK (15 ,8 ), src_id); + devfn =FIELD_GET (GENMASK (7 ,0 ), src_id); + + struct pci_dev *pdev __free(pci_dev_put) = +pci_get_domain_bus_and_slot (segment, bus, devfn); +if (!pdev) +return false; + +cxl_forward_error (dev, info); +return true; +} +void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) {/*
@@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)* RCH's downstream port. Check and handle them in the CXL.mem * device driver. */ -if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC && -is_aer_internal_error (info)) -pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) +return ; + +if (!is_aer_internal_error (info)) +return ; + +if (cxl_rch_handled_error_by_rdpas (dev, info)) +return ; + +pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); }static int handles_cxl_error_iter (struct pci_dev *dev,void *data)
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The AER capability parsing loop in `rcrb_to_aer()` lacks cycle detection and surprise removal…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The AER capability parsing loop in rcrb_to_aer() lacks cycle detection and surprise removal checks, leading to an infinite loop.
- [High] Reading AER registers in cxl_rch_get_err_src_id() can perform out-of-bounds MMIO reads beyond the mapped boundary.
- [Low] The patch introduces references to an undeclared variable dev and an undefined function cxl_forward_error(), while ignoring the correctly identified pdev. --
commit 268f954c78990287261b8d16aa1b0af35992909d
Author: Dave Jiang <[email protected]>
PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
This commit enables usage of RDPAS to shortcut error device discovery for CXL RCHs. It probes the RCRB directly to read the AER capability block and root status, rather than iterating through RCiEPs.
Quoted 4 lines
drivers/pci/pcie/aer_cxl_rch.c +0
-0
[ ... ]
Quoted 14 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0@@ -149,6 +149,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)return 0 ; } +static u16 rcrb_to_aer (void __iomem *rcrb) +{ +u16 offset; +u32 cap_hdr; + +/* Start from PCIe extended capabilities at offset 0x100 */ + offset =PCI_CFG_SPACE_SIZE ; + cap_hdr =readl (rcrb + offset); +if (cap_hdr ==0 || cap_hdr == ~0u ) +return 0 ;
drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 171
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+ +while (PCI_EXT_CAP_ID (cap_hdr) !=PCI_EXT_CAP_ID_ERR ) { + offset =PCI_EXT_CAP_NEXT (cap_hdr); +if (!offset) +return 0 ; + +if (offset >=SZ_4K ) +return 0 ; + + cap_hdr =readl (rcrb + offset);
[Severity: High]
Does this loop lack cycle detection and surprise removal checks?
If hardware fails or surprise removal occurs, readl() could return ~0u. The macro PCI_EXT_CAP_NEXT(~0u) yields 0xffc, which bypasses the !offset and offset >= SZ_4K checks, since 0xffc is less than 4096.
This could result in an infinite loop constantly reading from rcrb + 0xffc.
Quoted 13 lines
drivers/pci/pcie/aer_cxl_rch.c +13
-0+ } + +return offset; +} + +DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T)) +static u16 cxl_rch_get_err_src_id (u64 rcrb_base) +{ +u32 root_status, err_src; +void __iomem *aer_base; +u16 aer_offset; + + void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K);
drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 179
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+if (!rcrb) +return 0 ; + + aer_offset =rcrb_to_aer (rcrb); +if (!aer_offset) +return 0 ; + + aer_base = rcrb + aer_offset; + + root_status =readl (aer_base +PCI_ERR_ROOT_STATUS );
[Severity: High]
Can this result in an out-of-bounds MMIO read beyond the mapped boundary?
ioremap(rcrb_base, SZ_4K) requests exactly a 4096-byte MMIO mapping. While rcrb_to_aer() ensures the AER capability header starts within this region, adding register offsets like PCI_ERR_ROOT_STATUS or PCI_ERR_ROOT_ERR_SRC (0x34) to a capability base near the end of the block (e.g., 0xffc) will exceed the allocation boundary and potentially trigger a page fault.
Quoted 32 lines
drivers/pci/pcie/aer_cxl_rch.c +32
-0+if (!(root_status & (PCI_ERR_ROOT_COR_RCV |PCI_ERR_ROOT_UNCOR_RCV ))) +return 0 ; + + err_src =readl (aer_base +PCI_ERR_ROOT_ERR_SRC ); + +if (root_status &PCI_ERR_ROOT_UNCOR_RCV ) +return FIELD_GET (GENMASK (31 ,16 ), err_src); + +if (root_status &PCI_ERR_ROOT_COR_RCV ) +return FIELD_GET (GENMASK (15 ,0 ), err_src); + +return 0 ; +} + +static bool cxl_rch_handled_error_by_rdpas (struct pci_dev *rcec, +struct aer_err_info *info) +{ +u8 bus, devfn; +u64 dsp_addr; +u16 segment; +u16 src_id; +int rc; + + rc =cxl_get_rdpas_by_rcec (rcec, &dsp_addr); +if (rc) +return false; + + src_id =cxl_rch_get_err_src_id (dsp_addr); +if (!src_id) +return false; + +/* Try uncorrectable error source first, then correctable */
drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 221
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+ segment =pci_domain_nr (rcec->bus); + bus =FIELD_GET (GENMASK (15 ,8 ), src_id); + devfn =FIELD_GET (GENMASK (7 ,0 ), src_id); + + struct pci_dev *pdev __free(pci_dev_put) = +pci_get_domain_bus_and_slot (segment, bus, devfn); +if (!pdev) +return false; + +cxl_forward_error (dev, info);
[Severity: Low]
Does this code introduce a compilation failure due to an undeclared variable and undefined function?
The variable dev is not declared within this scope (the passed device parameter is named rcec), and cxl_forward_error() does not appear to be defined.
Additionally, it seems the correctly identified pdev is completely unused.
Quoted 2 lines
drivers/pci/pcie/aer_cxl_rch.c +2
-0+return true; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Li Ming
reviewerRe: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
In reply to Dave JiangQuoted 100 lines · Dave Jiang
The RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the
cxl_rdpasxarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.
Signed-off-by Dave Jiang <[email protected]>--- drivers/pci/pcie/aer_cxl_rch.c | 103 +++++++++++++++++++++++++++++++-- 1 file changed, 99 insertions(+), 4 deletions(-)drivers/pci/pcie/aer_cxl_rch.c +66
-1@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg, }EXPORT_SYMBOL_FOR_MODULES (cxl_parse_rdpas,"cxl_acpi" ); -static int __maybe_unused cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) +static int cxl_get_rdpas_by_rcec (struct pci_dev *rcec,u64 *dsp_addr) {u64 index;u64 *addr;@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)return 0 ; } +static u16 rcrb_to_aer (void __iomem *rcrb) +{ +u16 offset; +u32 cap_hdr; + +/* Start from PCIe extended capabilities at offset 0x100 */ + offset =PCI_CFG_SPACE_SIZE ; + cap_hdr =readl (rcrb + offset); +if (cap_hdr ==0 || cap_hdr == ~0u ) +return 0 ; + +while (PCI_EXT_CAP_ID (cap_hdr) !=PCI_EXT_CAP_ID_ERR ) { + offset =PCI_EXT_CAP_NEXT (cap_hdr); +if (!offset) +return 0 ; + +if (offset >=SZ_4K ) +return 0 ; + + cap_hdr =readl (rcrb + offset); + } + +return offset; +} + +DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T)) +static u16 cxl_rch_get_err_src_id (u64 rcrb_base) +{ +u32 root_status, err_src; +void __iomem *aer_base; +u16 aer_offset; + + void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K); +if (!rcrb) +return 0 ; + + aer_offset =rcrb_to_aer (rcrb); +if (!aer_offset) +return 0 ; + + aer_base = rcrb + aer_offset; + + root_status =readl (aer_base +PCI_ERR_ROOT_STATUS ); +if (!(root_status & (PCI_ERR_ROOT_COR_RCV |PCI_ERR_ROOT_UNCOR_RCV ))) +return 0 ; + + err_src =readl (aer_base +PCI_ERR_ROOT_ERR_SRC ); + +if (root_status &PCI_ERR_ROOT_UNCOR_RCV ) +return FIELD_GET (GENMASK (31 ,16 ), err_src); + +if (root_status &PCI_ERR_ROOT_COR_RCV ) +return FIELD_GET (GENMASK (15 ,0 ), err_src); + +return 0 ; +} + +static bool cxl_rch_handled_error_by_rdpas (struct pci_dev *rcec, +struct aer_err_info *info) +{ +u8 bus, devfn; +u64 dsp_addr; +u16 segment; +u16 src_id; +int rc;
drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 211
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+ + rc =cxl_get_rdpas_by_rcec (rcec, &dsp_addr); +if (rc) +return false; + + src_id =cxl_rch_get_err_src_id (dsp_addr); +if (!src_id) +return false; + +/* Try uncorrectable error source first, then correctable */
I am a little bit confused about this comment and cxl_rch_get_err_src_id(). I think aer_err_info->severify idendifies current ERR is a correctable error or not.
In cxl_rch_get_err_src_id() always handles UNCOR_ERR before COR_ERR, is it possible that getting an UNCOR_ERR src id for a COR_ERR aer_err_info?
E.g. aer_err_info is created for a COR_ERR, but the RCH DSP masks PCI_ERR_ROOT_UNCOR_RCV after the aer_err_info created.
Maybe I misunderstand something, please correct me if I am wrong.
Ming
Quoted 36 lines
drivers/pci/pcie/aer_cxl_rch.c +23
-3+ segment =pci_domain_nr (rcec->bus); + bus =FIELD_GET (GENMASK (15 ,8 ), src_id); + devfn =FIELD_GET (GENMASK (7 ,0 ), src_id); + + struct pci_dev *pdev __free(pci_dev_put) = +pci_get_domain_bus_and_slot (segment, bus, devfn); +if (!pdev) +return false; + +cxl_forward_error (dev, info); +return true; +} +void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) {/* @@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)* RCH's downstream port. Check and handle them in the CXL.mem * device driver. */ -if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC && -is_aer_internal_error (info)) -pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) +return ; + +if (!is_aer_internal_error (info)) +return ; + +if (cxl_rch_handled_error_by_rdpas (dev, info)) +return ; + +pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); }static int handles_cxl_error_iter (struct pci_dev *dev,void *data)
Dave Jiang
maintainerRe: [RFC PATCH 2/2] PCI/CXL: Enable usage of RDPAS to shortcut error device discovery
Quoted 113 lines · Li Ming
Quoted 98 lines · Dave Jiang
The RDPAS allows the CXL RCH error handler to find the device directly instead of iterating through a set number of RCiEP in order to discover which device triggered an error. For the CXL.io protocol, the base address provided from the
cxl_rdpasxarray points to the RCRB of the device. The RCRB mirrors the configuration space of the device via MMIO. The error handler can walk the RCRB to find the AER capability block and therefore read the root status as well as the error source in order to determine the BDF of the error device.Change the current RCH error handler behavior so it will probe the RCRB first to see if the error device can be discovered quickly before falling back to the current method of iterating through RCiEPs.
Signed-off-by Dave Jiang <[email protected]>---
drivers/pci/pcie/aer_cxl_rch.c| 103 +++++++++++++++++++++++++++++++--1 file changed, 99 insertions(+), 4 deletions(-)drivers/pci/pcie/aer_cxl_rch.c +65
-0@@ -80,7 +80,7 @@ int cxl_parse_rdpas(union acpi_subtable_headers *header, void *arg, } EXPORT_SYMBOL_FOR_MODULES(cxl_parse_rdpas, "cxl_acpi"); -static int __maybe_unused cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr) +static int cxl_get_rdpas_by_rcec(struct pci_dev *rcec, u64 *dsp_addr) { u64 index; u64 *addr;@@ -134,6 +134,94 @@ static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) return 0; } +static u16 rcrb_to_aer(void __iomem *rcrb) +{ + u16 offset; + u32 cap_hdr; + + /* Start from PCIe extended capabilities at offset 0x100 */ + offset = PCI_CFG_SPACE_SIZE; + cap_hdr = readl(rcrb + offset); + if (cap_hdr == 0 || cap_hdr == ~0u) + return 0; + + while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) { + offset = PCI_EXT_CAP_NEXT(cap_hdr); + if (!offset) + return 0; + + if (offset >= SZ_4K) + return 0; + + cap_hdr = readl(rcrb + offset); + } + + return offset; +} + +DEFINE_FREE(iounmap, void __iomem *, if (_T) iounmap(_T)) +static u16 cxl_rch_get_err_src_id (u64 rcrb_base) +{ + u32 root_status, err_src; + void __iomem *aer_base; + u16 aer_offset; + + void __iomem *rcrb __free(iounmap) = ioremap(rcrb_base, SZ_4K); + if (!rcrb) + return 0; + + aer_offset = rcrb_to_aer(rcrb); + if (!aer_offset) + return 0; + + aer_base = rcrb + aer_offset; + + root_status = readl(aer_base + PCI_ERR_ROOT_STATUS); + if (!(root_status & (PCI_ERR_ROOT_COR_RCV | PCI_ERR_ROOT_UNCOR_RCV))) + return 0; + + err_src = readl(aer_base + PCI_ERR_ROOT_ERR_SRC); + + if (root_status & PCI_ERR_ROOT_UNCOR_RCV) + return FIELD_GET(GENMASK(31, 16), err_src); + + if (root_status & PCI_ERR_ROOT_COR_RCV) + return FIELD_GET(GENMASK(15, 0), err_src); + + return 0; +} + +static bool cxl_rch_handled_error_by_rdpas (struct pci_dev *rcec, + struct aer_err_info *info) +{ + u8 bus, devfn; + u64 dsp_addr; + u16 segment; + u16 src_id; + int rc;drivers/pci/pcie/aer_cxl_rch.c static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data) · line 210
Quoted 10 lines
drivers/pci/pcie/aer_cxl_rch.c +10
-0+ + rc = cxl_get_rdpas_by_rcec(rcec, &dsp_addr); + if (rc) + return false; + + src_id = cxl_rch_get_err_src_id(dsp_addr); + if (!src_id) + return false; + + /* Try uncorrectable error source first, then correctable */I am a little bit confused about this comment and
cxl_rch_get_err_src_id(). I thinkaer_err_info->severifyidendifies current ERR is a correctable error or not.
Quoted 5 lines
In
cxl_rch_get_err_src_id()always handlesUNCOR_ERRbeforeCOR_ERR, is it possible that getting anUNCOR_ERRsrc id for aCOR_ERRaer_err_info?E.g.
aer_err_infois created for aCOR_ERR, but the RCH DSP masksPCI_ERR_ROOT_UNCOR_RCVafter theaer_err_infocreated.Maybe I misunderstand something, please correct me if I am wrong.
No. I was blindly just looking at the registers. I need to take into account what came from the aer_err_info.
DJ
Quoted 38 lines
Ming
Quoted 35 lines
drivers/pci/pcie/aer_cxl_rch.c +23
-3+ segment = pci_domain_nr(rcec->bus); + bus = FIELD_GET(GENMASK(15, 8), src_id); + devfn = FIELD_GET(GENMASK(7, 0), src_id); + + struct pci_dev *pdev __free(pci_dev_put) = + pci_get_domain_bus_and_slot(segment, bus, devfn); + if (!pdev) + return false; + + cxl_forward_error(dev, info); + return true; +} + void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info) { /*@@ -141,9 +229,16 @@ void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)* RCH's downstream port. Check and handle them in the CXL.mem * device driver. */ - if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC && - is_aer_internal_error(info)) - pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info); + if (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC) + return; + + if (!is_aer_internal_error(info)) + return; + + if (cxl_rch_handled_error_by_rdpas(dev, info)) + return; + + pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info); } static int handles_cxl_error_iter(struct pci_dev *dev, void *data)