cxl/pci: Add support for RCH RAS error handling
Subsystems (3)
v5 cover cxl/pci: Add support for RCH RAS error handling
Patches #1 to #18 are a rework of the Component Register setup. This is needed to share multiple CXL capabilities (HDM and RAS) for the same component, also there can be different components implementing the same capability, finally RCH mode should be supported too. The general approach to solve this is to:
* Unify code for components and capabilities in VH and RCH modes.
* Early setup of the Component Register base address.
* Create and store the register mappings to later use it for mapping the capability I/O ranges.
Patches #19 to #26 enable CXL RCH error handling. These are needed because RCH downstream port protocol error handling is implemented uniquely and not currently supported. These patches address the following:
* Discovery and mapping of RCH downstream port AER registers.
* AER portdrv changes to support CXL RCH protocol errors.
* Interrupt setup specific to RCH mode: enabling RCEC internal errors and disabling root port interrupts.
Changes in V5:
- Split 'cxl/rch: Prepare for logging RCH downstream port protocol errors' patch into 2 patches.
- Added:
cxl/core/regs: Rename phys_addr in cxl_map_component_regs() cxl/mem: Prepare for early RCH dport component register setup
- Correct comments CXL3.0 to CXL 3.0.
- changed cxl_port_get_comp_map() to static.
Changes in V4:
- Made port RAS register discovery common and called from
__devm_cxl_add_dport().
- Changed RCH AER register discovery to be called from
__devm_cxl_add_dport().
- Changed RAS and RCH AER register mapping to be called from
__devm_cxl_add_dport(). - Changed component register mapping to support all CXL component mapping, cxl_map_component_regs().
- Added cxl_regs to 'struct cxl_dport' for providing RCH downstream port mapped registers USED in error handler.
- PCI/AER:
- Improved description of PCIEAER_CXL option in Kconfig.
- Renamed function to pci_aer_unmask_internal_errors(), added pcie_aer_is_native() check.
- Improved comments and added spec refs.
- Renamed functions to cxl_rch_handle_error*().
- Modified cxl_rch_handle_error_iter() to only call the handler callbacks, this also simplifies refcounting of the pdev.
- Refactored handle_error_source(), created pci_aer_handle_error().
- Changed printk messages to pci_*() variants.
- Added check for pcie_aer_is_native() to the RCEC.
- Introduced function cxl_rch_enable_rcec().
- Updated patch description ("PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler").
Changes in V3:
- Correct base commit in cover sheet.
- Change hardcoded return 0 to NULL in regs.c.
- Remove calls to pci_disable_pcie_error_reporting(pdev) and pci_enable_pcie_error_reporting(pdev) in mem.c;
- Move RCEC interrupt unmask to PCIe port AER driver's probe.
- Fixes missing PCIEAER and PCIEPORTBUS config option error.
- Rename cxl_rcrb_setup() to cxl_setup_rcrb() in mem.c.
- Update cper_mem_err_unpack() patch subject and description.
Changes in V2:
- Refactor RCH initialization into cxl_mem driver.
- Includes RCH RAS and AER register discovery and mapping.
- Add RCEC protocol error interrupt forwarding to CXL endpoint handler.
- Change AER and RAS logging to use existing trace routines.
- Enable RCEC AER internal errors.
Dan Williams (1):
cxl/rch: Prepare for caching the MMIO mapped PCIe AER capability
Robert Richter (18):
cxl/acpi: Probe RCRB later during RCH downstream port creation
cxl: Rename member @dport of struct cxl_dport to @dev cxl/core/regs: Rename phys_addr in cxl_map_component_regs() cxl/core/regs: Add @dev to cxl_register_map
cxl/acpi: Moving add_host_bridge_uport() around
cxl/acpi: Directly bind the CEDT detected CHBCR to the Host Bridge's
port
cxl/regs: Remove early capability checks in Component Register setup cxl/mem: Prepare for early RCH dport component register setup cxl/pci: Early setup RCH dport component registers from RCRB cxl/port: Store the port's Component Register mappings in struct
cxl_port
cxl/port: Store the downstream port's Component Register mappings in
struct cxl_dport
cxl/pci: Store the endpoint's Component Register mappings in struct
cxl_dev_state
cxl/hdm: Use stored Component Register mappings to map HDM decoder
capability
cxl/port: Remove Component Register base address from struct cxl_port cxl/port: Remove Component Register base address from struct cxl_dport cxl/pci: Remove Component Register base address from struct
cxl_dev_state
PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem
dev handler
PCI/AER: Unmask RCEC internal errors to enable RCH downstream port
error handling
Terry Bowman (7):
cxl/pci: Refactor component register discovery for reuse cxl/pci: Add RCH downstream port AER register discovery PCI/AER: Refactor cper_print_aer() for use by CXL driver module cxl/pci: Update CXL error logging to use RAS register address cxl/pci: Map RCH downstream AER registers for logging protocol errors cxl/pci: Disable root port interrupts in RCH mode
cxl/pci: Add RCH downstream port error logging
base-commit: a70fc4ed20a6118837b0aecbbf789074935f473b
drivers/cxl/acpi.c | 191 +++++++++++++++++++--------------- drivers/cxl/core/hdm.c | 59 +++++------ drivers/cxl/core/pci.c | 140 ++++++++++++++++++++++--- drivers/cxl/core/port.c | 163 +++++++++++++++++++++++++---- drivers/cxl/core/region.c | 4 +- drivers/cxl/core/regs.c | 154 ++++++++++++++++++++++++--- drivers/cxl/cxl.h | 56 ++++++---- drivers/cxl/cxlmem.h | 5 +- drivers/cxl/mem.c | 16 +-- drivers/cxl/pci.c | 109 +++++++------------ drivers/cxl/port.c | 5 +- drivers/pci/pcie/Kconfig | 12 +++ drivers/pci/pcie/aer.c | 162 ++++++++++++++++++++++++++-- include/linux/aer.h | 2 +- tools/testing/cxl/Kbuild | 2 +- tools/testing/cxl/test/cxl.c | 10 +- tools/testing/cxl/test/mock.c | 12 +-- tools/testing/cxl/test/mock.h | 7 +- 18 files changed, 822 insertions(+), 287 deletions(-)
Signature · 1 line
2.34.1
PATCH v5 1/26 cxl/acpi: Probe RCRB later during RCH downstream port creation
From: Robert Richter <[email protected]>
The RCRB is extracted already during ACPI CEDT table parsing while the data of this is needed not earlier than dport creation. This implementation comes with drawbacks: During ACPI table scan there is already MMIO access including mapping and unmapping, but only ACPI data should be collected here. The collected data must be transferred through a couple of interfaces until it is finally consumed when creating the dport. This causes complex data structures and function interfaces. Additionally, RCRB parsing will be extended to also extract AER data, it would be much easier do this at a later point during port and dport creation when the data structures are available to hold that data.
To simplify all that, probe the RCRB at a later point during RCH downstream port creation. Change ACPI table parser to only extract the base address of either the component registers or the RCRB. Parse and extract the RCRB in devm_cxl_add_rch_dport().
This is in preparation to centralize all RCRB scanning.
--- drivers/cxl/acpi.c | 52 ++++++++++++++++------------------------- drivers/cxl/core/port.c | 21 +++++++++++++---- drivers/cxl/cxl.h | 1 - 3 files changed, 36 insertions(+), 38 deletions(-)
drivers/cxl/acpi.c +20 -32
@@ -373,20 +373,18 @@ static int add_host_bridge_uport(struct device *match, void *arg) }struct cxl_chbs_context { -struct device *dev;unsigned long long uid; -resource_size_t rcrb; -resource_size_t chbcr; +resource_size_t base;u32 cxl_version; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg,const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;
@@ -395,23 +393,16 @@ static int cxl_get_chbcr(union acpi_subtable_headers *header, void *arg,return 0 ; ctx->cxl_version = chbs->cxl_version; - ctx->rcrb =CXL_RESOURCE_NONE ; - ctx->chbcr =CXL_RESOURCE_NONE ; + ctx->base =CXL_RESOURCE_NONE ;if (!chbs->base)return 0 ; -if (chbs->cxl_version !=ACPI_CEDT_CHBS_VERSION_CXL11 ) { - ctx->chbcr = chbs->base; +if (chbs->cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 && + chbs->length !=CXL_RCRB_SIZE )return 0 ; - } -if (chbs->length !=CXL_RCRB_SIZE ) -return 0 ; - - ctx->rcrb = chbs->base; - ctx->chbcr =cxl_rcrb_to_component (ctx->dev, chbs->base, -CXL_RCRB_DOWNSTREAM ); + ctx->base = chbs->base;return 0 ; }
@@ -440,36 +431,33 @@ static int add_host_bridge_dport(struct device *match, void *arg)dev_dbg (match,"UID found: %lld\n" , uid); ctx = (struct cxl_chbs_context ) { - .dev = match, .uid = uid, }; -acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbcr, &ctx); +acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs, &ctx); -if (!ctx.chbcr) { +if (!ctx.base) {dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , uid);return 0 ; } -if (ctx.rcrb !=CXL_RESOURCE_NONE ) -dev_dbg (match,"RCRB found for UID %lld: %pa\n" , uid, &ctx.rcrb); - -if (ctx.chbcr ==CXL_RESOURCE_NONE ) { -dev_warn (match,"CHBCR invalid for Host Bridge (UID %lld)\n" , +if (ctx.base ==CXL_RESOURCE_NONE ) { +dev_warn (match,"CHBS invalid for Host Bridge (UID %lld)\n" , uid);return 0 ; } -dev_dbg (match,"CHBCR found: %pa\n" , &ctx.chbcr); - pci_root =acpi_pci_find_root (hb->handle); bridge = pci_root->bus->bridge; -if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) - dport =devm_cxl_add_rch_dport (root_port, bridge, uid, - ctx.chbcr, ctx.rcrb); -else - dport =devm_cxl_add_dport (root_port, bridge, uid, - ctx.chbcr); + +if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) { +dev_dbg (match,"RCRB found for UID %lld: %pa\n" , uid, &ctx.base); + dport =devm_cxl_add_rch_dport (root_port, bridge, uid, ctx.base); + }else { +dev_dbg (match,"CHBCR found for UID %lld: %pa\n" , uid, &ctx.base); + dport =devm_cxl_add_dport (root_port, bridge, uid, ctx.base); + } +if (IS_ERR (dport))return PTR_ERR (dport);
drivers/cxl/core/port.c +16 -5
@@ -938,12 +938,25 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (!dport)return ERR_PTR (-ENOMEM ); +if (rcrb !=CXL_RESOURCE_NONE ) { + component_reg_phys =cxl_rcrb_to_component (dport_dev, + rcrb,CXL_RCRB_DOWNSTREAM ); +if (component_reg_phys ==CXL_RESOURCE_NONE ) { +dev_warn (dport_dev,"Invalid Component Registers in RCRB" ); +return ERR_PTR (-ENXIO ); + } + + dport->rch = true; + } + +if (component_reg_phys !=CXL_RESOURCE_NONE ) +dev_dbg (dport_dev,"Component Registers found for dport: %pa\n" , + &component_reg_phys); + dport->dport = dport_dev; dport->port_id = port_id; dport->component_reg_phys = component_reg_phys; dport->port = port; -if (rcrb !=CXL_RESOURCE_NONE ) - dport->rch = true; dport->rcrb = rcrb;cond_cxl_root_lock (port);
@@ -1004,14 +1017,12 @@ EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport, CXL);* @port: the cxl_port that references this dport * @dport_dev: firmware or PCI device representing the dport * @port_id: identifier for this dport in a decoder's target list -* @component_reg_phys: optional location of CXL component registers * @rcrb: mandatory location of a Root Complex Register Block * * See CXL 3.0 9.11.8 CXL Devices Attached to an RCH */ struct cxl_dport *devm_cxl_add_rch_dport (struct cxl_port *port,struct device *dport_dev,int port_id, -resource_size_t component_reg_phys,resource_size_t rcrb) {struct cxl_dport *dport;
@@ -1022,7 +1033,7 @@ struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port, } dport =__devm_cxl_add_dport (port, dport_dev, port_id, - component_reg_phys, rcrb); +CXL_RESOURCE_NONE , rcrb);if (IS_ERR (dport)) {dev_dbg (dport_dev,"failed to add RCH dport to %s: %ld\n" ,dev_name (&port->dev),PTR_ERR (dport));
drivers/cxl/cxl.h +0 -1
@@ -671,7 +671,6 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port, resource_size_t component_reg_phys);struct cxl_dport *devm_cxl_add_rch_dport (struct cxl_port *port,struct device *dport_dev,int port_id, -resource_size_t component_reg_phys,resource_size_t rcrb);struct cxl_decoder *to_cxl_decoder (struct device *dev);
Signature · 1 line
2.34.1
RE: [PATCH v5 01/26] cxl/acpi: Probe RCRB later during RCH downstream port creation
Quoted 14 lines · Terry Bowman
From: Robert Richter <[email protected]>
The RCRB is extracted already during ACPI CEDT table parsing while the data of this is needed not earlier than dport creation. This implementation comes with drawbacks: During ACPI table scan there is already MMIO access including mapping and unmapping, but only ACPI data should be collected here. The collected data must be transferred through a couple of interfaces until it is finally consumed when creating the dport. This causes complex data structures and function interfaces. Additionally, RCRB parsing will be extended to also extract AER data, it would be much easier do this at a later point during port and dport creation when the data structures are available to hold that data.
Quoted 6 lines
To simplify all that, probe the RCRB at a later point during RCH downstream port creation. Change ACPI table parser to only extract the base address of either the component registers or the RCRB. Parse and extract the RCRB in
devm_cxl_add_rch_dport().This is in preparation to centralize all RCRB scanning.
I really like the approach of this patch, the cleanups just make sense, the changelog is great...
...just the small matter of massive cxl_test breakages. Given that QEMU only supports CXL VH topologies I would like to keep cxl_test emulation of RCH topologies to regression test ongoing core reworks against the RCH case.
The problem with cxl_test though is that due to how it uses linker tricks (--wrap=) to inject mock topology data, it cannot support a symbol that is exported by the core *and* consumed by the core. This patch moved cxl_rcrb_to_component() from being called by cxl_acpi and cxl_mem to being called internally by cxl_core and cxl_mem.
The fix path I chose is to add a new cxl_rcd_component_reg_phys() helper for retrieving the upstream-port register block out of the RCRB, and add a mock devm_cxl_add_rch_dport() that fakes the discovery of an RCH.
Below is the result for patch1. My plan is to go through the rest of the series and ensure cxl_test keeps working. Perhaps you can fold in the cxl_test fixups as I send them to be incorporated in your v6?
To be clear, I feel this is my maintenance burden to bear, I don't fault you for not using cxl_test, but I am invested in keeping it operational.
-- >8 --
drivers/cxl/acpi.c +3 -0
@@ -372,7 +372,9 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } +/* Note, @dev is used by mock_acpi_table_parse_cedt() */ struct cxl_chbs_context { +struct device *dev;unsigned long long uid;resource_size_t base;u32 cxl_version;
@@ -431,6 +433,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)dev_dbg (match,"UID found: %lld\n" , uid); ctx = (struct cxl_chbs_context ) { + .dev = match, .uid = uid, };acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs, &ctx);
drivers/cxl/core/core.h +8 -0
@@ -63,6 +63,14 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size);int cxl_dpa_free (struct cxl_endpoint_decoder *cxled);resource_size_t cxl_dpa_size (struct cxl_endpoint_decoder *cxled);resource_size_t cxl_dpa_resource_start (struct cxl_endpoint_decoder *cxled); + +enum cxl_rcrb { +CXL_RCRB_DOWNSTREAM , +CXL_RCRB_UPSTREAM , +}; +resource_size_t __rcrb_to_component (struct device *dev,resource_size_t rcrb, +enum cxl_rcrb which); +extern struct rw_semaphore cxl_dpa_rwsem;int cxl_memdev_init (void );
drivers/cxl/core/port.c +2 -2
@@ -939,8 +939,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,return ERR_PTR (-ENOMEM );if (rcrb !=CXL_RESOURCE_NONE ) { - component_reg_phys =cxl_rcrb_to_component (dport_dev, - rcrb,CXL_RCRB_DOWNSTREAM ); + component_reg_phys =__rcrb_to_component (dport_dev, rcrb, +CXL_RCRB_DOWNSTREAM );if (component_reg_phys ==CXL_RESOURCE_NONE ) {dev_warn (dport_dev,"Invalid Component Registers in RCRB" );return ERR_PTR (-ENXIO );
drivers/cxl/core/regs.c +11 -4
@@ -332,9 +332,8 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL ); -resource_size_t cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) +resource_size_t __rcrb_to_component (struct device *dev,resource_size_t rcrb, +enum cxl_rcrb which) {resource_size_t component_reg_phys;void __iomem *addr;
@@ -395,4 +394,12 @@ resource_size_t cxl_rcrb_to_component(struct device *dev,return component_reg_phys; } -EXPORT_SYMBOL_NS_GPL (cxl_rcrb_to_component,CXL ); + +resource_size_t cxl_rcd_component_reg_phys (struct device *dev, +struct cxl_dport *dport) +{ +if (!dport->rch) +return CXL_RESOURCE_NONE ; +return __rcrb_to_component (dev, dport->rcrb,CXL_RCRB_UPSTREAM ); +} +EXPORT_SYMBOL_NS_GPL (cxl_rcd_component_reg_phys,CXL );
drivers/cxl/cxl.h +3 -8
@@ -262,14 +262,9 @@ int cxl_map_device_regs(struct device *dev, struct cxl_device_regs *regs,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); - -enum cxl_rcrb { -CXL_RCRB_DOWNSTREAM , -CXL_RCRB_UPSTREAM , -}; -resource_size_t cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which); +struct cxl_dport ; +resource_size_t cxl_rcd_component_reg_phys (struct device *dev, +struct cxl_dport *dport);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20
drivers/cxl/mem.c +2 -2
@@ -72,8 +72,8 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,* typical register locator mechanism. */ if (parent_dport->rch && cxlds->component_reg_phys ==CXL_RESOURCE_NONE ) - component_reg_phys =cxl_rcrb_to_component ( - &cxlmd->dev, parent_dport->rcrb,CXL_RCRB_UPSTREAM ); + component_reg_phys = +cxl_rcd_component_reg_phys (&cxlmd->dev, parent_dport);else component_reg_phys = cxlds->component_reg_phys; endpoint = devm_cxl_add_port(host, &cxlmd->dev, component_reg_phys,
tools/testing/cxl/Kbuild +2 -1
@@ -12,7 +12,8 @@ ldflags-y += --wrap=devm_cxl_enumerate_decoders ldflags-y += --wrap=cxl_await_media_ready ldflags-y += --wrap=cxl_hdm_decode_init ldflags-y += --wrap=cxl_dvsec_rr_decode -ldflags-y += --wrap=cxl_rcrb_to_component +ldflags-y += --wrap=devm_cxl_add_rch_dport +ldflags-y += --wrap=cxl_rcd_component_reg_phys DRIVERS := ../../../drivers CXL_SRC := $(DRIVERS)/cxl
tools/testing/cxl/test/cxl.c +0 -10
@@ -971,15 +971,6 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)return 0 ; } -resource_size_t mock_cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) -{ -dev_dbg (dev,"rcrb: %pa which: %d\n" , &rcrb, which); - -return (resource_size_t ) which +1 ; -} -static struct cxl_mock_ops cxl_mock_ops = { .is_mock_adev = is_mock_adev, .is_mock_bridge = is_mock_bridge,
@@ -988,7 +979,6 @@ static struct cxl_mock_ops cxl_mock_ops = { .is_mock_dev = is_mock_dev, .acpi_table_parse_cedt = mock_acpi_table_parse_cedt, .acpi_evaluate_integer = mock_acpi_evaluate_integer, - .cxl_rcrb_to_component = mock_cxl_rcrb_to_component, .acpi_pci_find_root = mock_acpi_pci_find_root, .devm_cxl_port_enumerate_dports = mock_cxl_port_enumerate_dports, .devm_cxl_setup_hdm = mock_cxl_setup_hdm,
tools/testing/cxl/test/mock.c +27 -7
@@ -259,24 +259,44 @@ int __wrap_cxl_dvsec_rr_decode(struct device *dev, int dvsec, }EXPORT_SYMBOL_NS_GPL (__wrap_cxl_dvsec_rr_decode,CXL ); -resource_size_t __wrap_cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) +struct cxl_dport *__wrap_devm_cxl_add_rch_dport (struct cxl_port *port, +struct device *dport_dev, +int port_id, +resource_size_t rcrb) +{ +int index; +struct cxl_dport *dport; +struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); + +if (ops && ops->is_mock_port (dport_dev)) { + dport =devm_cxl_add_dport (port, dport_dev, port_id, +CXL_RESOURCE_NONE ); +if (!IS_ERR (dport)) + dport->rch = true; + }else + dport =devm_cxl_add_rch_dport (port, dport_dev, port_id, rcrb); +put_cxl_mock_ops (index); + +return dport; +} +EXPORT_SYMBOL_NS_GPL (__wrap_devm_cxl_add_rch_dport,CXL ); + +resource_size_t __wrap_cxl_rcd_component_reg_phys (struct device *dev, +struct cxl_dport *dport) {int index;resource_size_t component_reg_phys;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index);if (ops && ops->is_mock_port (dev)) - component_reg_phys = - ops->cxl_rcrb_to_component (dev, rcrb, which); + component_reg_phys =CXL_RESOURCE_NONE ;else - component_reg_phys =cxl_rcrb_to_component (dev, rcrb, which); + component_reg_phys =cxl_rcd_component_reg_phys (dev, dport);put_cxl_mock_ops (index);return component_reg_phys; } -EXPORT_SYMBOL_NS_GPL (__wrap_cxl_rcrb_to_component,CXL ); +EXPORT_SYMBOL_NS_GPL (__wrap_cxl_rcd_component_reg_phys,CXL );MODULE_LICENSE ("GPL v2" );MODULE_IMPORT_NS (ACPI );
tools/testing/cxl/test/mock.h +0 -3
@@ -15,9 +15,6 @@ struct cxl_mock_ops {acpi_string pathname, struct acpi_object_list *arguments, unsigned long long *data); -resource_size_t (*cxl_rcrb_to_component)(struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which);struct acpi_pci_root *(*acpi_pci_find_root)(acpi_handle handle);bool (*is_mock_bus)(struct pci_bus *bus);bool (*is_mock_port)(struct device *dev);
RE: [PATCH v5 01/26] cxl/acpi: Probe RCRB later during RCH downstream port creation
Quoted 24 lines · Dan Williams
Quoted 14 lines · Terry Bowman
From: Robert Richter <[email protected]>
The RCRB is extracted already during ACPI CEDT table parsing while the data of this is needed not earlier than dport creation. This implementation comes with drawbacks: During ACPI table scan there is already MMIO access including mapping and unmapping, but only ACPI data should be collected here. The collected data must be transferred through a couple of interfaces until it is finally consumed when creating the dport. This causes complex data structures and function interfaces. Additionally, RCRB parsing will be extended to also extract AER data, it would be much easier do this at a later point during port and dport creation when the data structures are available to hold that data.
Quoted 6 lines
To simplify all that, probe the RCRB at a later point during RCH downstream port creation. Change ACPI table parser to only extract the base address of either the component registers or the RCRB. Parse and extract the RCRB in
devm_cxl_add_rch_dport().This is in preparation to centralize all RCRB scanning.
I really like the approach of this patch, the cleanups just make sense, the changelog is great...
[..]
Quoted 2 lines
To be clear, I feel this is my maintenance burden to bear, I don't fault you for not using
cxl_test, but I am invested in keeping it operational.
On to patch2...
$ stg push
Pushing patch "cxl-rch-prepare-for-caching" ... done (conflict)
Error: 8 merge conflict(s)
CONFLICT (content): Merge conflict in tools/testing/cxl/test/mock.h
CONFLICT (content): Merge conflict in tools/testing/cxl/test/mock.c
CONFLICT (content): Merge conflict in tools/testing/cxl/test/cxl.c
CONFLICT (content): Merge conflict in tools/testing/cxl/Kbuild
CONFLICT (content): Merge conflict in drivers/cxl/mem.c
CONFLICT (content): Merge conflict in drivers/cxl/cxl.h
CONFLICT (content): Merge conflict in drivers/cxl/core/regs.c
CONFLICT (content): Merge conflict in drivers/cxl/core/port.c
Now at patch "cxl-rch-prepare-for-caching"...oh, looks like you are trying to keep cxl_test going. I had stopped at patch1 because that one already raised a circular dependency build error:
depmod: ERROR: Cycle detected: cxl_mock -> cxl_core -> cxl_mock
Let me see what's going on here...
PATCH v5 2/26 cxl/rch: Prepare for caching the MMIO mapped PCIe AER capability
In reply to Terry BowmanFrom: Dan Williams <[email protected]>
Prepare cxl_probe_rcrb() for retrieving more than just the component register block. The RCH AER handling code wants to get back to the AER capability that happens to be MMIO mapped rather then configuration cycles.
Move RCRB specific downstream port data, like the RCRB base and the AER capability offset, into its own data structure ('struct cxl_rcrb_info') for cxl_probe_rcrb() to fill. Extend 'struct cxl_dport' to include a 'struct cxl_rcrb_info' attribute.
This centralizes all RCRB scanning in one routine.
--- drivers/cxl/core/port.c | 6 +++--- drivers/cxl/core/regs.c | 10 ++++++---- drivers/cxl/cxl.h | 19 ++++++++++++------- drivers/cxl/mem.c | 5 +++-- tools/testing/cxl/Kbuild | 2 +- tools/testing/cxl/test/cxl.c | 10 ++++++---- tools/testing/cxl/test/mock.c | 12 ++++++------ tools/testing/cxl/test/mock.h | 7 ++++--- 8 files changed, 41 insertions(+), 30 deletions(-)
drivers/cxl/core/port.c +3 -3
@@ -939,8 +939,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,return ERR_PTR (-ENOMEM );if (rcrb !=CXL_RESOURCE_NONE ) { - component_reg_phys =cxl_rcrb_to_component (dport_dev, - rcrb,CXL_RCRB_DOWNSTREAM ); + component_reg_phys = +cxl_probe_rcrb (dport_dev, rcrb, &dport->rcrb, +CXL_RCRB_DOWNSTREAM );if (component_reg_phys ==CXL_RESOURCE_NONE ) {dev_warn (dport_dev,"Invalid Component Registers in RCRB" );return ERR_PTR (-ENXIO );
@@ -957,7 +958,6 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, dport->port_id = port_id; dport->component_reg_phys = component_reg_phys; dport->port = port; - dport->rcrb = rcrb;cond_cxl_root_lock (port); rc =add_dport (port, dport);
drivers/cxl/core/regs.c +6 -4
@@ -332,9 +332,8 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL ); -resource_size_t cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) +resource_size_t cxl_probe_rcrb (struct device *dev,resource_size_t rcrb, +struct cxl_rcrb_info *ri,enum cxl_rcrb which) {resource_size_t component_reg_phys;void __iomem *addr;
@@ -344,6 +343,8 @@ resource_size_t cxl_rcrb_to_component(struct device *dev,if (which ==CXL_RCRB_UPSTREAM ) rcrb +=SZ_4K ; +else if (ri) + ri->base = rcrb;/* * RCRB's BAR[0..1] point to component block containing CXL
@@ -364,6 +365,7 @@ resource_size_t cxl_rcrb_to_component(struct device *dev, cmd =readw (addr +PCI_COMMAND ); bar0 =readl (addr +PCI_BASE_ADDRESS_0 ); bar1 =readl (addr +PCI_BASE_ADDRESS_1 ); +iounmap (addr);release_mem_region (rcrb,SZ_4K );
@@ -395,4 +397,4 @@ resource_size_t cxl_rcrb_to_component(struct device *dev,return component_reg_phys; } -EXPORT_SYMBOL_NS_GPL (cxl_rcrb_to_component,CXL ); +EXPORT_SYMBOL_NS_GPL (cxl_probe_rcrb,CXL );
drivers/cxl/cxl.h +12 -7
@@ -267,9 +267,9 @@ enum cxl_rcrb {CXL_RCRB_DOWNSTREAM , CXL_RCRB_UPSTREAM, }; -resource_size_t cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which); +struct cxl_rcrb_info ; +resource_size_t cxl_probe_rcrb (struct device *dev,resource_size_t rcrb, +struct cxl_rcrb_info *ri,enum cxl_rcrb which);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20
@@ -587,22 +587,27 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)return xa_load (&port->dports, (unsigned long )dport_dev); } +struct cxl_rcrb_info { +resource_size_t base; +u16 aer_cap; +}; +/** * struct cxl_dport - CXL downstream port * @dport: PCI bridge or firmware device representing the downstream link +* @port: reference to cxl_port that contains this downstream port * @port_id: unique hardware identifier for dport in decoder target list * @component_reg_phys: downstream port component registers -* @rcrb: base address for the Root Complex Register Block * @rch: Indicate whether this dport was enumerated in RCH or VH mode -* @port: reference to cxl_port that contains this downstream port +* @rcrb: Data about the Root Complex Register Block layout */ struct cxl_dport {struct device *dport; +struct cxl_port *port;int port_id;resource_size_t component_reg_phys; -resource_size_t rcrb;bool rch; -struct cxl_port *port; +struct cxl_rcrb_info rcrb; };/**
drivers/cxl/mem.c +3 -2
@@ -72,8 +72,9 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,* typical register locator mechanism. */ if (parent_dport->rch && cxlds->component_reg_phys ==CXL_RESOURCE_NONE ) - component_reg_phys =cxl_rcrb_to_component ( - &cxlmd->dev, parent_dport->rcrb,CXL_RCRB_UPSTREAM ); + component_reg_phys = +cxl_probe_rcrb (&cxlmd->dev, parent_dport->rcrb.base, +NULL ,CXL_RCRB_UPSTREAM );else component_reg_phys = cxlds->component_reg_phys; endpoint = devm_cxl_add_port(host, &cxlmd->dev, component_reg_phys,
tools/testing/cxl/Kbuild +1 -1
@@ -12,7 +12,7 @@ ldflags-y += --wrap=devm_cxl_enumerate_decoders ldflags-y += --wrap=cxl_await_media_ready ldflags-y += --wrap=cxl_hdm_decode_init ldflags-y += --wrap=cxl_dvsec_rr_decode -ldflags-y += --wrap=cxl_rcrb_to_component +ldflags-y += --wrap=cxl_probe_rcrb DRIVERS := ../../../drivers CXL_SRC := $(DRIVERS)/cxl
tools/testing/cxl/test/cxl.c +6 -4
@@ -971,12 +971,14 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)return 0 ; } -resource_size_t mock_cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) +resource_size_t mock_cxl_probe_rcrb (struct device *dev,resource_size_t rcrb, +struct cxl_rcrb_info *ri,enum cxl_rcrb which) {dev_dbg (dev,"rcrb: %pa which: %d\n" , &rcrb, which); +if (which ==CXL_RCRB_DOWNSTREAM ) + ri->base = rcrb; +return (resource_size_t ) which +1 ; }
@@ -988,7 +990,7 @@ static struct cxl_mock_ops cxl_mock_ops = { .is_mock_dev = is_mock_dev, .acpi_table_parse_cedt = mock_acpi_table_parse_cedt, .acpi_evaluate_integer = mock_acpi_evaluate_integer, - .cxl_rcrb_to_component = mock_cxl_rcrb_to_component, + .cxl_probe_rcrb = mock_cxl_probe_rcrb, .acpi_pci_find_root = mock_acpi_pci_find_root, .devm_cxl_port_enumerate_dports = mock_cxl_port_enumerate_dports, .devm_cxl_setup_hdm = mock_cxl_setup_hdm,
tools/testing/cxl/test/mock.c +6 -6
@@ -259,9 +259,9 @@ int __wrap_cxl_dvsec_rr_decode(struct device *dev, int dvsec, }EXPORT_SYMBOL_NS_GPL (__wrap_cxl_dvsec_rr_decode,CXL ); -resource_size_t __wrap_cxl_rcrb_to_component (struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which) +resource_size_t __wrap_cxl_probe_rcrb (struct device *dev,resource_size_t rcrb, +struct cxl_rcrb_info *ri, +enum cxl_rcrb which) {int index;resource_size_t component_reg_phys;
@@ -269,14 +269,14 @@ resource_size_t __wrap_cxl_rcrb_to_component(struct device *dev,if (ops && ops->is_mock_port (dev)) component_reg_phys = - ops->cxl_rcrb_to_component (dev, rcrb, which); + ops->cxl_probe_rcrb (dev, rcrb, ri, which);else - component_reg_phys =cxl_rcrb_to_component (dev, rcrb, which); + component_reg_phys =cxl_probe_rcrb (dev, rcrb, ri, which);put_cxl_mock_ops (index);return component_reg_phys; } -EXPORT_SYMBOL_NS_GPL (__wrap_cxl_rcrb_to_component,CXL ); +EXPORT_SYMBOL_NS_GPL (__wrap_cxl_probe_rcrb,CXL );MODULE_LICENSE ("GPL v2" );MODULE_IMPORT_NS (ACPI );
tools/testing/cxl/test/mock.h +4 -3
@@ -15,9 +15,10 @@ struct cxl_mock_ops {acpi_string pathname, struct acpi_object_list *arguments, unsigned long long *data); -resource_size_t (*cxl_rcrb_to_component)(struct device *dev, -resource_size_t rcrb, -enum cxl_rcrb which); +resource_size_t (*cxl_probe_rcrb)(struct device *dev, +resource_size_t rcrb, +struct cxl_rcrb_info *ri, +enum cxl_rcrb which);struct acpi_pci_root *(*acpi_pci_find_root)(acpi_handle handle);bool (*is_mock_bus)(struct pci_bus *bus);bool (*is_mock_port)(struct device *dev);
Signature · 1 line
2.34.1
RE: [PATCH v5 02/26] cxl/rch: Prepare for caching the MMIO mapped PCIe AER capability
Quoted 7 lines · Terry Bowman
From: Dan Williams <[email protected]>
Prepare
cxl_probe_rcrb()for retrieving more than just the component register block. The RCH AER handling code wants to get back to the AER capability that happens to be MMIO mapped rather then configuration cycles.
Quoted 6 lines
Move RCRB specific downstream port data, like the RCRB base and the AER capability offset, into its own data structure ('struct
cxl_rcrb_info') forcxl_probe_rcrb()to fill. Extend 'structcxl_dport' to include a 'structcxl_rcrb_info' attribute.This centralizes all RCRB scanning in one routine.
With the addition of cxl_rcd_component_reg_phys() and __rcrb_to_component() in the previous patch, this one becomes cleaner to implement. Given the collisions it needed to be re-written. The following is passing cxl_test:
-- >8 --
drivers/cxl/core/core.h +3 -1
@@ -68,7 +68,9 @@ enum cxl_rcrb {CXL_RCRB_DOWNSTREAM , CXL_RCRB_UPSTREAM, }; -resource_size_t __rcrb_to_component (struct device *dev,resource_size_t rcrb, +struct cxl_rcrb_info ; +resource_size_t __rcrb_to_component (struct device *dev, +struct cxl_rcrb_info *ri,enum cxl_rcrb which);extern struct rw_semaphore cxl_dpa_rwsem;
drivers/cxl/core/port.c +2 -2
@@ -939,7 +939,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,return ERR_PTR (-ENOMEM );if (rcrb !=CXL_RESOURCE_NONE ) { - component_reg_phys =__rcrb_to_component (dport_dev, rcrb, + dport->rcrb.base = rcrb; + component_reg_phys =__rcrb_to_component (dport_dev, &dport->rcrb,CXL_RCRB_DOWNSTREAM );if (component_reg_phys ==CXL_RESOURCE_NONE ) {dev_warn (dport_dev,"Invalid Component Registers in RCRB" );
@@ -957,7 +958,6 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, dport->port_id = port_id; dport->component_reg_phys = component_reg_phys; dport->port = port; - dport->rcrb = rcrb;cond_cxl_root_lock (port); rc =add_dport (port, dport);
drivers/cxl/core/regs.c +3 -2
@@ -332,10 +332,11 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL ); -resource_size_t __rcrb_to_component (struct device *dev,resource_size_t rcrb, +resource_size_t __rcrb_to_component (struct device *dev,struct cxl_rcrb_info *ri,enum cxl_rcrb which) {resource_size_t component_reg_phys; +resource_size_t rcrb = ri->base;void __iomem *addr;u32 bar0, bar1;u16 cmd;
@@ -400,6 +401,6 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev, {if (!dport->rch)return CXL_RESOURCE_NONE ; -return __rcrb_to_component (dev, dport->rcrb,CXL_RCRB_UPSTREAM ); +return __rcrb_to_component (dev, &dport->rcrb,CXL_RCRB_UPSTREAM ); }EXPORT_SYMBOL_NS_GPL (cxl_rcd_component_reg_phys,CXL );
drivers/cxl/cxl.h +7 -2
@@ -582,12 +582,17 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)return xa_load (&port->dports, (unsigned long )dport_dev); } +struct cxl_rcrb_info { +resource_size_t base; +u16 aer_cap; +}; +/** * struct cxl_dport - CXL downstream port * @dport: PCI bridge or firmware device representing the downstream link * @port_id: unique hardware identifier for dport in decoder target list * @component_reg_phys: downstream port component registers -* @rcrb: base address for the Root Complex Register Block +* @rcrb: Data about the Root Complex Register Block layout * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port */
@@ -595,7 +600,7 @@ struct cxl_dport {struct device *dport;int port_id;resource_size_t component_reg_phys; -resource_size_t rcrb; +struct cxl_rcrb_info rcrb;bool rch;struct cxl_port *port; };
tools/testing/cxl/test/mock.c +3 -1
@@ -271,8 +271,10 @@ struct cxl_dport *__wrap_devm_cxl_add_rch_dport(struct cxl_port *port,if (ops && ops->is_mock_port (dport_dev)) { dport =devm_cxl_add_dport (port, dport_dev, port_id,CXL_RESOURCE_NONE ); -if (!IS_ERR (dport)) +if (!IS_ERR (dport)) { + dport->rcrb.base = rcrb; dport->rch = true; + } }else dport =devm_cxl_add_rch_dport (port, dport_dev, port_id, rcrb);put_cxl_mock_ops (index);
PATCH v5 3/26 cxl: Rename member @dport of struct cxl_dport to @dev
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Reading code like dport->dport does not immediately suggest that this points to the corresponding device structure of the dport. Rename struct member @dport to @dev.
While at it, also rename @new argument of add_dport() to @dport. This better describes the variable as a dport (e.g. new->dport becomes to dport->dev).
--- drivers/cxl/core/port.c | 20 ++++++++++---------- drivers/cxl/core/region.c | 4 ++-- drivers/cxl/cxl.h | 4 ++-- 3 files changed, 14 insertions(+), 14 deletions(-)
drivers/cxl/core/port.c +10 -10
@@ -605,7 +605,7 @@ static int devm_cxl_link_parent_dport(struct device *host,if (!parent_dport)return 0 ; - rc =sysfs_create_link (&port->dev.kobj, &parent_dport->dport->kobj, + rc =sysfs_create_link (&port->dev.kobj, &parent_dport->dev->kobj,"parent_dport" );if (rc)return rc;
@@ -658,7 +658,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,if (iter->host_bridge) port->host_bridge = iter->host_bridge;else if (parent_dport->rch) - port->host_bridge = parent_dport->dport; + port->host_bridge = parent_dport->dev;else port->host_bridge = iter->uport;dev_dbg (uport,"host-bridge: %s\n" ,dev_name (port->host_bridge));
@@ -847,22 +847,22 @@ static struct cxl_dport *find_dport(struct cxl_port *port, int id)return NULL ; } -static int add_dport (struct cxl_port *port,struct cxl_dport *new) +static int add_dport (struct cxl_port *port,struct cxl_dport *dport) {struct cxl_dport *dup;int rc;device_lock_assert (&port->dev); - dup =find_dport (port, new->port_id); + dup =find_dport (port, dport->port_id);if (dup) {dev_err (&port->dev,"unable to add dport%d-%s non-unique port id (%s)\n" , - new->port_id,dev_name (new->dport), -dev_name (dup->dport)); + dport->port_id,dev_name (dport->dev), +dev_name (dup->dev));return -EBUSY ; } - rc =xa_insert (&port->dports, (unsigned long )new->dport, new, + rc =xa_insert (&port->dports, (unsigned long )dport->dev, dport,GFP_KERNEL );if (rc)return rc;
@@ -895,8 +895,8 @@ static void cxl_dport_remove(void *data)struct cxl_dport *dport = data;struct cxl_port *port = dport->port; -xa_erase (&port->dports, (unsigned long ) dport->dport); -put_device (dport->dport); +xa_erase (&port->dports, (unsigned long ) dport->dev); +put_device (dport->dev); }static void cxl_dport_unlink (void *data)
@@ -954,7 +954,7 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,dev_dbg (dport_dev,"Component Registers found for dport: %pa\n" , &component_reg_phys); - dport->dport = dport_dev; + dport->dev = dport_dev; dport->port_id = port_id; dport->component_reg_phys = component_reg_phys; dport->port = port;
drivers/cxl/core/region.c +2 -2
@@ -1162,7 +1162,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,dev_dbg (&cxlr->dev,"%s:%s: %s expected %s at %d\n" ,dev_name (port->uport),dev_name (&port->dev),dev_name (&cxlsd->cxld.dev), -dev_name (ep->dport->dport), +dev_name (ep->dport->dev), cxl_rr->nr_targets_set);return -ENXIO ; }
@@ -1173,7 +1173,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, cxl_rr->nr_targets_set += inc;dev_dbg (&cxlr->dev,"%s:%s target[%d] = %s for %s:%s @ %d\n" ,dev_name (port->uport),dev_name (&port->dev), - cxl_rr->nr_targets_set -1 ,dev_name (ep->dport->dport), + cxl_rr->nr_targets_set -1 ,dev_name (ep->dport->dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos);return 0 ;
drivers/cxl/cxl.h +2 -2
@@ -594,7 +594,7 @@ struct cxl_rcrb_info {/** * struct cxl_dport - CXL downstream port -* @dport: PCI bridge or firmware device representing the downstream link +* @dev: PCI bridge or firmware device representing the downstream link * @port: reference to cxl_port that contains this downstream port * @port_id: unique hardware identifier for dport in decoder target list * @component_reg_phys: downstream port component registers
@@ -602,7 +602,7 @@ struct cxl_rcrb_info {* @rcrb: Data about the Root Complex Register Block layout */ struct cxl_dport { -struct device *dport; +struct device *dev;struct cxl_port *port;int port_id;resource_size_t component_reg_phys;
Signature · 1 line
2.34.1
RE: [PATCH v5 03/26] cxl: Rename member @dport of struct cxl_dport to @dev
Quoted 5 lines · Terry Bowman
From: Robert Richter <[email protected]>
Reading code like
dport->dportdoes not immediately suggest that this points to the corresponding device structure of the dport. Rename struct member @dport to @dev.
This one I don't agree with.
This can switch to ->dport_dev if you like. The reason for ->dport was for symmetry with the ->uport of a 'struct cxl_port'. So if you change this to ->dport_dev then also make the ->uport_dev change for symmetry.
Unlike a 'struct cxl_port' a 'struct cxl_dport' is not a device in its own right which is what I see when I read dport->dev.
Quoted 4 lines
While at it, also rename @new argument of
add_dport()to @dport. This better describes the variable as a dport (e.g.new->dportbecomes todport->dev).
There is already other occurrences of dport_dev as an argument, so I think that works here too.
Re: [PATCH v5 03/26] cxl: Rename member @dport of struct cxl_dport to @dev
Hi Dan,
Quoted 7 lines · Dan Williams
Quoted 5 lines · Terry Bowman
From: Robert Richter <[email protected]>
Reading code like
dport->dportdoes not immediately suggest that this points to the corresponding device structure of the dport. Rename struct member @dport to @dev.
Quoted 9 lines
This one I don't agree with.
This can switch to ->dport_dev if you like. The reason for ->dport was for symmetry with the ->uport of a 'struct
cxl_port'. So if you change this to ->dport_dev then also make the ->uport_dev change for symmetry.Unlike a 'struct
cxl_port' a 'structcxl_dport' is not a device in its own right which is what I see when I readdport->dev.
Ok, I'll change the structure member names and code to use the following:
struct cxl_dport::dev -> struct cxl_dport::dport_dev
struct cxl_port::uport -> struct cxl_port::uport_dev
Regards,
Terry
Quoted 7 lines
Quoted 4 lines
While at it, also rename @new argument of
add_dport()to @dport. This better describes the variable as a dport (e.g.new->dportbecomes todport->dev).There is already other occurrences of
dport_devas an argument, so I think that works here too.
Re: [PATCH v5 03/26] cxl: Rename member @dport of struct cxl_dport to @dev
Quoted 17 lines · Terry Bowman
Hi Dan,
Quoted 13 lines · Dan Williams
Quoted 5 lines · Terry Bowman
From: Robert Richter <[email protected]>
Reading code like
dport->dportdoes not immediately suggest that this points to the corresponding device structure of the dport. Rename struct member @dport to @dev.This one I don't agree with.
This can switch to ->dport_dev if you like. The reason for ->dport was for symmetry with the ->uport of a 'struct
cxl_port'. So if you change this to ->dport_dev then also make the ->uport_dev change for symmetry.
Quoted 8 lines
Quoted 3 lines
Unlike a 'struct
cxl_port' a 'structcxl_dport' is not a device in its own right which is what I see when I readdport->dev.Ok, I'll change the structure member names and code to use the following:
structcxl_dport::dev-> structcxl_dport::dport_devstruct
cxl_port::uport-> structcxl_port::uport_dev
Sounds good, to save duplication of work I already have this that you can just adopt:
-- >8 --
Subject: cxl: Rename 'uport' to 'uport_dev'
From: Dan Williams <[email protected]>
For symmetry with the recent rename of ->dport_dev for a 'struct cxl_dport', add the "_dev" suffix to the ->uport property of a 'struct cxl_port'. These devices represent the downstream-port-device and upstream-port-device respectively in the CXL/PCIe topology.
--- drivers/cxl/core/pci.c | 4 +-- drivers/cxl/core/port.c | 61 ++++++++++++++++++++++------------------- drivers/cxl/core/region.c | 48 +++++++++++++++++--------------- drivers/cxl/cxl.h | 13 +++++---- drivers/cxl/cxlmem.h | 4 +-- drivers/cxl/mem.c | 2 + drivers/cxl/port.c | 2 + tools/testing/cxl/test/cxl.c | 20 +++++++------ tools/testing/cxl/test/mock.c | 10 +++---- 9 files changed, 86 insertions(+), 78 deletions(-)
drivers/cxl/core/pci.c +2 -2
@@ -67,7 +67,7 @@ static int match_add_dports(struct pci_dev *pdev, void *data)/** * devm_cxl_port_enumerate_dports - enumerate downstream ports of the upstream port -* @port: cxl_port whose ->uport is the upstream of dports to be enumerated +* @port: cxl_port whose ->uport_dev is the upstream of dports to be enumerated * * Returns a positive number of dports enumerated or a negative error * code.
@@ -622,7 +622,7 @@ static int cxl_cdat_read_table(struct device *dev,*/ void read_cdat_data (struct cxl_port *port) { -struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport); +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev);struct device *host = cxlmd->dev.parent;struct device *dev = &port->dev;struct pci_doe_mb *cdat_doe;
drivers/cxl/core/port.c +33 -28
174 lines not on this page · raw message
drivers/cxl/core/region.c +25 -23
175 lines not on this page · raw message
drivers/cxl/cxl.h +7 -6
@@ -536,7 +536,7 @@ struct cxl_dax_region {* downstream port devices to construct a CXL memory * decode hierarchy. * @dev: this port's device -* @uport: PCI or platform device implementing the upstream port capability +* @uport_dev: PCI or platform device implementing the upstream port capability * @host_bridge: Shortcut to the platform attach point for this port * @id: id for port device-name * @dports: cxl_dport instances referenced by decoders
@@ -555,7 +555,7 @@ struct cxl_dax_region {*/ struct cxl_port {struct device dev; -struct device *uport; +struct device *uport_dev;struct device *host_bridge;int id;struct xarray dports;
@@ -641,21 +641,22 @@ struct cxl_region_ref {/* * The platform firmware device hosting the root is also the top of the * CXL port topology. All other CXL ports have another CXL port as their -* parent and their ->uport / host device is out-of-line of the port +* parent and their ->uport_dev / host device is out-of-line of the port * ancestry. */ static inline bool is_cxl_root (struct cxl_port *port) { -return port->uport == port->dev.parent; +return port->uport_dev == port->dev.parent; }bool is_cxl_port (const struct device *dev);struct cxl_port *to_cxl_port (const struct device *dev);struct pci_bus ; -int devm_cxl_register_pci_bus (struct device *host,struct device *uport, +int devm_cxl_register_pci_bus (struct device *host,struct device *uport_dev,struct pci_bus *bus);struct pci_bus *cxl_port_to_pci_bus (struct cxl_port *port); -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *host, +struct device *uport_dev,resource_size_t component_reg_phys,struct cxl_dport *parent_dport);struct cxl_port *find_cxl_root (struct cxl_port *port);
drivers/cxl/cxlmem.h +2 -2
@@ -72,13 +72,13 @@ cxled_to_memdev(struct cxl_endpoint_decoder *cxled) {struct cxl_port *port =to_cxl_port (cxled->cxld.dev.parent); -return to_cxl_memdev (port->uport); +return to_cxl_memdev (port->uport_dev); }bool is_cxl_memdev (const struct device *dev);static inline bool is_cxl_endpoint (struct cxl_port *port) { -return is_cxl_memdev (port->uport); +return is_cxl_memdev (port->uport_dev); }struct cxl_memdev *devm_cxl_add_memdev (struct cxl_dev_state *cxlds);
drivers/cxl/mem.c +1 -1
@@ -163,7 +163,7 @@ static int cxl_mem_probe(struct device *dev) }if (dport->rch) - endpoint_parent = parent_port->uport; + endpoint_parent = parent_port->uport_dev;else endpoint_parent = &parent_port->dev;
drivers/cxl/port.c +1 -1
@@ -91,7 +91,7 @@ static int cxl_switch_port_probe(struct cxl_port *port)static int cxl_endpoint_port_probe (struct cxl_port *port) {struct cxl_endpoint_dvsec_info info = { .port = port }; -struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport); +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev);struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_hdm *cxlhdm;struct cxl_port *root;
tools/testing/cxl/test/cxl.c +10 -10
@@ -754,7 +754,7 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld)/* check is endpoint is attach to host-bridge0 */ port =cxled_to_port (cxled); do { - if (port->uport == &cxl_host_bridge[0]->dev) { + if (port->uport_dev == &cxl_host_bridge[0]->dev) { hb0 = true; break; }
@@ -889,7 +889,7 @@ static int mock_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,mock_init_hdm_decoder (cxld);if (target_count) { - rc =device_for_each_child (port->uport, &ctx, + rc =device_for_each_child (port->uport_dev, &ctx, map_targets);if (rc) {put_device (&cxld->dev);
@@ -919,29 +919,29 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)int i, array_size;if (port->depth ==1 ) { -if (is_multi_bridge (port->uport)) { +if (is_multi_bridge (port->uport_dev)) { array_size =ARRAY_SIZE (cxl_root_port); array = cxl_root_port; - }else if (is_single_bridge (port->uport)) { + }else if (is_single_bridge (port->uport_dev)) { array_size =ARRAY_SIZE (cxl_root_single); array = cxl_root_single; }else {dev_dbg (&port->dev,"%s: unknown bridge type\n" , -dev_name (port->uport)); +dev_name (port->uport_dev));return -ENXIO ; } }else if (port->depth ==2 ) {struct cxl_port *parent =to_cxl_port (port->dev.parent); -if (is_multi_bridge (parent->uport)) { +if (is_multi_bridge (parent->uport_dev)) { array_size =ARRAY_SIZE (cxl_switch_dport); array = cxl_switch_dport; - }else if (is_single_bridge (parent->uport)) { + }else if (is_single_bridge (parent->uport_dev)) { array_size =ARRAY_SIZE (cxl_swd_single); array = cxl_swd_single; }else {dev_dbg (&port->dev,"%s: unknown bridge type\n" , -dev_name (port->uport)); +dev_name (port->uport_dev));return -ENXIO ; } }else {
@@ -954,9 +954,9 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)struct platform_device *pdev = array[i];struct cxl_dport *dport; -if (pdev->dev.parent != port->uport) { +if (pdev->dev.parent != port->uport_dev) {dev_dbg (&port->dev,"%s: mismatch parent %s\n" , -dev_name (port->uport), +dev_name (port->uport_dev),dev_name (pdev->dev.parent));continue ; }
tools/testing/cxl/test/mock.c +5 -5
@@ -139,7 +139,7 @@ struct cxl_hdm *__wrap_devm_cxl_setup_hdm(struct cxl_port *port,struct cxl_hdm *cxlhdm;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) cxlhdm = ops->devm_cxl_setup_hdm (port, info);else cxlhdm =devm_cxl_setup_hdm (port, info);
@@ -154,7 +154,7 @@ int __wrap_devm_cxl_enable_hdm(struct cxl_port *port, struct cxl_hdm *cxlhdm)int index, rc;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc =0 ;else rc =devm_cxl_enable_hdm (port, cxlhdm);
@@ -169,7 +169,7 @@ int __wrap_devm_cxl_add_passthrough_decoder(struct cxl_port *port)int rc, index;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_add_passthrough_decoder (port);else rc =devm_cxl_add_passthrough_decoder (port);
@@ -186,7 +186,7 @@ int __wrap_devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,struct cxl_port *port = cxlhdm->port;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_enumerate_decoders (cxlhdm, info);else rc =devm_cxl_enumerate_decoders (cxlhdm, info);
@@ -201,7 +201,7 @@ int __wrap_devm_cxl_port_enumerate_dports(struct cxl_port *port)int rc, index;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_port_enumerate_dports (port);else rc =devm_cxl_port_enumerate_dports (port);
Re: [PATCH v5 03/26] cxl: Rename member @dport of struct cxl_dport to @dev
Quoted 22 lines · Dan Williams
Quoted 21 lines · Terry Bowman
Hi Dan,
Quoted 16 lines · Dan Williams
Quoted 5 lines · Terry Bowman
From: Robert Richter <[email protected]>
Reading code like
dport->dportdoes not immediately suggest that this points to the corresponding device structure of the dport. Rename struct member @dport to @dev.This one I don't agree with.
This can switch to ->dport_dev if you like. The reason for ->dport was for symmetry with the ->uport of a 'struct
cxl_port'. So if you change this to ->dport_dev then also make the ->uport_dev change for symmetry.Unlike a 'struct
cxl_port' a 'structcxl_dport' is not a device in its own right which is what I see when I readdport->dev.
Quoted 7 lines
Quoted 4 lines
Ok, I'll change the structure member names and code to use the following:
structcxl_dport::dev-> structcxl_dport::dport_devstruct
cxl_port::uport-> structcxl_port::uport_devSounds good, to save duplication of work I already have this that you can just adopt:
Thanks Dan. I'll insert your patch into the series.
Regards,
Terry
Quoted 646 lines
-- >8 --
Subject: cxl: Rename 'uport' to 'uport_dev'From: Dan Williams <[email protected]>
For symmetry with the recent rename of ->dport_dev for a 'struct
cxl_dport', add the "_dev" suffix to the ->uport property of a 'structcxl_port'. These devices represent the downstream-port-device and upstream-port-device respectively in the CXL/PCIe topology.Signed-off-by Dan Williams <[email protected]>--- drivers/cxl/core/pci.c | 4 +-- drivers/cxl/core/port.c | 61 ++++++++++++++++++++++------------------- drivers/cxl/core/region.c | 48 +++++++++++++++++--------------- drivers/cxl/cxl.h | 13 +++++---- drivers/cxl/cxlmem.h | 4 +-- drivers/cxl/mem.c | 2 + drivers/cxl/port.c | 2 + tools/testing/cxl/test/cxl.c | 20 +++++++------ tools/testing/cxl/test/mock.c | 10 +++---- 9 files changed, 86 insertions(+), 78 deletions(-)drivers/cxl/core/pci.c +2
-2@@ -67,7 +67,7 @@ static int match_add_dports(struct pci_dev *pdev, void *data)/** * devm_cxl_port_enumerate_dports - enumerate downstream ports of the upstream port -* @port: cxl_port whose ->uport is the upstream of dports to be enumerated +* @port: cxl_port whose ->uport_dev is the upstream of dports to be enumerated * * Returns a positive number of dports enumerated or a negative error * code. @@ -622,7 +622,7 @@ static int cxl_cdat_read_table(struct device *dev,*/ void read_cdat_data (struct cxl_port *port) { -struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport); +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev);struct device *host = cxlmd->dev.parent;struct device *dev = &port->dev;struct pci_doe_mb *cdat_doe;drivers/cxl/core/port.c +33
-28@@ -561,9 +561,9 @@ static void unregister_port(void *_port)* unregistered while holding their parent port lock. */ if (!parent) - lock_dev = port->uport; + lock_dev = port->uport_dev;else if (is_cxl_root (parent)) - lock_dev = parent->uport; + lock_dev = parent->uport_dev;else lock_dev = &parent->dev;@@ -583,7 +583,8 @@ static int devm_cxl_link_uport(struct device *host, struct cxl_port *port) {int rc; - rc =sysfs_create_link (&port->dev.kobj, &port->uport->kobj,"uport" ); + rc =sysfs_create_link (&port->dev.kobj, &port->uport_dev->kobj, +"uport" );if (rc)return rc;return devm_add_action_or_reset (host, cxl_unlink_uport, port);@@ -614,7 +615,7 @@ static int devm_cxl_link_parent_dport(struct device *host,static struct lock_class_key cxl_port_key; -static struct cxl_port *cxl_port_alloc (struct device *uport, +static struct cxl_port *cxl_port_alloc (struct device *uport_dev,resource_size_t component_reg_phys,struct cxl_dport *parent_dport) {@@ -630,7 +631,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,if (rc <0 ) goto err; port->id = rc; - port->uport = uport; + port->uport_dev = uport_dev;/* * The top-level cxl_port "cxl_root" does not have a cxl_port as @@ -660,10 +661,11 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,else if (parent_dport->rch) port->host_bridge = parent_dport->dport_dev;else - port->host_bridge = iter->uport; -dev_dbg (uport,"host-bridge: %s\n" ,dev_name (port->host_bridge)); + port->host_bridge = iter->uport_dev; +dev_dbg (uport_dev,"host-bridge: %s\n" , +dev_name (port->host_bridge)); }else - dev->parent = uport; + dev->parent = uport_dev; port->component_reg_phys = component_reg_phys;ida_init (&port->decoder_ida);@@ -687,7 +689,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, }static struct cxl_port *__devm_cxl_add_port (struct device *host, -struct device *uport, +struct device *uport_dev,resource_size_t component_reg_phys,struct cxl_dport *parent_dport) {@@ -695,12 +697,12 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,struct device *dev;int rc; - port =cxl_port_alloc (uport, component_reg_phys, parent_dport); + port =cxl_port_alloc (uport_dev, component_reg_phys, parent_dport);if (IS_ERR (port))return port; dev = &port->dev; -if (is_cxl_memdev (uport)) +if (is_cxl_memdev (uport_dev)) rc =dev_set_name (dev,"endpoint%d" , port->id);else if (parent_dport) rc =dev_set_name (dev,"port%d" , port->id);@@ -735,28 +737,29 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,/** * devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy * @host: host device for devm operations -* @uport: "physical" device implementing this upstream port +* @uport_dev: "physical" device implementing this upstream port * @component_reg_phys: (optional) for configurable cxl_port instances * @parent_dport: next hop up in the CXL memory decode hierarchy */ -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *host, +struct device *uport_dev,resource_size_t component_reg_phys,struct cxl_dport *parent_dport) {struct cxl_port *port, *parent_port; - port =__devm_cxl_add_port (host, uport, component_reg_phys, + port =__devm_cxl_add_port (host, uport_dev, component_reg_phys, parent_dport); parent_port = parent_dport ? parent_dport->port :NULL ;if (IS_ERR (port)) { -dev_dbg (uport,"Failed to add%s%s%s: %ld\n" , +dev_dbg (uport_dev,"Failed to add%s%s%s: %ld\n" , parent_port ?" port to " :"" , parent_port ?dev_name (&parent_port->dev) :"" , parent_port ?"" :" root port" ,PTR_ERR (port)); }else { - dev_dbg(uport, "%s added%s%s%s\n", + dev_dbg(uport_dev, "%s added%s%s%s\n", dev_name(&port->dev), parent_port ? " to " : "", parent_port ? dev_name(&parent_port->dev) : "",@@ -773,33 +776,34 @@ struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port)if (is_cxl_root (port))return NULL ; -if (dev_is_pci (port->uport)) { -struct pci_dev *pdev =to_pci_dev (port->uport); +if (dev_is_pci (port->uport_dev)) { +struct pci_dev *pdev =to_pci_dev (port->uport_dev);return pdev->subordinate; } -return xa_load (&cxl_root_buses, (unsigned long )port->uport); +return xa_load (&cxl_root_buses, (unsigned long )port->uport_dev); }EXPORT_SYMBOL_NS_GPL (cxl_port_to_pci_bus,CXL ); -static void unregister_pci_bus (void *uport) +static void unregister_pci_bus (void *uport_dev) { -xa_erase (&cxl_root_buses, (unsigned long )uport); +xa_erase (&cxl_root_buses, (unsigned long )uport_dev); } -int devm_cxl_register_pci_bus (struct device *host,struct device *uport, +int devm_cxl_register_pci_bus (struct device *host,struct device *uport_dev,struct pci_bus *bus) {int rc; -if (dev_is_pci (uport)) +if (dev_is_pci (uport_dev))return -EINVAL ; - rc =xa_insert (&cxl_root_buses, (unsigned long )uport, bus,GFP_KERNEL ); + rc =xa_insert (&cxl_root_buses, (unsigned long )uport_dev, bus, +GFP_KERNEL );if (rc)return rc; -return devm_add_action_or_reset (host, unregister_pci_bus, uport); +return devm_add_action_or_reset (host, unregister_pci_bus, uport_dev); }EXPORT_SYMBOL_NS_GPL (devm_cxl_register_pci_bus,CXL );@@ -920,7 +924,7 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,int rc;if (is_cxl_root (port)) - host = port->uport; + host = port->uport_dev;else host = &port->dev;@@ -1374,7 +1378,7 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd, rc =PTR_ERR (port); else {dev_dbg (&cxlmd->dev,"add to new port %s:%s\n" , -dev_name (&port->dev),dev_name (port->uport)); +dev_name (&port->dev),dev_name (port->uport_dev)); rc =cxl_add_ep (dport, &cxlmd->dev);if (rc == -EBUSY ) {/* @@ -1436,7 +1440,8 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)if (port) {dev_dbg (&cxlmd->dev,"found already registered port %s:%s\n" , -dev_name (&port->dev),dev_name (port->uport)); +dev_name (&port->dev), +dev_name (port->uport_dev)); rc =cxl_add_ep (dport, &cxlmd->dev);/* drivers/cxl/core/region.c +25
-23@@ -906,10 +906,10 @@ static int cxl_port_attach_region(struct cxl_port *port,dev_dbg (&cxlr->dev,"%s:%s %s add: %s:%s @ %d next: %s nr_eps: %d nr_targets: %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),dev_name (&cxld->dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos, - ep ? ep->next ?dev_name (ep->next->uport) : + ep ? ep->next ?dev_name (ep->next->uport_dev) :dev_name (&cxlmd->dev) :"none" , cxl_rr->nr_eps, cxl_rr->nr_targets);@@ -984,7 +984,7 @@ static int check_last_peer(struct cxl_endpoint_decoder *cxled,*/ if (pos < distance) {dev_dbg (&cxlr->dev,"%s:%s: cannot host %s:%s at %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos);return -ENXIO ; }@@ -994,7 +994,7 @@ static int check_last_peer(struct cxl_endpoint_decoder *cxled,if (ep->dport != ep_peer->dport) {dev_dbg (&cxlr->dev,"%s:%s: %s:%s pos %d mismatched peer %s:%s\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos,dev_name (&cxlmd_peer->dev),dev_name (&cxled_peer->cxld.dev));@@ -1026,7 +1026,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,*/ if (!is_power_of_2 (cxl_rr->nr_targets)) {dev_dbg (&cxlr->dev,"%s:%s: invalid target count %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev), cxl_rr->nr_targets);return -EINVAL ; }@@ -1076,7 +1076,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, rc =granularity_to_eig (parent_ig, &peig);if (rc) {dev_dbg (&cxlr->dev,"%s:%s: invalid parent granularity: %d\n" , -dev_name (parent_port->uport), +dev_name (parent_port->uport_dev),dev_name (&parent_port->dev), parent_ig);return rc; }@@ -1084,7 +1084,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, rc =ways_to_eiw (parent_iw, &peiw);if (rc) {dev_dbg (&cxlr->dev,"%s:%s: invalid parent interleave: %d\n" , -dev_name (parent_port->uport), +dev_name (parent_port->uport_dev),dev_name (&parent_port->dev), parent_iw);return rc; }@@ -1093,7 +1093,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, rc =ways_to_eiw (iw, &eiw);if (rc) {dev_dbg (&cxlr->dev,"%s:%s: invalid port interleave: %d\n" , -dev_name (port->uport),dev_name (&port->dev), iw); +dev_name (port->uport_dev),dev_name (&port->dev), iw);return rc; }@@ -1113,7 +1113,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, rc =eig_to_granularity (eig, &ig);if (rc) {dev_dbg (&cxlr->dev,"%s:%s: invalid interleave: %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),256 << eig);return rc; }@@ -1126,11 +1126,11 @@ static int cxl_port_setup_targets(struct cxl_port *port, ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)) {dev_err (&cxlr->dev,"%s:%s %s expected iw: %d ig: %d %pr\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev), __func__, iw, ig, p->res); dev_err(&cxlr->dev, "%s:%s %s got iw: %d ig: %d state: %s %#llx:%#llx\n", - dev_name(port->uport), dev_name(&port->dev), + dev_name(port->uport_dev), dev_name(&port->dev), __func__, cxld->interleave_ways, cxld->interleave_granularity, (cxld->flags & CXL_DECODER_F_ENABLE) ?@@ -1147,20 +1147,20 @@ static int cxl_port_setup_targets(struct cxl_port *port, .end = p->res->end, }; } -dev_dbg (&cxlr->dev,"%s:%s iw: %d ig: %d\n" ,dev_name (port->uport), +dev_dbg (&cxlr->dev,"%s:%s iw: %d ig: %d\n" ,dev_name (port->uport_dev),dev_name (&port->dev), iw, ig); add_target:if (cxl_rr->nr_targets_set == cxl_rr->nr_targets) {dev_dbg (&cxlr->dev,"%s:%s: targets full trying to add %s:%s at %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos);return -ENXIO ; }if (test_bit (CXL_REGION_F_AUTO , &cxlr->flags)) {if (cxlsd->target[cxl_rr->nr_targets_set] != ep->dport) {dev_dbg (&cxlr->dev,"%s:%s: %s expected %s at %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev),dev_name (&cxlsd->cxld.dev),dev_name (ep->dport->dport_dev), cxl_rr->nr_targets_set);@@ -1172,7 +1172,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, out_target_set: cxl_rr->nr_targets_set += inc;dev_dbg (&cxlr->dev,"%s:%s target[%d] = %s for %s:%s @ %d\n" , -dev_name (port->uport),dev_name (&port->dev), +dev_name (port->uport_dev),dev_name (&port->dev), cxl_rr->nr_targets_set -1 ,dev_name (ep->dport->dport_dev),dev_name (&cxlmd->dev),dev_name (&cxled->cxld.dev), pos);@@ -1492,7 +1492,7 @@ static int cmp_decode_pos(const void *a, const void *b)if (!dev) {struct range *range = &cxled_a->cxld.hpa_range; -dev_err (port->uport, +dev_err (port->uport_dev,"failed to find decoder that maps %#llx-%#llx\n" , range->start, range->end); goto err;@@ -1507,14 +1507,15 @@ static int cmp_decode_pos(const void *a, const void *b)put_device (dev);if (a_pos <0 || b_pos <0 ) { -dev_err (port->uport, +dev_err (port->uport_dev,"failed to find shared decoder for %s and %s\n" ,dev_name (cxlmd_a->dev.parent),dev_name (cxlmd_b->dev.parent)); goto err; } -dev_dbg (port->uport,"%s comes %s %s\n" ,dev_name (cxlmd_a->dev.parent), +dev_dbg (port->uport_dev,"%s comes %s %s\n" , +dev_name (cxlmd_a->dev.parent), a_pos - b_pos <0 ?"before" :"after" ,dev_name (cxlmd_b->dev.parent));@@ -2059,11 +2060,11 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,if (rc) goto err; - rc =devm_add_action_or_reset (port->uport, unregister_region, cxlr); + rc =devm_add_action_or_reset (port->uport_dev, unregister_region, cxlr);if (rc)return ERR_PTR (rc); -dev_dbg (port->uport,"%s: created %s\n" , +dev_dbg (port->uport_dev,"%s: created %s\n" ,dev_name (&cxlrd->cxlsd.cxld.dev),dev_name (dev));return cxlr;@@ -2191,7 +2192,7 @@ static ssize_t delete_region_store(struct device *dev,if (IS_ERR (cxlr))return PTR_ERR (cxlr); -devm_release_action (port->uport, unregister_region, cxlr); +devm_release_action (port->uport_dev, unregister_region, cxlr);put_device (&cxlr->dev);return len;@@ -2356,7 +2357,8 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port) rc =device_for_each_child (&port->dev, &ctx, poison_by_decoder);if (rc ==1 ) - rc =cxl_get_poison_unmapped (to_cxl_memdev (port->uport), &ctx); + rc =cxl_get_poison_unmapped (to_cxl_memdev (port->uport_dev), + &ctx);up_read (&cxl_region_rwsem);return rc;@@ -2732,7 +2734,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, err:up_write (&cxl_region_rwsem); -devm_release_action (port->uport, unregister_region, cxlr); +devm_release_action (port->uport_dev, unregister_region, cxlr);return ERR_PTR (rc); }drivers/cxl/cxl.h +7
-6@@ -536,7 +536,7 @@ struct cxl_dax_region {* downstream port devices to construct a CXL memory * decode hierarchy. * @dev: this port's device -* @uport: PCI or platform device implementing the upstream port capability +* @uport_dev: PCI or platform device implementing the upstream port capability * @host_bridge: Shortcut to the platform attach point for this port * @id: id for port device-name * @dports: cxl_dport instances referenced by decoders @@ -555,7 +555,7 @@ struct cxl_dax_region {*/ struct cxl_port {struct device dev; -struct device *uport; +struct device *uport_dev;struct device *host_bridge;int id;struct xarray dports;@@ -641,21 +641,22 @@ struct cxl_region_ref {/* * The platform firmware device hosting the root is also the top of the * CXL port topology. All other CXL ports have another CXL port as their -* parent and their ->uport / host device is out-of-line of the port +* parent and their ->uport_dev / host device is out-of-line of the port * ancestry. */ static inline bool is_cxl_root (struct cxl_port *port) { -return port->uport == port->dev.parent; +return port->uport_dev == port->dev.parent; }bool is_cxl_port (const struct device *dev);struct cxl_port *to_cxl_port (const struct device *dev);struct pci_bus ; -int devm_cxl_register_pci_bus (struct device *host,struct device *uport, +int devm_cxl_register_pci_bus (struct device *host,struct device *uport_dev,struct pci_bus *bus);struct pci_bus *cxl_port_to_pci_bus (struct cxl_port *port); -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *host, +struct device *uport_dev,resource_size_t component_reg_phys,struct cxl_dport *parent_dport);struct cxl_port *find_cxl_root (struct cxl_port *port);drivers/cxl/cxlmem.h +2
-2@@ -72,13 +72,13 @@ cxled_to_memdev(struct cxl_endpoint_decoder *cxled) {struct cxl_port *port =to_cxl_port (cxled->cxld.dev.parent); -return to_cxl_memdev (port->uport); +return to_cxl_memdev (port->uport_dev); }bool is_cxl_memdev (const struct device *dev);static inline bool is_cxl_endpoint (struct cxl_port *port) { -return is_cxl_memdev (port->uport); +return is_cxl_memdev (port->uport_dev); }struct cxl_memdev *devm_cxl_add_memdev (struct cxl_dev_state *cxlds);drivers/cxl/mem.c +1
-1@@ -163,7 +163,7 @@ static int cxl_mem_probe(struct device *dev) }if (dport->rch) - endpoint_parent = parent_port->uport; + endpoint_parent = parent_port->uport_dev;else endpoint_parent = &parent_port->dev;drivers/cxl/port.c +1
-1@@ -91,7 +91,7 @@ static int cxl_switch_port_probe(struct cxl_port *port)static int cxl_endpoint_port_probe (struct cxl_port *port) {struct cxl_endpoint_dvsec_info info = { .port = port }; -struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport); +struct cxl_memdev *cxlmd =to_cxl_memdev (port->uport_dev);struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_hdm *cxlhdm;struct cxl_port *root;tools/testing/cxl/test/cxl.c +10
-10@@ -754,7 +754,7 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld)/* check is endpoint is attach to host-bridge0 */ port =cxled_to_port (cxled); do { - if (port->uport == &cxl_host_bridge[0]->dev) { + if (port->uport_dev == &cxl_host_bridge[0]->dev) { hb0 = true; break; }@@ -889,7 +889,7 @@ static int mock_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,mock_init_hdm_decoder (cxld);if (target_count) { - rc =device_for_each_child (port->uport, &ctx, + rc =device_for_each_child (port->uport_dev, &ctx, map_targets);if (rc) {put_device (&cxld->dev);@@ -919,29 +919,29 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)int i, array_size;if (port->depth ==1 ) { -if (is_multi_bridge (port->uport)) { +if (is_multi_bridge (port->uport_dev)) { array_size =ARRAY_SIZE (cxl_root_port); array = cxl_root_port; - }else if (is_single_bridge (port->uport)) { + }else if (is_single_bridge (port->uport_dev)) { array_size =ARRAY_SIZE (cxl_root_single); array = cxl_root_single; }else {dev_dbg (&port->dev,"%s: unknown bridge type\n" , -dev_name (port->uport)); +dev_name (port->uport_dev));return -ENXIO ; } }else if (port->depth ==2 ) {struct cxl_port *parent =to_cxl_port (port->dev.parent); -if (is_multi_bridge (parent->uport)) { +if (is_multi_bridge (parent->uport_dev)) { array_size =ARRAY_SIZE (cxl_switch_dport); array = cxl_switch_dport; - }else if (is_single_bridge (parent->uport)) { + }else if (is_single_bridge (parent->uport_dev)) { array_size =ARRAY_SIZE (cxl_swd_single); array = cxl_swd_single; }else {dev_dbg (&port->dev,"%s: unknown bridge type\n" , -dev_name (port->uport)); +dev_name (port->uport_dev));return -ENXIO ; } }else {@@ -954,9 +954,9 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port)struct platform_device *pdev = array[i];struct cxl_dport *dport; -if (pdev->dev.parent != port->uport) { +if (pdev->dev.parent != port->uport_dev) {dev_dbg (&port->dev,"%s: mismatch parent %s\n" , -dev_name (port->uport), +dev_name (port->uport_dev),dev_name (pdev->dev.parent));continue ; }tools/testing/cxl/test/mock.c +5
-5@@ -139,7 +139,7 @@ struct cxl_hdm *__wrap_devm_cxl_setup_hdm(struct cxl_port *port,struct cxl_hdm *cxlhdm;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) cxlhdm = ops->devm_cxl_setup_hdm (port, info);else cxlhdm =devm_cxl_setup_hdm (port, info);@@ -154,7 +154,7 @@ int __wrap_devm_cxl_enable_hdm(struct cxl_port *port, struct cxl_hdm *cxlhdm)int index, rc;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc =0 ;else rc =devm_cxl_enable_hdm (port, cxlhdm);@@ -169,7 +169,7 @@ int __wrap_devm_cxl_add_passthrough_decoder(struct cxl_port *port)int rc, index;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_add_passthrough_decoder (port);else rc =devm_cxl_add_passthrough_decoder (port);@@ -186,7 +186,7 @@ int __wrap_devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,struct cxl_port *port = cxlhdm->port;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_enumerate_decoders (cxlhdm, info);else rc =devm_cxl_enumerate_decoders (cxlhdm, info);@@ -201,7 +201,7 @@ int __wrap_devm_cxl_port_enumerate_dports(struct cxl_port *port)int rc, index;struct cxl_mock_ops *ops =get_cxl_mock_ops (&index); -if (ops && ops->is_mock_port (port->uport)) +if (ops && ops->is_mock_port (port->uport_dev)) rc = ops->devm_cxl_port_enumerate_dports (port);else rc =devm_cxl_port_enumerate_dports (port);
PATCH v5 4/26 cxl/core/regs: Rename phys_addr in cxl_map_component_regs()
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Trivial change that renames variable phys_addr in cxl_map_component_regs() to shorten its length to keep the 80 char size limit for the line and also for consistency between the different paths.
--- drivers/cxl/core/regs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/cxl/core/regs.c +3 -3
@@ -213,16 +213,16 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs,for (i =0 ; i <ARRAY_SIZE (mapinfo); i++) {struct mapinfo *mi = &mapinfo[i]; -resource_size_t phys_addr; +resource_size_t addr;resource_size_t length;if (!mi->rmap->valid)continue ;if (!test_bit (mi->rmap->id, &map_mask))continue ; - phys_addr = map->resource + mi->rmap->offset; + addr = map->resource + mi->rmap->offset; length = mi->rmap->size; - *(mi->addr) =devm_cxl_iomap_block (dev, phys_addr, length); + *(mi->addr) =devm_cxl_iomap_block (dev, addr, length);if (!*(mi->addr))return -ENOMEM ; }
Signature · 1 line
2.34.1
RE: [PATCH v5 04/26] cxl/core/regs: Rename phys_addr in cxl_map_component_regs()
Quoted 24 lines · Terry Bowman
From: Robert Richter <[email protected]>
Trivial change that renames variable
phys_addrincxl_map_component_regs()to shorten its length to keep the 80 char size limit for the line and also for consistency between the different paths.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/regs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)drivers/cxl/core/regs.c +1
-1@@ -213,16 +213,16 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs,for (i =0 ; i <ARRAY_SIZE (mapinfo); i++) {struct mapinfo *mi = &mapinfo[i]; -resource_size_t phys_addr; +resource_size_t addr;resource_size_t length;
drivers/cxl/core/regs.c int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, · line 225
Quoted 10 lines
drivers/cxl/core/regs.c +2
-2if (!mi->rmap->valid)continue ;if (!test_bit (mi->rmap->id, &map_mask))continue ; - phys_addr = map->resource + mi->rmap->offset; + addr = map->resource + mi->rmap->offset; length = mi->rmap->size; - *(mi->addr) =devm_cxl_iomap_block (dev, phys_addr, length); + *(mi->addr) =devm_cxl_iomap_block (dev, addr, length);
...but this line is only 75 chars, so not sure why you bothered, I don't understand the consistency comment either.
PATCH v5 5/26 cxl/core/regs: Add @dev to cxl_register_map
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
The corresponding device of a register mapping is used for devm operations and logging. For operations with struct cxl_register_map the device needs to be kept track separately. To simpify the involved function interfaces, add @dev to cxl_register_map.
While at it also reorder function arguments of cxl_map_device_regs() and cxl_map_component_regs() to have the object @cxl_register_map first.
In a result a bunch of functions are available to be used with a @cxl_register_map object.
This patch is in preparation of reworking the component register setup code.
--- drivers/cxl/core/hdm.c | 4 ++-- drivers/cxl/core/regs.c | 16 +++++++++++----- drivers/cxl/cxl.h | 10 ++++++---- drivers/cxl/pci.c | 24 ++++++++++++------------ 4 files changed, 31 insertions(+), 23 deletions(-)
drivers/cxl/core/hdm.c +2 -2
@@ -85,6 +85,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb, struct cxl_component_regs *regs) {struct cxl_register_map map = { + .dev = &port->dev, .resource = port->component_reg_phys, .base = crb, .max_size =CXL_COMPONENT_REG_BLOCK_SIZE ,
@@ -97,8 +98,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb,return -ENODEV ; } -return cxl_map_component_regs (&port->dev, regs, &map, -BIT (CXL_CM_CAP_CAP_ID_HDM )); +return cxl_map_component_regs (&map, regs,BIT (CXL_CM_CAP_CAP_ID_HDM )); }static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info)
drivers/cxl/core/regs.c +11 -5
@@ -199,9 +199,11 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map,unsigned long map_mask) +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs, +unsigned long map_mask) { +struct device *dev = map->dev;struct mapinfo {struct cxl_reg_map *rmap;void __iomem **addr;
@@ -231,10 +233,10 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, }EXPORT_SYMBOL_NS_GPL (cxl_map_component_regs,CXL ); -int cxl_map_device_regs (struct device *dev, -struct cxl_device_regs *regs, -struct cxl_register_map *map) +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs) { +struct device *dev = map->dev;resource_size_t phys_addr = map->resource;struct mapinfo {struct cxl_reg_map *rmap;
@@ -302,7 +304,10 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,u32 regloc_size, regblocks;int regloc, i; +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; map->resource =CXL_RESOURCE_NONE ; + regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL ,CXL_DVSEC_REG_LOCATOR );if (!regloc)
@@ -328,6 +333,7 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, } map->resource =CXL_RESOURCE_NONE ; +return -ENODEV ; }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );
drivers/cxl/cxl.h +6 -4
@@ -231,6 +231,7 @@ struct cxl_device_reg_map {/** * struct cxl_register_map - DVSEC harvested register block mapping parameters +* @dev: device for devm operations and logging * @base: virtual base of the register-block-BAR + @block_offset * @resource: physical resource base of the register block * @max_size: maximum mapping size to perform register search
@@ -239,6 +240,7 @@ struct cxl_device_reg_map {* @device_map: cxl_reg_maps for device registers */ struct cxl_register_map { +struct device *dev;void __iomem *base;resource_size_t resource;resource_size_t max_size;
@@ -253,11 +255,11 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map, +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs,unsigned long map_mask); -int cxl_map_device_regs (struct device *dev,struct cxl_device_regs *regs, -struct cxl_register_map *map); +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs);enum cxl_regloc_type ; int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
drivers/cxl/pci.c +12 -12
@@ -274,9 +274,9 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } -static int cxl_map_regblock (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_map_regblock (struct cxl_register_map *map) { -struct device *dev = &pdev->dev; +struct device *dev = map->dev; map->base =ioremap (map->resource, map->max_size);if (!map->base) {
@@ -285,21 +285,21 @@ static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) }dev_dbg (dev,"Mapped CXL Memory Device resource %pa\n" , &map->resource); +return 0 ; } -static void cxl_unmap_regblock (struct pci_dev *pdev, -struct cxl_register_map *map) +static void cxl_unmap_regblock (struct cxl_register_map *map) {iounmap (map->base); map->base =NULL ; } -static int cxl_probe_regs (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_probe_regs (struct cxl_register_map *map) {struct cxl_component_reg_map *comp_map;struct cxl_device_reg_map *dev_map; -struct device *dev = &pdev->dev; +struct device *dev = map->dev;void __iomem *base = map->base;switch (map->reg_type) {
@@ -346,12 +346,12 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,if (rc)return rc; - rc =cxl_map_regblock (pdev, map); + rc =cxl_map_regblock (map);if (rc)return rc; - rc =cxl_probe_regs (pdev, map); -cxl_unmap_regblock (pdev, map); + rc =cxl_probe_regs (map); +cxl_unmap_regblock (map);return rc; }
@@ -688,7 +688,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; - rc =cxl_map_device_regs (&pdev->dev, &cxlds->regs.device_regs, &map); + rc =cxl_map_device_regs (&map, &cxlds->regs.device_regs);if (rc)return rc;
@@ -703,8 +703,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) cxlds->component_reg_phys = map.resource; - rc =cxl_map_component_regs (&pdev->dev, &cxlds->regs.component, - &map,BIT (CXL_CM_CAP_CAP_ID_RAS )); + rc =cxl_map_component_regs (&map, &cxlds->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ));if (rc)dev_dbg (&pdev->dev,"Failed to map RAS capability.\n" );
Signature · 1 line
2.34.1
RE: [PATCH v5 05/26] cxl/core/regs: Add @dev to cxl_register_map
Quoted 11 lines · Terry Bowman
From: Robert Richter <[email protected]>
The corresponding device of a register mapping is used for devm operations and logging. For operations with struct
cxl_register_mapthe device needs to be kept track separately. To simpify the involved function interfaces, add @dev tocxl_register_map.While at it also reorder function arguments of
cxl_map_device_regs()andcxl_map_component_regs()to have the object @cxl_register_map first.
Quoted 5 lines
In a result a bunch of functions are available to be used with a @cxl_register_map object.
This patch is in preparation of reworking the component register setup code.
Looks good to me some small formatting requests below:
Quoted 65 lines
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/hdm.c | 4 ++-- drivers/cxl/core/regs.c | 16 +++++++++++----- drivers/cxl/cxl.h | 10 ++++++---- drivers/cxl/pci.c | 24 ++++++++++++------------ 4 files changed, 31 insertions(+), 23 deletions(-)drivers/cxl/core/hdm.c +2
-2@@ -85,6 +85,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb, struct cxl_component_regs *regs) {struct cxl_register_map map = { + .dev = &port->dev, .resource = port->component_reg_phys, .base = crb, .max_size =CXL_COMPONENT_REG_BLOCK_SIZE ,@@ -97,8 +98,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb,return -ENODEV ; } -return cxl_map_component_regs (&port->dev, regs, &map, -BIT (CXL_CM_CAP_CAP_ID_HDM )); +return cxl_map_component_regs (&map, regs,BIT (CXL_CM_CAP_CAP_ID_HDM )); }static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info)drivers/cxl/core/regs.c +7
-5@@ -199,9 +199,11 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map,unsigned long map_mask) +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs, +unsigned long map_mask) { +struct device *dev = map->dev;struct mapinfo {struct cxl_reg_map *rmap;void __iomem **addr;@@ -231,10 +233,10 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, }EXPORT_SYMBOL_NS_GPL (cxl_map_component_regs,CXL ); -int cxl_map_device_regs (struct device *dev, -struct cxl_device_regs *regs, -struct cxl_register_map *map) +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs) { +struct device *dev = map->dev;resource_size_t phys_addr = map->resource;struct mapinfo {struct cxl_reg_map *rmap;
drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 309
Quoted 6 lines
drivers/cxl/core/regs.c +2
-0@@ -302,7 +304,10 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,u32 regloc_size, regblocks;int regloc, i; +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; map->resource =CXL_RESOURCE_NONE ;
Use a designated initializer here like other locations:
drivers/cxl/core/regs.c +4 -3
@@ -304,9 +304,10 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,u32 regloc_size, regblocks;int regloc, i; -memset (map,0 ,sizeof (*map)); - map->dev = &pdev->dev; - map->resource =CXL_RESOURCE_NONE ; + *map = (struct cxl_register_map ) { + .dev = &pdev->dev, + .resource =CXL_RESOURCE_NONE , + }; regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL ,CXL_DVSEC_REG_LOCATOR );
drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 310
Quoted 1 line
drivers/cxl/core/regs.c +1
-0+
Remove this unrelated whitespace change.
drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 336
Quoted 8 lines
drivers/cxl/core/regs.c +1
-0regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL ,CXL_DVSEC_REG_LOCATOR );if (!regloc)@@ -328,6 +333,7 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, } map->resource =CXL_RESOURCE_NONE ; +
...and this one too, before Jonathan notices.
Quoted 56 lines
drivers/cxl/core/regs.c +0
-0return -ENODEV ; }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );drivers/cxl/cxl.h +6
-4@@ -231,6 +231,7 @@ struct cxl_device_reg_map {/** * struct cxl_register_map - DVSEC harvested register block mapping parameters +* @dev: device for devm operations and logging * @base: virtual base of the register-block-BAR + @block_offset * @resource: physical resource base of the register block * @max_size: maximum mapping size to perform register search @@ -239,6 +240,7 @@ struct cxl_device_reg_map {* @device_map: cxl_reg_maps for device registers */ struct cxl_register_map { +struct device *dev;void __iomem *base;resource_size_t resource;resource_size_t max_size;@@ -253,11 +255,11 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map, +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs,unsigned long map_mask); -int cxl_map_device_regs (struct device *dev,struct cxl_device_regs *regs, -struct cxl_register_map *map); +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs);enum cxl_regloc_type ; int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,drivers/cxl/pci.c +2
-2@@ -274,9 +274,9 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } -static int cxl_map_regblock (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_map_regblock (struct cxl_register_map *map) { -struct device *dev = &pdev->dev; +struct device *dev = map->dev; map->base =ioremap (map->resource, map->max_size);if (!map->base) {
drivers/cxl/pci.c static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) · line 288
Quoted 4 lines
drivers/cxl/pci.c +1
-0@@ -285,21 +285,21 @@ static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) }dev_dbg (dev,"Mapped CXL Memory Device resource %pa\n" , &map->resource); +
Ditto.
Quoted 60 lines
drivers/cxl/pci.c +9
-10return 0 ; } -static void cxl_unmap_regblock (struct pci_dev *pdev, -struct cxl_register_map *map) +static void cxl_unmap_regblock (struct cxl_register_map *map) {iounmap (map->base); map->base =NULL ; } -static int cxl_probe_regs (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_probe_regs (struct cxl_register_map *map) {struct cxl_component_reg_map *comp_map;struct cxl_device_reg_map *dev_map; -struct device *dev = &pdev->dev; +struct device *dev = map->dev;void __iomem *base = map->base;switch (map->reg_type) {@@ -346,12 +346,12 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,if (rc)return rc; - rc =cxl_map_regblock (pdev, map); + rc =cxl_map_regblock (map);if (rc)return rc; - rc =cxl_probe_regs (pdev, map); -cxl_unmap_regblock (pdev, map); + rc =cxl_probe_regs (map); +cxl_unmap_regblock (map);return rc; }@@ -688,7 +688,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; - rc =cxl_map_device_regs (&pdev->dev, &cxlds->regs.device_regs, &map); + rc =cxl_map_device_regs (&map, &cxlds->regs.device_regs);if (rc)return rc;@@ -703,8 +703,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) cxlds->component_reg_phys = map.resource; - rc =cxl_map_component_regs (&pdev->dev, &cxlds->regs.component, - &map,BIT (CXL_CM_CAP_CAP_ID_RAS )); + rc =cxl_map_component_regs (&map, &cxlds->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ));if (rc)dev_dbg (&pdev->dev,"Failed to map RAS capability.\n" );Signature · 1 line
2.34.1
Re: [PATCH v5 05/26] cxl/core/regs: Add @dev to cxl_register_map
Thanks Dan. I'll make the changes you mention below.
Regards,
Terry
Quoted 254 lines · Dan Williams
Quoted 11 lines · Terry Bowman
From: Robert Richter <[email protected]>
The corresponding device of a register mapping is used for devm operations and logging. For operations with struct
cxl_register_mapthe device needs to be kept track separately. To simpify the involved function interfaces, add @dev tocxl_register_map.While at it also reorder function arguments of
cxl_map_device_regs()andcxl_map_component_regs()to have the object @cxl_register_map first.Quoted 5 lines
In a result a bunch of functions are available to be used with a @cxl_register_map object.
This patch is in preparation of reworking the component register setup code.
Looks good to me some small formatting requests below:
Quoted 65 lines
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/hdm.c | 4 ++-- drivers/cxl/core/regs.c | 16 +++++++++++----- drivers/cxl/cxl.h | 10 ++++++---- drivers/cxl/pci.c | 24 ++++++++++++------------ 4 files changed, 31 insertions(+), 23 deletions(-)drivers/cxl/core/hdm.c +2
-2@@ -85,6 +85,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb, struct cxl_component_regs *regs) {struct cxl_register_map map = { + .dev = &port->dev, .resource = port->component_reg_phys, .base = crb, .max_size =CXL_COMPONENT_REG_BLOCK_SIZE ,@@ -97,8 +98,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb,return -ENODEV ; } -return cxl_map_component_regs (&port->dev, regs, &map, -BIT (CXL_CM_CAP_CAP_ID_HDM )); +return cxl_map_component_regs (&map, regs,BIT (CXL_CM_CAP_CAP_ID_HDM )); }static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info)drivers/cxl/core/regs.c +7
-5@@ -199,9 +199,11 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map,unsigned long map_mask) +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs, +unsigned long map_mask) { +struct device *dev = map->dev;struct mapinfo {struct cxl_reg_map *rmap;void __iomem **addr;@@ -231,10 +233,10 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, }EXPORT_SYMBOL_NS_GPL (cxl_map_component_regs,CXL ); -int cxl_map_device_regs (struct device *dev, -struct cxl_device_regs *regs, -struct cxl_register_map *map) +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs) { +struct device *dev = map->dev;resource_size_t phys_addr = map->resource;struct mapinfo {struct cxl_reg_map *rmap;drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 309
Quoted 6 lines
drivers/cxl/core/regs.c +2
-0@@ -302,7 +304,10 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,u32 regloc_size, regblocks;int regloc, i; +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; map->resource =CXL_RESOURCE_NONE ;Use a designated initializer here like other locations:
drivers/cxl/core/regs.c +4
-3@@ -304,9 +304,10 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,u32 regloc_size, regblocks;int regloc, i; -memset (map,0 ,sizeof (*map)); - map->dev = &pdev->dev; - map->resource =CXL_RESOURCE_NONE ; + *map = (struct cxl_register_map ) { + .dev = &pdev->dev, + .resource =CXL_RESOURCE_NONE , + }; regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL ,CXL_DVSEC_REG_LOCATOR );drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 310
Quoted 2 lines
drivers/cxl/core/regs.c +1
-0+Remove this unrelated whitespace change.
drivers/cxl/core/regs.c int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, · line 336
Quoted 8 lines
drivers/cxl/core/regs.c +1
-0regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL ,CXL_DVSEC_REG_LOCATOR );if (!regloc)@@ -328,6 +333,7 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, } map->resource =CXL_RESOURCE_NONE ; +...and this one too, before Jonathan notices.
Quoted 56 lines
drivers/cxl/core/regs.c +0
-0return -ENODEV ; }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );drivers/cxl/cxl.h +6
-4@@ -231,6 +231,7 @@ struct cxl_device_reg_map {/** * struct cxl_register_map - DVSEC harvested register block mapping parameters +* @dev: device for devm operations and logging * @base: virtual base of the register-block-BAR + @block_offset * @resource: physical resource base of the register block * @max_size: maximum mapping size to perform register search @@ -239,6 +240,7 @@ struct cxl_device_reg_map {* @device_map: cxl_reg_maps for device registers */ struct cxl_register_map { +struct device *dev;void __iomem *base;resource_size_t resource;resource_size_t max_size;@@ -253,11 +255,11 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); -int cxl_map_component_regs (struct device *dev,struct cxl_component_regs *regs, -struct cxl_register_map *map, +int cxl_map_component_regs (struct cxl_register_map *map, +struct cxl_component_regs *regs,unsigned long map_mask); -int cxl_map_device_regs (struct device *dev,struct cxl_device_regs *regs, -struct cxl_register_map *map); +int cxl_map_device_regs (struct cxl_register_map *map, +struct cxl_device_regs *regs);enum cxl_regloc_type ; int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,drivers/cxl/pci.c +2
-2@@ -274,9 +274,9 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } -static int cxl_map_regblock (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_map_regblock (struct cxl_register_map *map) { -struct device *dev = &pdev->dev; +struct device *dev = map->dev; map->base =ioremap (map->resource, map->max_size);if (!map->base) {drivers/cxl/pci.c static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) · line 288
Quoted 4 lines
drivers/cxl/pci.c +1
-0@@ -285,21 +285,21 @@ static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) }dev_dbg (dev,"Mapped CXL Memory Device resource %pa\n" , &map->resource); +Ditto.
Quoted 60 lines
drivers/cxl/pci.c +9
-10return 0 ; } -static void cxl_unmap_regblock (struct pci_dev *pdev, -struct cxl_register_map *map) +static void cxl_unmap_regblock (struct cxl_register_map *map) {iounmap (map->base); map->base =NULL ; } -static int cxl_probe_regs (struct pci_dev *pdev,struct cxl_register_map *map) +static int cxl_probe_regs (struct cxl_register_map *map) {struct cxl_component_reg_map *comp_map;struct cxl_device_reg_map *dev_map; -struct device *dev = &pdev->dev; +struct device *dev = map->dev;void __iomem *base = map->base;switch (map->reg_type) {@@ -346,12 +346,12 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,if (rc)return rc; - rc =cxl_map_regblock (pdev, map); + rc =cxl_map_regblock (map);if (rc)return rc; - rc =cxl_probe_regs (pdev, map); -cxl_unmap_regblock (pdev, map); + rc =cxl_probe_regs (map); +cxl_unmap_regblock (map);return rc; }@@ -688,7 +688,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; - rc =cxl_map_device_regs (&pdev->dev, &cxlds->regs.device_regs, &map); + rc =cxl_map_device_regs (&map, &cxlds->regs.device_regs);if (rc)return rc;@@ -703,8 +703,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) cxlds->component_reg_phys = map.resource; - rc =cxl_map_component_regs (&pdev->dev, &cxlds->regs.component, - &map,BIT (CXL_CM_CAP_CAP_ID_RAS )); + rc =cxl_map_component_regs (&map, &cxlds->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ));if (rc)dev_dbg (&pdev->dev,"Failed to map RAS capability.\n" );Signature · 1 line
2.34.1
PATCH v5 6/26 cxl/pci: Refactor component register discovery for reuse
In reply to Terry BowmanThe endpoint implements component register setup code. Refactor it for reuse with RCRB, downstream port, and upstream port setup.
Move PCI specifics from cxl_setup_regs() into cxl_pci_setup_regs().
Move cxl_setup_regs() into cxl/core/regs.c and export it. This also includes supporting static functions cxl_map_registerblock(), cxl_unmap_register_block() and cxl_probe_regs().
--- drivers/cxl/core/regs.c | 78 ++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + drivers/cxl/pci.c | 80 +++-------------------------------------- 3 files changed, 84 insertions(+), 75 deletions(-)
drivers/cxl/core/regs.c +78 -0
@@ -338,6 +338,84 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, }EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL ); +static int cxl_map_regblock (struct cxl_register_map *map) +{ +struct device *dev = map->dev; + + map->base =ioremap (map->resource, map->max_size); +if (!map->base) { +dev_err (dev,"failed to map registers\n" ); +return -ENOMEM ; + } + +dev_dbg (dev,"Mapped CXL Memory Device resource %pa\n" , &map->resource); + +return 0 ; +} + +static void cxl_unmap_regblock (struct cxl_register_map *map) +{ +iounmap (map->base); + map->base =NULL ; +} + +static int cxl_probe_regs (struct cxl_register_map *map) +{ +struct cxl_component_reg_map *comp_map; +struct cxl_device_reg_map *dev_map; +struct device *dev = map->dev; +void __iomem *base = map->base; + +switch (map->reg_type) { +case CXL_REGLOC_RBI_COMPONENT : + comp_map = &map->component_map; +cxl_probe_component_regs (dev, base, comp_map); +if (!comp_map->hdm_decoder.valid) { +dev_err (dev,"HDM decoder registers not found\n" ); +return -ENXIO ; + } + +if (!comp_map->ras.valid) +dev_dbg (dev,"RAS registers not found\n" ); + +dev_dbg (dev,"Set up component registers\n" ); +break ; +case CXL_REGLOC_RBI_MEMDEV : + dev_map = &map->device_map; +cxl_probe_device_regs (dev, base, dev_map); +if (!dev_map->status.valid || !dev_map->mbox.valid || + !dev_map->memdev.valid) { +dev_err (dev,"registers not found: %s%s%s\n" , + !dev_map->status.valid ?"status " :"" , + !dev_map->mbox.valid ?"mbox " :"" , + !dev_map->memdev.valid ?"memdev " :"" ); +return -ENXIO ; + } + +dev_dbg (dev,"Probing device registers...\n" ); +break ; +default : +break ; + } + +return 0 ; +} + +int cxl_setup_regs (struct cxl_register_map *map) +{ +int rc; + + rc =cxl_map_regblock (map); +if (rc) +return rc; + + rc =cxl_probe_regs (map); +cxl_unmap_regblock (map); + +return rc; +} +EXPORT_SYMBOL_NS_GPL (cxl_setup_regs,CXL ); +resource_size_t cxl_probe_rcrb (struct device *dev,resource_size_t rcrb,struct cxl_rcrb_info *ri,enum cxl_rcrb which) {
drivers/cxl/cxl.h +1 -0
@@ -264,6 +264,7 @@ int cxl_map_device_regs(struct cxl_register_map *map,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); +int cxl_setup_regs (struct cxl_register_map *map);enum cxl_rcrb {CXL_RCRB_DOWNSTREAM ,
drivers/cxl/pci.c +5 -75
104 lines not on this page · raw message
Signature · 1 line
2.34.1
RE: [PATCH v5 06/26] cxl/pci: Refactor component register discovery for reuse
Quoted 8 lines · Terry Bowman
The endpoint implements component register setup code. Refactor it for reuse with RCRB, downstream port, and upstream port setup.
Move PCI specifics from
cxl_setup_regs()intocxl_pci_setup_regs().Move
cxl_setup_regs()intocxl/core/regs.cand export it. This also includes supporting static functionscxl_map_registerblock(),cxl_unmap_register_block()andcxl_probe_regs().
Looks ok to me. I hit some minor conflicts rebasing it on changes proposed earlier, but no other concerns from me.
PATCH v5 7/26 cxl/acpi: Moving add_host_bridge_uport() around
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Just moving code to reorder functions to later share cxl_get_chbs() with add_host_bridge_uport().
This makes changes in the next patch visible. No other changes at all.
--- drivers/cxl/acpi.c | 90 +++++++++++++++++++++++----------------------- 1 file changed, 45 insertions(+), 45 deletions(-)
drivers/cxl/acpi.c +45 -45
@@ -327,51 +327,6 @@ __mock struct acpi_device *to_cxl_host_bridge(struct device *host,return NULL ; } -/* -* A host bridge is a dport to a CFMWS decode and it is a uport to the -* dport (PCIe Root Ports) in the host bridge. -*/ -static int add_host_bridge_uport (struct device *match,void *arg) -{ -struct cxl_port *root_port = arg; -struct device *host = root_port->dev.parent; -struct acpi_device *hb =to_cxl_host_bridge (host, match); -struct acpi_pci_root *pci_root; -struct cxl_dport *dport; -struct cxl_port *port; -struct device *bridge; -int rc; - -if (!hb) -return 0 ; - - pci_root =acpi_pci_find_root (hb->handle); - bridge = pci_root->bus->bridge; - dport =cxl_find_dport_by_dev (root_port, bridge); -if (!dport) { -dev_dbg (host,"host bridge expected and not found\n" ); -return 0 ; - } - -if (dport->rch) { -dev_info (bridge,"host supports CXL (restricted)\n" ); -return 0 ; - } - - rc =devm_cxl_register_pci_bus (host, bridge, pci_root->bus); -if (rc) -return rc; - - port =devm_cxl_add_port (host, bridge, dport->component_reg_phys, - dport); -if (IS_ERR (port)) -return PTR_ERR (port); - -dev_info (bridge,"host supports CXL\n" ); - -return 0 ; -} -struct cxl_chbs_context {unsigned long long uid;resource_size_t base;
@@ -464,6 +419,51 @@ static int add_host_bridge_dport(struct device *match, void *arg)return 0 ; } +/* +* A host bridge is a dport to a CFMWS decode and it is a uport to the +* dport (PCIe Root Ports) in the host bridge. +*/ +static int add_host_bridge_uport (struct device *match,void *arg) +{ +struct cxl_port *root_port = arg; +struct device *host = root_port->dev.parent; +struct acpi_device *hb =to_cxl_host_bridge (host, match); +struct acpi_pci_root *pci_root; +struct cxl_dport *dport; +struct cxl_port *port; +struct device *bridge; +int rc; + +if (!hb) +return 0 ; + + pci_root =acpi_pci_find_root (hb->handle); + bridge = pci_root->bus->bridge; + dport =cxl_find_dport_by_dev (root_port, bridge); +if (!dport) { +dev_dbg (host,"host bridge expected and not found\n" ); +return 0 ; + } + +if (dport->rch) { +dev_info (bridge,"host supports CXL (restricted)\n" ); +return 0 ; + } + + rc =devm_cxl_register_pci_bus (host, bridge, pci_root->bus); +if (rc) +return rc; + + port =devm_cxl_add_port (host, bridge, dport->component_reg_phys, + dport); +if (IS_ERR (port)) +return PTR_ERR (port); + +dev_info (bridge,"host supports CXL\n" ); + +return 0 ; +} +static int add_root_nvdimm_bridge (struct device *match,void *data) {struct cxl_decoder *cxld;
Signature · 1 line
2.34.1
RE: [PATCH v5 07/26] cxl/acpi: Moving add_host_bridge_uport() around
Minor quibble with the gerund phrase and this not being clear about what it is doing. Just say:
"cxl/acpi: Move add_host_bridge_uport() after cxl_get_chbs()"
Quoted 6 lines · Terry Bowman
From: Robert Richter <[email protected]>
Just moving code to reorder functions to later share
cxl_get_chbs()withadd_host_bridge_uport().This makes changes in the next patch visible. No other changes at all.
Other than subject line, looks good.
Re: [PATCH v5 07/26] cxl/acpi: Moving add_host_bridge_uport() around
Thanks Dan. I'll update.
Regards,
Terry
Quoted 14 lines · Dan Williams
Minor quibble with the gerund phrase and this not being clear about what it is doing. Just say:
"cxl/acpi: Move
add_host_bridge_uport()aftercxl_get_chbs()"Quoted 6 lines · Terry Bowman
From: Robert Richter <[email protected]>
Just moving code to reorder functions to later share
cxl_get_chbs()withadd_host_bridge_uport().This makes changes in the next patch visible. No other changes at all.
Other than subject line, looks good.
PATCH v5 8/26 cxl/acpi: Directly bind the CEDT detected CHBCR to the Host Bridge's port
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
During a Host Bridge's downstream port enumeration the CHBS entries in the CEDT table are parsed, its Component Register base address extracted and then stored in struct cxl_dport. The CHBS may contain either the RCRB (RCH mode) or the Host Bridge's Component Registers (CHBCR, VH mode). The RCRB further contains the CXL downstream port register base address, while in VH mode the CXL Downstream Switch Ports are visible in the PCI hierarchy and the DP's component regs are disovered using the CXL DVSEC register locator capability. The Component Registers derived from the CHBS for both modes are different and thus also must be treated differently. That is, in RCH mode, the component regs base should be bound to the dport, but in VH mode to the CXL host bridge's port object.
The current implementation stores the CHBCR in addition in struct cxl_dport and copies it later from there to struct cxl_port. As a result, the dport contains the wrong Component Registers base address and, e.g. the RAS capability of a CXL Root Port cannot be detected.
To fix the CHBCR binding, attach it directly to the Host Bridge's @cxl_port structure. Do this during port creation of the Host Bridge in add_host_bridge_uport(). Factor out CHBS parsing code in add_host_bridge_dport() and use it in both functions.
--- drivers/cxl/acpi.c | 65 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 50 insertions(+), 15 deletions(-)
drivers/cxl/acpi.c +50 -15
@@ -333,8 +333,8 @@ struct cxl_chbs_context {u32 cxl_version; }; -static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs_iter (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs;
@@ -362,6 +362,22 @@ static int cxl_get_chbs(union acpi_subtable_headers *header, void *arg,return 0 ; } +static int cxl_get_chbs (struct acpi_device *hb,struct cxl_chbs_context *ctx) +{ +unsigned long long uid; +int rc; + + rc =acpi_evaluate_integer (hb->handle,METHOD_NAME__UID ,NULL , &uid); +if (rc !=AE_OK ) +return -ENOENT ; + +memset (ctx,0 ,sizeof (*ctx)); + ctx->uid = uid; +acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs_iter, ctx); + +return 0 ; +} +static int add_host_bridge_dport (struct device *match,void *arg) {acpi_status rc;
@@ -377,19 +393,15 @@ static int add_host_bridge_dport(struct device *match, void *arg)if (!hb)return 0 ; - rc =acpi_evaluate_integer (hb->handle,METHOD_NAME__UID ,NULL , &uid); -if (rc !=AE_OK ) { + rc =cxl_get_chbs (hb, &ctx); +if (rc == -ENOENT )dev_err (match,"unable to retrieve _UID\n" ); -return -ENODEV ; - } +if (rc) +return rc; + uid = ctx.uid;dev_dbg (match,"UID found: %lld\n" , uid); - ctx = (struct cxl_chbs_context ) { - .uid = uid, - }; -acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs, &ctx); -if (!ctx.base) {dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , uid);
@@ -405,12 +417,17 @@ static int add_host_bridge_dport(struct device *match, void *arg) pci_root =acpi_pci_find_root (hb->handle); bridge = pci_root->bus->bridge; +/* +* In RCH mode, bind the component regs base to the dport. In +* VH mode it will be bound to the CXL host bridge's port +* object later in add_host_bridge_uport(). +*/ if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) {dev_dbg (match,"RCRB found for UID %lld: %pa\n" , uid, &ctx.base); dport =devm_cxl_add_rch_dport (root_port, bridge, uid, ctx.base); }else { -dev_dbg (match,"CHBCR found for UID %lld: %pa\n" , uid, &ctx.base); - dport =devm_cxl_add_dport (root_port, bridge, uid, ctx.base); + dport =devm_cxl_add_dport (root_port, bridge, uid, +CXL_RESOURCE_NONE ); }if (IS_ERR (dport))
@@ -432,6 +449,8 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_dport *dport;struct cxl_port *port;struct device *bridge; +struct cxl_chbs_context ctx; +resource_size_t component_reg_phys;int rc;if (!hb)
@@ -450,12 +469,28 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } + rc =cxl_get_chbs (hb, &ctx); +if (rc) +return rc; + +if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) +/* RCH mode, should never happen */ +return 0 ; + +if (ctx.base) + component_reg_phys = ctx.base; +else + component_reg_phys =CXL_RESOURCE_NONE ; + +if (component_reg_phys !=CXL_RESOURCE_NONE ) +dev_dbg (match,"CHBCR found for UID %lld: %pa\n" , + ctx.uid, &component_reg_phys); + rc =devm_cxl_register_pci_bus (host, bridge, pci_root->bus);if (rc)return rc; - port =devm_cxl_add_port (host, bridge, dport->component_reg_phys, - dport); + port =devm_cxl_add_port (host, bridge, component_reg_phys, dport);if (IS_ERR (port))return PTR_ERR (port);
Signature · 1 line
2.34.1
RE: [PATCH v5 08/26] cxl/acpi: Directly bind the CEDT detected CHBCR to the Host Bridge's port
Quoted 25 lines · Terry Bowman
From: Robert Richter <[email protected]>
During a Host Bridge's downstream port enumeration the CHBS entries in the CEDT table are parsed, its Component Register base address extracted and then stored in struct
cxl_dport. The CHBS may contain either the RCRB (RCH mode) or the Host Bridge's Component Registers (CHBCR, VH mode). The RCRB further contains the CXL downstream port register base address, while in VH mode the CXL Downstream Switch Ports are visible in the PCI hierarchy and the DP's component regs are disovered using the CXL DVSEC register locator capability. The Component Registers derived from the CHBS for both modes are different and thus also must be treated differently. That is, in RCH mode, the component regs base should be bound to the dport, but in VH mode to the CXL host bridge's port object.The current implementation stores the CHBCR in addition in struct
cxl_dportand copies it later from there to structcxl_port. As a result, the dport contains the wrong Component Registers base address and, e.g. the RAS capability of a CXL Root Port cannot be detected.To fix the CHBCR binding, attach it directly to the Host Bridge's @cxl_port structure. Do this during port creation of the Host Bridge in
add_host_bridge_uport(). Factor out CHBS parsing code inadd_host_bridge_dport()and use it in both functions.
Quoted 6 lines
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/acpi.c | 65 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 50 insertions(+), 15 deletions(-)
I like where this patch is going. I needed to rebase and fix it up to get cxl_test working again, and spotted some additional cleanups along the way. I notice that after this patch dport->component_reg_phys is totally unused. So I would like to see that removal patch comes next in the series before going into the port->component_reg_phys reworks.
-- >8 --
drivers/cxl/acpi.c +29 -28
@@ -341,7 +341,7 @@ static int cxl_get_chbs_iter(union acpi_subtable_headers *header, void *arg,struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->base) +if (ctx->base !=CXL_RESOURCE_NONE )return 0 ; chbs = (struct acpi_cedt_chbs *) header;
@@ -350,8 +350,6 @@ static int cxl_get_chbs_iter(union acpi_subtable_headers *header, void *arg,return 0 ; ctx->cxl_version = chbs->cxl_version; - ctx->base =CXL_RESOURCE_NONE ; -if (!chbs->base)return 0 ;
@@ -364,17 +362,26 @@ static int cxl_get_chbs_iter(union acpi_subtable_headers *header, void *arg,return 0 ; } -static int cxl_get_chbs (struct acpi_device *hb,struct cxl_chbs_context *ctx) +static int cxl_get_chbs (struct device *dev,struct acpi_device *hb, +struct cxl_chbs_context *ctx) {unsigned long long uid;int rc; rc =acpi_evaluate_integer (hb->handle,METHOD_NAME__UID ,NULL , &uid); -if (rc !=AE_OK ) +if (rc !=AE_OK ) { +dev_err (dev,"unable to retrieve _UID\n" );return -ENOENT ; + } + +dev_dbg (dev,"UID found: %lld\n" , uid); + *ctx = (struct cxl_chbs_context ) { + .dev = dev, + .uid = uid, + .base =CXL_RESOURCE_NONE , + .cxl_version =UINT_MAX , + }; -memset (ctx,0 ,sizeof (*ctx)); - ctx->uid = uid;acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs_iter, ctx);return 0 ;
@@ -384,7 +391,6 @@ static int add_host_bridge_dport(struct device *match, void *arg) {acpi_status rc;struct device *bridge; -unsigned long long uid;struct cxl_dport *dport;struct cxl_chbs_context ctx;struct acpi_pci_root *pci_root;
@@ -395,24 +401,19 @@ static int add_host_bridge_dport(struct device *match, void *arg)if (!hb)return 0 ; - rc =cxl_get_chbs (hb, &ctx); -if (rc == -ENOENT ) -dev_err (match,"unable to retrieve _UID\n" ); + rc =cxl_get_chbs (match, hb, &ctx);if (rc)return rc; - uid = ctx.uid; -dev_dbg (match,"UID found: %lld\n" , uid); - -if (!ctx.base) { +if (ctx.cxl_version ==UINT_MAX ) {dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , - uid); + ctx.uid);return 0 ; }if (ctx.base ==CXL_RESOURCE_NONE ) {dev_warn (match,"CHBS invalid for Host Bridge (UID %lld)\n" , - uid); + ctx.uid);return 0 ; }
@@ -425,10 +426,12 @@ static int add_host_bridge_dport(struct device *match, void *arg)* object later in add_host_bridge_uport(). */ if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) { -dev_dbg (match,"RCRB found for UID %lld: %pa\n" , uid, &ctx.base); - dport =devm_cxl_add_rch_dport (root_port, bridge, uid, ctx.base); +dev_dbg (match,"RCRB found for UID %lld: %pa\n" , ctx.uid, + &ctx.base); + dport =devm_cxl_add_rch_dport (root_port, bridge, ctx.uid, + ctx.base); }else { - dport =devm_cxl_add_dport (root_port, bridge, uid, + dport =devm_cxl_add_dport (root_port, bridge, ctx.uid,CXL_RESOURCE_NONE ); }
@@ -471,19 +474,17 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } - rc =cxl_get_chbs (hb, &ctx); + rc =cxl_get_chbs (match, hb, &ctx);if (rc)return rc; -if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) -/* RCH mode, should never happen */ +if (ctx.cxl_version ==ACPI_CEDT_CHBS_VERSION_CXL11 ) { +dev_warn (bridge, +"CXL CHBS version mismatch, skip port registration\n" );return 0 ; + } -if (ctx.base) - component_reg_phys = ctx.base; -else - component_reg_phys =CXL_RESOURCE_NONE ; - + component_reg_phys = ctx.base;if (component_reg_phys !=CXL_RESOURCE_NONE )dev_dbg (match,"CHBCR found for UID %lld: %pa\n" , ctx.uid, &component_reg_phys);
PATCH v5 9/26 cxl/regs: Remove early capability checks in Component Register setup
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
When probing the Component Registers in function cxl_probe_regs() there are also checks for the existence of the HDM and RAS capabilities. The checks may fail for components that do not implement the HDM capability causing the Component Registers setup to fail too.
Remove the checks for a generalized use of cxl_probe_regs() and check them directly before mapping the RAS or HDM capabilities. This allows it to setup other Component Registers esp. of an RCH Downstream Port, which will be implemented in a follow-on patch.
--- drivers/cxl/core/regs.c | 8 -------- drivers/cxl/pci.c | 2 ++ drivers/cxl/port.c | 5 ++++- 3 files changed, 6 insertions(+), 9 deletions(-)
drivers/cxl/core/regs.c +0 -8
@@ -370,14 +370,6 @@ static int cxl_probe_regs(struct cxl_register_map *map)case CXL_REGLOC_RBI_COMPONENT : comp_map = &map->component_map;cxl_probe_component_regs (dev, base, comp_map); -if (!comp_map->hdm_decoder.valid) { -dev_err (dev,"HDM decoder registers not found\n" ); -return -ENXIO ; - } - -if (!comp_map->ras.valid) -dev_dbg (dev,"RAS registers not found\n" ); -dev_dbg (dev,"Set up component registers\n" );break ;case CXL_REGLOC_RBI_MEMDEV :
drivers/cxl/pci.c +2 -0
@@ -630,6 +630,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) rc =cxl_pci_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &map);if (rc)dev_warn (&pdev->dev,"No component registers (%d)\n" , rc); +else if (!map.component_map.ras.valid) +dev_dbg (&pdev->dev,"RAS registers not found\n" ); cxlds->component_reg_phys = map.resource;
drivers/cxl/port.c +4 -1
@@ -102,8 +102,11 @@ static int cxl_endpoint_port_probe(struct cxl_port *port)return rc; cxlhdm =devm_cxl_setup_hdm (port, &info); -if (IS_ERR (cxlhdm)) +if (IS_ERR (cxlhdm)) { +if (PTR_ERR (cxlhdm) == -ENODEV ) +dev_err (&port->dev,"HDM decoder registers not found\n" );return PTR_ERR (cxlhdm); + }/* Cache the data early to ensure is_visible() works */ read_cdat_data (port);
Signature · 1 line
2.34.1
RE: [PATCH v5 09/26] cxl/regs: Remove early capability checks in Component Register setup
Quoted 7 lines · Terry Bowman
From: Robert Richter <[email protected]>
When probing the Component Registers in function
cxl_probe_regs()there are also checks for the existence of the HDM and RAS capabilities. The checks may fail for components that do not implement the HDM capability causing the Component Registers setup to fail too.
Quoted 8 lines
Remove the checks for a generalized use of
cxl_probe_regs()and check them directly before mapping the RAS or HDM capabilities. This allows it to setup other Component Registers esp. of an RCH Downstream Port, which will be implemented in a follow-on patch.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Looks good.
PATCH v5 10/26 cxl/mem: Prepare for early RCH dport component register setup
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
In order to move the RCH dport component register setup to cxl_pci the base address must be stored in CXL device state (cxlds) for both modes, RCH and VH. Store it in cxlds->component_reg_phys and use it for endpoint creation.
--- drivers/cxl/mem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +4 -5
@@ -51,7 +51,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,struct cxl_port *parent_port = parent_dport->port;struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_port *endpoint, *iter, *down; -resource_size_t component_reg_phys;int rc;/*
@@ -72,12 +71,12 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,* typical register locator mechanism. */ if (parent_dport->rch && cxlds->component_reg_phys ==CXL_RESOURCE_NONE ) - component_reg_phys = + cxlds->component_reg_phys =cxl_probe_rcrb (&cxlmd->dev, parent_dport->rcrb.base,NULL ,CXL_RCRB_UPSTREAM ); -else - component_reg_phys = cxlds->component_reg_phys; - endpoint =devm_cxl_add_port (host, &cxlmd->dev, component_reg_phys, + + endpoint =devm_cxl_add_port (host, &cxlmd->dev, + cxlds->component_reg_phys, parent_dport);if (IS_ERR (endpoint))return PTR_ERR (endpoint);
Signature · 1 line
2.34.1
RE: [PATCH v5 10/26] cxl/mem: Prepare for early RCH dport component register setup
Quoted 6 lines · Terry Bowman
From: Robert Richter <[email protected]>
In order to move the RCH dport component register setup to
cxl_pcithe base address must be stored in CXL device state (cxlds) for both modes, RCH and VH. Store it incxlds->component_reg_physand use it for endpoint creation.
Looks good, I just needed to rebase it to usecxl_rcd_component_reg_phys().
PATCH v5 11/26 cxl/pci: Early setup RCH dport component registers from RCRB
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
CXL RAS capabilities must be enabled and accessible as soon as the CXL endpoint is detected in the PCI hierarchy and bound to the cxl_pci driver. This needs to be independent of other modules such as cxl_port or cxl_mem.
CXL RAS capabilities reside in the Component Registers. For an RCH this is determined by probing RCRB which is implemented very late once the CXL Memory Device is created.
Change this by moving the RCRB probe to the cxl_pci driver. Do this by using a new introduced function cxl_pci_find_port() similar to cxl_mem_find_port() to determine the involved dport by the endpoint's PCI handle. Plug this into the existing cxl_pci_setup_regs() function to setup Component Registers. Probe the RCRB in case the Component Registers cannot be located through the CXL Register Locator capability.
This unifies code and early sets up the Component Registers at the same time for both, VH and RCH mode. Only the cxl_pci driver is involved for this. This allows an early mapping of the CXL RAS capability registers.
--- drivers/cxl/core/port.c | 7 +++++++ drivers/cxl/cxl.h | 2 ++ drivers/cxl/mem.c | 10 ---------- drivers/cxl/pci.c | 37 ++++++++++++++++++++++++++++++++++++- 4 files changed, 45 insertions(+), 11 deletions(-)
drivers/cxl/core/port.c +7 -0
@@ -1476,6 +1476,13 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd) }EXPORT_SYMBOL_NS_GPL (devm_cxl_enumerate_ports,CXL ); +struct cxl_port *cxl_pci_find_port (struct pci_dev *pdev, +struct cxl_dport **dport) +{ +return find_cxl_port (pdev->dev.parent, dport); +} +EXPORT_SYMBOL_NS_GPL (cxl_pci_find_port,CXL ); +struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd,struct cxl_dport **dport) {
drivers/cxl/cxl.h +2 -0
@@ -670,6 +670,8 @@ struct cxl_port *find_cxl_root(struct cxl_port *port);int devm_cxl_enumerate_ports (struct cxl_memdev *cxlmd);void cxl_bus_rescan (void );void cxl_bus_drain (void ); +struct cxl_port *cxl_pci_find_port (struct pci_dev *pdev, +struct cxl_dport **dport);struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd,struct cxl_dport **dport);bool schedule_cxl_memdev_detach (struct cxl_memdev *cxlmd);
drivers/cxl/mem.c +0 -10
@@ -65,16 +65,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, ep->next = down; } -/* -* The component registers for an RCD might come from the -* host-bridge RCRB if they are not already mapped via the -* typical register locator mechanism. -*/ -if (parent_dport->rch && cxlds->component_reg_phys ==CXL_RESOURCE_NONE ) - cxlds->component_reg_phys = -cxl_probe_rcrb (&cxlmd->dev, parent_dport->rcrb.base, -NULL ,CXL_RCRB_UPSTREAM ); - endpoint =devm_cxl_add_port (host, &cxlmd->dev, cxlds->component_reg_phys, parent_dport);
drivers/cxl/pci.c +36 -1
@@ -274,13 +274,48 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +/* Extract RCRB, use same function interface as cxl_find_regblock(). */ +static int cxl_rcrb_get_comp_regs (struct pci_dev *pdev, +enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +struct cxl_dport *dport; +resource_size_t component_reg_phys; + +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; + map->resource =CXL_RESOURCE_NONE ; + +if (type !=CXL_REGLOC_RBI_COMPONENT ) +return -ENODEV ; + +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return -ENXIO ; + + component_reg_phys =cxl_probe_rcrb (&pdev->dev, dport->rcrb.base, +NULL ,CXL_RCRB_UPSTREAM ); +if (component_reg_phys ==CXL_RESOURCE_NONE ) +return -ENXIO ; + + map->resource = component_reg_phys; + map->reg_type = type; + map->max_size =CXL_COMPONENT_REG_BLOCK_SIZE ; + +return 0 ; +} +static int cxl_pci_setup_regs (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map) {int rc; +/* +* If the Register Locator DVSEC does not contain the +* Component Registers, assume it is an RCH and try to extract +* them from an RCRB. +*/ rc =cxl_find_regblock (pdev, type, map); -if (rc) +if (rc &&cxl_rcrb_get_comp_regs (pdev, type, map))return rc;return cxl_setup_regs (map);
Signature · 1 line
2.34.1
RE: [PATCH v5 11/26] cxl/pci: Early setup RCH dport component registers from RCRB
Quoted 19 lines · Terry Bowman
From: Robert Richter <[email protected]>
CXL RAS capabilities must be enabled and accessible as soon as the CXL endpoint is detected in the PCI hierarchy and bound to the
cxl_pcidriver. This needs to be independent of other modules such ascxl_portorcxl_mem.CXL RAS capabilities reside in the Component Registers. For an RCH this is determined by probing RCRB which is implemented very late once the CXL Memory Device is created.
Change this by moving the RCRB probe to the
cxl_pcidriver. Do this by using a new introduced functioncxl_pci_find_port()similar tocxl_mem_find_port()to determine the involved dport by the endpoint's PCI handle. Plug this into the existingcxl_pci_setup_regs()function to setup Component Registers. Probe the RCRB in case the Component Registers cannot be located through the CXL Register Locator capability.
Quoted 4 lines
This unifies code and early sets up the Component Registers at the same time for both, VH and RCH mode. Only the
cxl_pcidriver is involved for this. This allows an early mapping of the CXL RAS capability registers.
This is problematic because it creates an implicit dependency between cxl_acpi and cxl_pci. More comments below:
Quoted 75 lines
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/port.c | 7 +++++++ drivers/cxl/cxl.h | 2 ++ drivers/cxl/mem.c | 10 ---------- drivers/cxl/pci.c | 37 ++++++++++++++++++++++++++++++++++++- 4 files changed, 45 insertions(+), 11 deletions(-)drivers/cxl/core/port.c +7
-0@@ -1476,6 +1476,13 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd) }EXPORT_SYMBOL_NS_GPL (devm_cxl_enumerate_ports,CXL ); +struct cxl_port *cxl_pci_find_port (struct pci_dev *pdev, +struct cxl_dport **dport) +{ +return find_cxl_port (pdev->dev.parent, dport); +} +EXPORT_SYMBOL_NS_GPL (cxl_pci_find_port,CXL ); +struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd,struct cxl_dport **dport) {drivers/cxl/cxl.h +2
-0@@ -670,6 +670,8 @@ struct cxl_port *find_cxl_root(struct cxl_port *port);int devm_cxl_enumerate_ports (struct cxl_memdev *cxlmd);void cxl_bus_rescan (void );void cxl_bus_drain (void ); +struct cxl_port *cxl_pci_find_port (struct pci_dev *pdev, +struct cxl_dport **dport);struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd,struct cxl_dport **dport);bool schedule_cxl_memdev_detach (struct cxl_memdev *cxlmd);drivers/cxl/mem.c +0
-10@@ -65,16 +65,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, ep->next = down; } -/* -* The component registers for an RCD might come from the -* host-bridge RCRB if they are not already mapped via the -* typical register locator mechanism. -*/ -if (parent_dport->rch && cxlds->component_reg_phys ==CXL_RESOURCE_NONE ) - cxlds->component_reg_phys = -cxl_probe_rcrb (&cxlmd->dev, parent_dport->rcrb.base, -NULL ,CXL_RCRB_UPSTREAM ); - endpoint =devm_cxl_add_port (host, &cxlmd->dev, cxlds->component_reg_phys, parent_dport);drivers/cxl/pci.c +4
-0@@ -274,13 +274,48 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +/* Extract RCRB, use same function interface as cxl_find_regblock(). */ +static int cxl_rcrb_get_comp_regs (struct pci_dev *pdev, +enum cxl_regloc_type type, +struct cxl_register_map *map)
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 290
Quoted 10 lines
drivers/cxl/pci.c +10
-0+{ +struct cxl_dport *dport; +resource_size_t component_reg_phys; + +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; + map->resource =CXL_RESOURCE_NONE ; + +if (type !=CXL_REGLOC_RBI_COMPONENT ) +return -ENODEV ;
Why would a function called cxl_rcrb_get_comp_regs() need a type parameter that only takes one acceptable value. I would drop the parameter and move this distinction to the caller.
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 293
Quoted 3 lines
drivers/cxl/pci.c +3
-0+ +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return -ENXIO ;
This should return -EPROBE_DEFER in the !cxl_pci_find_port() case to try to await cxl_acpi initialization.
Quoted 17 lines
drivers/cxl/pci.c +13
-0+ + component_reg_phys =cxl_probe_rcrb (&pdev->dev, dport->rcrb.base, +NULL ,CXL_RCRB_UPSTREAM ); +if (component_reg_phys ==CXL_RESOURCE_NONE ) +return -ENXIO ; + + map->resource = component_reg_phys; + map->reg_type = type; + map->max_size =CXL_COMPONENT_REG_BLOCK_SIZE ; + +return 0 ; +} +static int cxl_pci_setup_regs (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map) {int rc;
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 319
Quoted 10 lines
drivers/cxl/pci.c +6
-1+/* +* If the Register Locator DVSEC does not contain the +* Component Registers, assume it is an RCH and try to extract +* them from an RCRB. +*/ rc =cxl_find_regblock (pdev, type, map); -if (rc) +if (rc &&cxl_rcrb_get_comp_regs (pdev, type, map))return rc;
I am not a big fan of just assuming this is an RCD, especially when there is the is_cxl_restricted() helper just beneath this.
RE: [PATCH v5 11/26] cxl/pci: Early setup RCH dport component registers from RCRB
In reply to Terry BowmanQuoted 34 lines · Terry Bowman
From: Robert Richter <[email protected]>
CXL RAS capabilities must be enabled and accessible as soon as the CXL endpoint is detected in the PCI hierarchy and bound to the
cxl_pcidriver. This needs to be independent of other modules such ascxl_portorcxl_mem.CXL RAS capabilities reside in the Component Registers. For an RCH this is determined by probing RCRB which is implemented very late once the CXL Memory Device is created.
Change this by moving the RCRB probe to the
cxl_pcidriver. Do this by using a new introduced functioncxl_pci_find_port()similar tocxl_mem_find_port()to determine the involved dport by the endpoint's PCI handle. Plug this into the existingcxl_pci_setup_regs()function to setup Component Registers. Probe the RCRB in case the Component Registers cannot be located through the CXL Register Locator capability.This unifies code and early sets up the Component Registers at the same time for both, VH and RCH mode. Only the
cxl_pcidriver is involved for this. This allows an early mapping of the CXL RAS capability registers.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/port.c | 7 +++++++ drivers/cxl/cxl.h | 2 ++ drivers/cxl/mem.c | 10 ---------- drivers/cxl/pci.c | 37 ++++++++++++++++++++++++++++++++++++- 4 files changed, 45 insertions(+), 11 deletions(-)
[..]
Quoted 24 lines
drivers/cxl/pci.c +16
-0@@ -274,13 +274,48 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +/* Extract RCRB, use same function interface as cxl_find_regblock(). */ +static int cxl_rcrb_get_comp_regs (struct pci_dev *pdev, +enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +struct cxl_dport *dport; +resource_size_t component_reg_phys; + +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; + map->resource =CXL_RESOURCE_NONE ; + +if (type !=CXL_REGLOC_RBI_COMPONENT ) +return -ENODEV ; + +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch)
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 302
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return -ENXIO ; + + component_reg_phys =cxl_probe_rcrb (&pdev->dev, dport->rcrb.base, +NULL ,CXL_RCRB_UPSTREAM ); +if (component_reg_phys ==CXL_RESOURCE_NONE ) +return -ENXIO ; + + map->resource = component_reg_phys; + map->reg_type = type; + map->max_size =CXL_COMPONENT_REG_BLOCK_SIZE ;
One more note, I would prefer a designated initializer for this.
RE: [PATCH v5 11/26] cxl/pci: Early setup RCH dport component registers from RCRB
In reply to Terry BowmanQuoted 27 lines · Terry Bowman
From: Robert Richter <[email protected]>
CXL RAS capabilities must be enabled and accessible as soon as the CXL endpoint is detected in the PCI hierarchy and bound to the
cxl_pcidriver. This needs to be independent of other modules such ascxl_portorcxl_mem.CXL RAS capabilities reside in the Component Registers. For an RCH this is determined by probing RCRB which is implemented very late once the CXL Memory Device is created.
Change this by moving the RCRB probe to the
cxl_pcidriver. Do this by using a new introduced functioncxl_pci_find_port()similar tocxl_mem_find_port()to determine the involved dport by the endpoint's PCI handle. Plug this into the existingcxl_pci_setup_regs()function to setup Component Registers. Probe the RCRB in case the Component Registers cannot be located through the CXL Register Locator capability.This unifies code and early sets up the Component Registers at the same time for both, VH and RCH mode. Only the
cxl_pcidriver is involved for this. This allows an early mapping of the CXL RAS capability registers.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
[..]
Quoted 15 lines
drivers/cxl/pci.c +7
-0@@ -274,13 +274,48 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +/* Extract RCRB, use same function interface as cxl_find_regblock(). */ +static int cxl_rcrb_get_comp_regs (struct pci_dev *pdev, +enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +struct cxl_dport *dport; +resource_size_t component_reg_phys;
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 293
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ +memset (map,0 ,sizeof (*map)); + map->dev = &pdev->dev; + map->resource =CXL_RESOURCE_NONE ; + +if (type !=CXL_REGLOC_RBI_COMPONENT ) +return -ENODEV ; + +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return -ENXIO ;
I noticed while reviewing a later patch that this pattern leaks the port reference from cxl_pci_find_port().
I.e. this needs to do:
port = cxl_pci_find_port(...); if (!port) return -ENXIO; if (!dport->rch) goto out; ... out: put_device(&port->dev);
PATCH v5 12/26 cxl/port: Store the port's Component Register mappings in struct cxl_port
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
CXL capabilities are stored in the Component Registers. To use them, the specific I/O ranges of the capabilities must be determined by probing the registers. For this, the whole Component Register range needs to be mapped temporarily to detect the offset and length of a capability range.
In order to use more than one capability of a component (e.g. RAS and HDM) the Component Register are probed and its mappings created multiple times. This also causes overlapping I/O ranges as the whole Component Register range must be mapped again while a capability's I/O range is already mapped.
Different capabilities cannot be setup at the same time. E.g. the RAS capability must be made available as soon as the PCI driver is bound, the HDM decoder is setup later during port enumeration. Moreover, during early setup it is still unknown if a certain capability is needed. A central capability setup is therefore not possible, capabilities must be individually enabled once needed during initialization.
To avoid a duplicate register probe and overlapping I/O mappings, only probe the Component Registers one time and store the Component Register mapping in struct port. The stored mappings can be used later to iomap the capability register range when enabling the capability, which will be implemented in a follow-on patch.
--- drivers/cxl/core/port.c | 33 +++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 35 insertions(+)
drivers/cxl/core/port.c +33 -0
@@ -686,6 +686,28 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,return ERR_PTR (rc); } +static int cxl_setup_comp_regs (struct device *dev,struct cxl_register_map *map, +resource_size_t component_reg_phys) +{ +if (component_reg_phys ==CXL_RESOURCE_NONE ) +return -ENODEV ; + +memset (map,0 ,sizeof (*map)); + map->dev = dev; + map->reg_type =CXL_REGLOC_RBI_COMPONENT ; + map->resource = component_reg_phys; + map->max_size =CXL_COMPONENT_REG_BLOCK_SIZE ; + +return cxl_setup_regs (map); +} + +static inline int cxl_port_setup_regs (struct cxl_port *port, +resource_size_t component_reg_phys) +{ +return cxl_setup_comp_regs (&port->dev, &port->comp_map, + component_reg_phys); +} + static struct cxl_port *__devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys,
@@ -709,6 +731,17 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,if (rc) goto err; +/* +* Some components may not use capablities or their +* implementation is optional. A component register block may +* not be present then and component_reg_phys is therefore +* unset. Instead run the check later when setting up the +* capabilities. +*/ + rc =cxl_port_setup_regs (port, component_reg_phys); +if (rc && rc != -ENODEV ) + goto err; + rc =device_add (dev);if (rc) goto err;
drivers/cxl/cxl.h +2 -0
@@ -552,6 +552,7 @@ struct cxl_dax_region {* @regions: cxl_region_ref instances, regions mapped by this port * @parent_dport: dport that points to this port in the parent * @decoder_ida: allocator for decoder ids +* @comp_map: component register capability mappings * @nr_dports: number of entries in @dports * @hdm_end: track last allocated HDM decoder instance for allocation ordering * @commit_end: cursor to track highest committed decoder for commit ordering
@@ -571,6 +572,7 @@ struct cxl_port {struct xarray regions;struct cxl_dport *parent_dport;struct ida decoder_ida; +struct cxl_register_map comp_map;int nr_dports;int hdm_end;int commit_end;
Signature · 1 line
2.34.1
RE: [PATCH v5 12/26] cxl/port: Store the port's Component Register mappings in struct cxl_port
Quoted 72 lines · Terry Bowman
From: Robert Richter <[email protected]>
CXL capabilities are stored in the Component Registers. To use them, the specific I/O ranges of the capabilities must be determined by probing the registers. For this, the whole Component Register range needs to be mapped temporarily to detect the offset and length of a capability range.
In order to use more than one capability of a component (e.g. RAS and HDM) the Component Register are probed and its mappings created multiple times. This also causes overlapping I/O ranges as the whole Component Register range must be mapped again while a capability's I/O range is already mapped.
Different capabilities cannot be setup at the same time. E.g. the RAS capability must be made available as soon as the PCI driver is bound, the HDM decoder is setup later during port enumeration. Moreover, during early setup it is still unknown if a certain capability is needed. A central capability setup is therefore not possible, capabilities must be individually enabled once needed during initialization.
To avoid a duplicate register probe and overlapping I/O mappings, only probe the Component Registers one time and store the Component Register mapping in struct port. The stored mappings can be used later to iomap the capability register range when enabling the capability, which will be implemented in a follow-on patch.
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/port.c | 33 +++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 35 insertions(+)drivers/cxl/core/port.c +22
-0@@ -686,6 +686,28 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,return ERR_PTR (rc); } +static int cxl_setup_comp_regs (struct device *dev,struct cxl_register_map *map, +resource_size_t component_reg_phys) +{ +if (component_reg_phys ==CXL_RESOURCE_NONE ) +return -ENODEV ; + +memset (map,0 ,sizeof (*map)); + map->dev = dev; + map->reg_type =CXL_REGLOC_RBI_COMPONENT ; + map->resource = component_reg_phys; + map->max_size =CXL_COMPONENT_REG_BLOCK_SIZE ; + +return cxl_setup_regs (map); +} + +static inline int cxl_port_setup_regs (struct cxl_port *port, +resource_size_t component_reg_phys) +{ +return cxl_setup_comp_regs (&port->dev, &port->comp_map, + component_reg_phys); +} + static struct cxl_port *__devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys,@@ -709,6 +731,17 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,if (rc) goto err;
drivers/cxl/core/port.c static struct cxl_port *__devm_cxl_add_port(struct device *host, · line 743
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+/* +* Some components may not use capablities or their +* implementation is optional. A component register block may +* not be present then and component_reg_phys is therefore +* unset. Instead run the check later when setting up the +* capabilities. +*/ + rc =cxl_port_setup_regs (port, component_reg_phys); +if (rc && rc != -ENODEV ) + goto err;
I would prefer to unwrap some of the levels and just do:
port->comp_map = (struct cxl_register_map) {
.dev = &port->dev;
.reg_type = CXL_REGLOC_RBI_COMPONENT;
.resource = component_reg_phys;
.max_size = CXL_COMPONENT_REG_BLOCK_SIZE;
};
if (component_reg_phys != CXL_RESOURCE_NONE) {
rc = cxl_setup_regs(&port->comp_map);
if (rc)
goto err;
}...earlier inside of cxl_port_alloc() since that part is common to all ports. Then the confusing comment becomes unnecessary because it is clear that CXL_RESOURCE_NONE is an acceptable case.
Other than that, the approach and rationale looks good.
PATCH v5 13/26 cxl/port: Store the downstream port's Component Register mappings in struct cxl_dport
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Same as for ports, also store the downstream port's Component Register mappings, use struct cxl_dport for that.
--- drivers/cxl/core/port.c | 11 +++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 13 insertions(+)
drivers/cxl/core/port.c +11 -0
@@ -708,6 +708,13 @@ static inline int cxl_port_setup_regs(struct cxl_port *port, component_reg_phys); } +static inline int cxl_dport_setup_regs (struct cxl_dport *dport, +resource_size_t component_reg_phys) +{ +return cxl_setup_comp_regs (dport->dev, &dport->comp_map, + component_reg_phys); +} + static struct cxl_port *__devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys,
@@ -992,6 +999,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, dport->component_reg_phys = component_reg_phys; dport->port = port; + rc =cxl_dport_setup_regs (dport, component_reg_phys); +if (rc && rc != -ENODEV ) +return ERR_PTR (rc); +cond_cxl_root_lock (port); rc =add_dport (port, dport);cond_cxl_root_unlock (port);
drivers/cxl/cxl.h +2 -0
@@ -601,6 +601,7 @@ struct cxl_rcrb_info {* struct cxl_dport - CXL downstream port * @dev: PCI bridge or firmware device representing the downstream link * @port: reference to cxl_port that contains this downstream port +* @comp_map: component register capability mappings * @port_id: unique hardware identifier for dport in decoder target list * @component_reg_phys: downstream port component registers * @rch: Indicate whether this dport was enumerated in RCH or VH mode
@@ -609,6 +610,7 @@ struct cxl_rcrb_info {struct cxl_dport {struct device *dev;struct cxl_port *port; +struct cxl_register_map comp_map;int port_id;resource_size_t component_reg_phys;bool rch;
Signature · 1 line
2.34.1
RE: [PATCH v5 13/26] cxl/port: Store the downstream port's Component Register mappings in struct cxl_dport
Quoted 32 lines · Terry Bowman
From: Robert Richter <[email protected]>
Same as for ports, also store the downstream port's Component Register mappings, use struct
cxl_dportfor that.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/port.c | 11 +++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 13 insertions(+)drivers/cxl/core/port.c +7
-0@@ -708,6 +708,13 @@ static inline int cxl_port_setup_regs(struct cxl_port *port, component_reg_phys); } +static inline int cxl_dport_setup_regs (struct cxl_dport *dport, +resource_size_t component_reg_phys) +{ +return cxl_setup_comp_regs (dport->dev, &dport->comp_map, + component_reg_phys); +} + static struct cxl_port *__devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys,
drivers/cxl/core/port.c __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, · line 1004
Quoted 6 lines
drivers/cxl/core/port.c +3
-0@@ -992,6 +999,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, dport->component_reg_phys = component_reg_phys; dport->port = port; + rc =cxl_dport_setup_regs (dport, component_reg_phys); +if (rc && rc != -ENODEV ) +return ERR_PTR (rc);
Ah I see that you wanted to share a similar function between this case and the last patch, but I would still remove one layer of indirection and make the setup return 0 if there is nothing to do rather than the
if (rc && rc != -ENODEV)
...the stresses out the reader wondering where that special error code case is generated.
Otherwise, makes sense.
PATCH v5 14/26 cxl/pci: Store the endpoint's Component Register mappings in struct cxl_dev_state
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Same as for ports and dports, also store the endpoint's Component Register mappings, use struct cxl_dev_state for that.
--- drivers/cxl/cxlmem.h | 3 ++- drivers/cxl/pci.c | 9 +++++---- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/cxl/cxlmem.h +2 -1
@@ -263,6 +263,7 @@ struct cxl_poison_state {* * @dev: The device associated with this CXL state * @cxlmd: The device representing the CXL.mem capabilities of @dev +* @comp_map: component register capability mappings * @regs: Parsed register blocks * @cxl_dvsec: Offset to the PCIe device DVSEC * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
@@ -299,7 +300,7 @@ struct cxl_poison_state {struct cxl_dev_state {struct device *dev;struct cxl_memdev *cxlmd; - +struct cxl_register_map comp_map;struct cxl_regs regs;int cxl_dvsec;
drivers/cxl/pci.c +5 -4
@@ -662,15 +662,16 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)* still be useful for management functions so don't return an error. */ cxlds->component_reg_phys =CXL_RESOURCE_NONE ; - rc =cxl_pci_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &map); + rc =cxl_pci_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , + &cxlds->comp_map);if (rc)dev_warn (&pdev->dev,"No component registers (%d)\n" , rc); -else if (!map.component_map.ras.valid) +else if (!cxlds->comp_map.component_map.ras.valid)dev_dbg (&pdev->dev,"RAS registers not found\n" ); - cxlds->component_reg_phys = map.resource; + cxlds->component_reg_phys = cxlds->comp_map.resource; - rc =cxl_map_component_regs (&map, &cxlds->regs.component, + rc =cxl_map_component_regs (&cxlds->comp_map, &cxlds->regs.component,BIT (CXL_CM_CAP_CAP_ID_RAS ));if (rc)dev_dbg (&pdev->dev,"Failed to map RAS capability.\n" );
Signature · 1 line
2.34.1
Concept of LD-ID in QEMU
Hi,
For DCD sideband there needs to be LD-ID. Is the following approach acceptable?
-device cxl-type3,bus=swport0,volatile-memdev=vmem0,dc-memdev=vmem1,id=cxl-vmem0,num-dc-regions=2,ldid=0 \
-device cxl-type3,bus=swport0,volatile-memdev=vmem1,dc-memdev=vmem2,id=cxl-vmem1,num-dc-regions=2,ldid=1 \
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=24,target=cxl-vmem0,cxl-vmem1")
With this configuration, the same i2c device is handing both LDs and in FMAPI commands we use the LDID specified above.
Thanks,
Shesha.
Re: Concept of LD-ID in QEMU
Quoted 2 lines · On Wed, 7 Jun 2023 23:01:11 +0000 Shesha Bhushan Sreenivasamurthy…
Hi,
For DCD sideband there needs to be LD-ID. Is the following approach acceptable?
QEMU question so +CC qemu-devel
Quoted 3 lines
-device cxl-type3,bus=swport0,volatile-memdev=vmem0,dc-memdev=vmem1,id=cxl-vmem0,num-dc-regions=2,ldid=0 \
-device cxl-type3,bus=swport0,volatile-memdev=vmem1,dc-memdev=vmem2,id=cxl-vmem1,num-dc-regions=2,ldid=1 \
Those will be PCI functions at this level so you can't do this until we have more advanced switch support (it has to know about multiple VHs - right now we only support fixed config switches). You could connect them to different switch ports - effectively that will be what it looks like when we do emulate a configurable switch.
Quoted 3 lines
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=24,target=cxl-vmem0,cxl-vmem1")
With this configuration, the same i2c device is handing both LDs and in FMAPI commands we use the LDID specified above.
This effectively becomes a partial implementation of either an MLD or an MH-SLD
To manage the actual memory access, those will almost certainly need a bunch of other shared infrastructure. So I'd ultimately expect the i2c_mctp_cxl device to target whatever device represents that shared infrastructure - it might be a separate device or a 'lead' type 3 device.
So I'm not sure how this will fit together longer term. We need same infrastructure to work for a mailbox CCI on a MH-SLD/MLD as well and in that case there isn't a separate device to which we can provide multiple targets as you've done in your proposal here.
So I think we need to work out how to handle all of (I've probably forgotten something) X marks done or in progress.
X 1) i2c_mctp_cxl to an SLD (no PCI Mailbox definition for this one)
2) i2c_mctp_cxl directly to an MLD (your case)
X 3) i2c_mctp_cxl to a fixed config switch (single fixed VH no MLD capable ports)
X 4) PCI mailbox via switch CCI device that fixed config switch (no MLD capable ports)
Even with this simple design some fun things you can do.
5) i2c_mctp_cxl to a configurable switch (probably a separate as yet to be defined management interface - that messes with hotplug)
6) PCI mailbox via switch CCI to configurable switch (again to defined management interface).
7) i2c_mctp_cxl to an MH-SLD - probably to which ever device also has support for
tunneling to the FM owned LD via the PCI mailbox.
X 8) PCI mailbox on MH-SLD tunneling to the FM owned LD.
9) i2c_mctp_cxl to an MH-MLD - similar to above - this one isn't that much more complex than MH-SLD case.
X 10) PCI mailbox to MH-MLD - similar to above.
11) Tunneling via the switch CCI (then over PCI-VDM - though that detail isn't visible in QEMU) to an SLD
12) Tunneling via the switch CCI (then PCI-VDM) to an MH-SLD and on to he FM owned LD.
13) Tunneling via the switch CCI (then over PCI-VDM) to an MLD / MH-MLD
Current i2c_mctp_cxl covers 1 and 3
I'm part way through the tunnelling support for (8 and 100) - need to revisit and wire up the switch CCI PoC properly which will give us 4.
2 needs MLD support in general which we could maybe make work with a static binding in a switch but that
would be odd - so we probably need to emulate a configurable switch for that
5,6 need configurable switch
7 needs same as 2 plus tunneling part similar to 4
9 again probably configurable switch for the MLD part to make sense
11 is fairly straight forward - but not done yet.
12 also not too bad, but needs the MH-SLD part to be fleshed out (some work on going ) 13 needs pretty much everything defined.
Trying to get the command line interface and device model right until we have PoC code for a few more cases is going to be at most a draft of what it might look like.
So in short, lots to do. For now feel free to hack whatever you need in to be able to test the FM-API side of things, we can move that towards a clean command line definition once we have one figured out!
Jonathan
Quoted 3 lines
Thanks,
Shesha.
Re: Concept of LD-ID in QEMU
Shesha,
You've sent an email with the 'In-reply-to' set to one of Terry's patches. Please check why that happened and make sure you don't do that in future as it hides your unrelated thread in email clients and the archives!
See
https://lore.kernel.org/linux-cxl/[email protected]/T/#t
for example
Jonathan
Quoted 76 lines · On Thu, 8 Jun 2023 11:31:53 +0100 Jonathan Cameron…
Quoted 2 lines · On Wed, 7 Jun 2023 23:01:11 +0000 Shesha Bhushan Sreenivasamurthy…
Hi,
For DCD sideband there needs to be LD-ID. Is the following approach acceptable?QEMU question so +CC qemu-devel
Quoted 3 lines
-device cxl-type3,bus=swport0,volatile-memdev=vmem0,dc-memdev=vmem1,id=cxl-vmem0,num-dc-regions=2,ldid=0 \
-device cxl-type3,bus=swport0,volatile-memdev=vmem1,dc-memdev=vmem2,id=cxl-vmem1,num-dc-regions=2,ldid=1 \Those will be PCI functions at this level so you can't do this until we have more advanced switch support (it has to know about multiple VHs - right now we only support fixed config switches). You could connect them to different switch ports - effectively that will be what it looks like when we do emulate a configurable switch.
Quoted 3 lines
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=24,target=cxl-vmem0,cxl-vmem1")
With this configuration, the same i2c device is handing both LDs and in FMAPI commands we use the LDID specified above.
This effectively becomes a partial implementation of either an MLD or an MH-SLD
To manage the actual memory access, those will almost certainly need a bunch of other shared infrastructure. So I'd ultimately expect thei2c_mctp_cxldevice to target whatever device represents that shared infrastructure - it might be a separate device or a 'lead' type 3 device.So I'm not sure how this will fit together longer term. We need same infrastructure to work for a mailbox CCI on a MH-SLD/MLD as well and in that case there isn't a separate device to which we can provide multiple targets as you've done in your proposal here.
So I think we need to work out how to handle all of (I've probably forgotten something) X marks done or in progress.
X 1)
i2c_mctp_cxlto an SLD (no PCI Mailbox definition for this one)
2)i2c_mctp_cxldirectly to an MLD (your case)
X 3)i2c_mctp_cxlto a fixed config switch (single fixed VH no MLD capable ports)
X 4) PCI mailbox via switch CCI device that fixed config switch (no MLD capable ports)
Even with this simple design some fun things you can do.
5)i2c_mctp_cxlto a configurable switch (probably a separate as yet to be defined management interface - that messes with hotplug)
6) PCI mailbox via switch CCI to configurable switch (again to defined management interface).
7)i2c_mctp_cxlto an MH-SLD - probably to which ever device also has support for
tunneling to the FM owned LD via the PCI mailbox.
X 8) PCI mailbox on MH-SLD tunneling to the FM owned LD.
9)i2c_mctp_cxlto an MH-MLD - similar to above - this one isn't that much more complex than MH-SLD case.
X 10) PCI mailbox to MH-MLD - similar to above.
11) Tunneling via the switch CCI (then over PCI-VDM - though that detail isn't visible in QEMU) to an SLD
12) Tunneling via the switch CCI (then PCI-VDM) to an MH-SLD and on to he FM owned LD.
13) Tunneling via the switch CCI (then over PCI-VDM) to an MLD / MH-MLDCurrent
i2c_mctp_cxlcovers 1 and 3
I'm part way through the tunnelling support for (8 and 100) - need to revisit and wire up the switch CCI PoC properly which will give us 4.2 needs MLD support in general which we could maybe make work with a static binding in a switch but that
would be odd - so we probably need to emulate a configurable switch for that
5,6 need configurable switch
7 needs same as 2 plus tunneling part similar to 4
9 again probably configurable switch for the MLD part to make sense
11 is fairly straight forward - but not done yet.
12 also not too bad, but needs the MH-SLD part to be fleshed out (some work on going ) 13 needs pretty much everything defined.Trying to get the command line interface and device model right until we have PoC code for a few more cases is going to be at most a draft of what it might look like.
So in short, lots to do. For now feel free to hack whatever you need in to be able to test the FM-API side of things, we can move that towards a clean command line definition once we have one figured out!
Jonathan
Quoted 3 lines
Thanks,
Shesha.
Re: [EXT] Re: Concept of LD-ID in QEMU
Hi,
Thinking a bit more, LD in CXL are PCIe endpoint functions. Therefore 1-1 mapping of cxl-i2c device per PCIe device is sufficient, and we use function number in BDF as the LD-ID. Does it makes sense ?
From: Jonathan Cameron <[email protected]>
Sent: Thursday, June 8, 2023 3:36 AM
To: Shesha Bhushan Sreenivasamurthy <[email protected]>
Cc: [email protected] <[email protected]>; <"[email protected]" <[email protected]>
Subject: [EXT] Re: Concept of LD-ID in QEMU
External Email
----------------------------------------------------------------------
Shesha,
You've sent an email with the 'In-reply-to' set to one of Terry's patches. Please check why that happened and make sure you don't do that in future as it hides your unrelated thread in email clients and the archives!
ss - Apologies. Will be careful.
Jonathan
Quoted 76 lines · On Thu, 8 Jun 2023 11:31:53 +0100 Jonathan Cameron…
Quoted 2 lines · On Wed, 7 Jun 2023 23:01:11 +0000 Shesha Bhushan Sreenivasamurthy…
Hi,
For DCD sideband there needs to be LD-ID. Is the following approach acceptable?QEMU question so +CC qemu-devel
Quoted 3 lines
-device cxl-type3,bus=swport0,volatile-memdev=vmem0,dc-memdev=vmem1,id=cxl-vmem0,num-dc-regions=2,ldid=0 \
-device cxl-type3,bus=swport0,volatile-memdev=vmem1,dc-memdev=vmem2,id=cxl-vmem1,num-dc-regions=2,ldid=1 \Those will be PCI functions at this level so you can't do this until we have more advanced switch support (it has to know about multiple VHs - right now we only support fixed config switches). You could connect them to different switch ports - effectively that will be what it looks like when we do emulate a configurable switch.
Quoted 3 lines
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=24,target=cxl-vmem0,cxl-vmem1")
With this configuration, the same i2c device is handing both LDs and in FMAPI commands we use the LDID specified above.
This effectively becomes a partial implementation of either an MLD or an MH-SLD
To manage the actual memory access, those will almost certainly need a bunch of other shared infrastructure. So I'd ultimately expect thei2c_mctp_cxldevice to target whatever device represents that shared infrastructure - it might be a separate device or a 'lead' type 3 device.So I'm not sure how this will fit together longer term. We need same infrastructure to work for a mailbox CCI on a MH-SLD/MLD as well and in that case there isn't a separate device to which we can provide multiple targets as you've done in your proposal here.
So I think we need to work out how to handle all of (I've probably forgotten something) X marks done or in progress.
X 1)
i2c_mctp_cxlto an SLD (no PCI Mailbox definition for this one)
2)i2c_mctp_cxldirectly to an MLD (your case)
X 3)i2c_mctp_cxlto a fixed config switch (single fixed VH no MLD capable ports)
X 4) PCI mailbox via switch CCI device that fixed config switch (no MLD capable ports)
Even with this simple design some fun things you can do.
5)i2c_mctp_cxlto a configurable switch (probably a separate as yet to be defined management interface - that messes with hotplug)
6) PCI mailbox via switch CCI to configurable switch (again to defined management interface).
7)i2c_mctp_cxlto an MH-SLD - probably to which ever device also has support for
tunneling to the FM owned LD via the PCI mailbox.
X 8) PCI mailbox on MH-SLD tunneling to the FM owned LD.
9)i2c_mctp_cxlto an MH-MLD - similar to above - this one isn't that much more complex than MH-SLD case.
X 10) PCI mailbox to MH-MLD - similar to above.
11) Tunneling via the switch CCI (then over PCI-VDM - though that detail isn't visible in QEMU) to an SLD
12) Tunneling via the switch CCI (then PCI-VDM) to an MH-SLD and on to he FM owned LD.
13) Tunneling via the switch CCI (then over PCI-VDM) to an MLD / MH-MLD
Currenti2c_mctp_cxlcovers 1 and 3
I'm part way through the tunnelling support for (8 and 100) - need to revisit and wire up the switch CCI PoC properly which will give us 4.2 needs MLD support in general which we could maybe make work with a static binding in a switch but that would be odd - so we probably need to emulate a configurable switch for that
5,6 need configurable switch
7 needs same as 2 plus tunneling part similar to 4
9 again probably configurable switch for the MLD part to make sense
11 is fairly straight forward - but not done yet.
12 also not too bad, but needs the MH-SLD part to be fleshed out (some work on going ) 13 needs pretty much everything defined.Trying to get the command line interface and device model right until we have PoC code for a few more cases is going to be at most a draft of what it might look like.
So in short, lots to do. For now feel free to hack whatever you need in to be able to test the FM-API side of things, we can move that towards a clean command line definition once we have one figured out!
Jonathan
Quoted 3 lines
Thanks,
Shesha.
Re: [EXT] Re: Concept of LD-ID in QEMU
Quoted 3 lines · On Thu, 8 Jun 2023 23:38:34 +0000 Shesha Bhushan Sreenivasamurthy…
Hi,
Thinking a bit more, LD in CXL are PCIe endpoint functions. Therefore 1-1 mapping of cxl-i2c device per PCIe device is sufficient, and we use function number in BDF as the LD-ID. Does it makes sense ?
LDs are PCIe endpoint functions (always function 0) as seen from the Virtual Heirarchies (as they end up under a particular vPBB - which look like a downstream port of a switch to the host) but they aren't from a more general topology point of view of actual fabric topology and when we are tunneling we address them via physical port, not virtual port I think (not read that bit of the spec for a while). See figure 7-23 in CXL 3.0
Outer tunneling command targets a port number (unwrapped at the switch), inner one targets the LD - unwrapped at the MLD and sent to appropriate LD including FM owned LD (if I understand this stack directly).
Also no relationship between BDF and LD-ID so don't do that as the maximum ID is only 16 which would rather limit your PCI toplogies if that's the BDF as well.
For now just do what you originally said and add an ID (starting from 0). We can probably do that automatically once more infrastructure is in place.
Jonathan
Quoted 104 lines
From: Jonathan Cameron <[email protected]>
Sent: Thursday, June 8, 2023 3:36 AM
To: Shesha Bhushan Sreenivasamurthy <[email protected]>
Cc: [email protected] <[email protected]>; <"[email protected]" <[email protected]>
Subject: [EXT] Re: Concept of LD-ID in QEMU
External Email----------------------------------------------------------------------
Shesha,
You've sent an email with the 'In-reply-to' set to one of Terry's patches. Please check why that happened and make sure you don't do that in future as it hides your unrelated thread in email clients and the archives!
ss - Apologies. Will be careful.
Jonathan
Quoted 76 lines · On Thu, 8 Jun 2023 11:31:53 +0100 Jonathan Cameron…
Quoted 2 lines · On Wed, 7 Jun 2023 23:01:11 +0000 Shesha Bhushan Sreenivasamurthy…
Hi,
For DCD sideband there needs to be LD-ID. Is the following approach acceptable?QEMU question so +CC qemu-devel
Quoted 3 lines
-device cxl-type3,bus=swport0,volatile-memdev=vmem0,dc-memdev=vmem1,id=cxl-vmem0,num-dc-regions=2,ldid=0 \
-device cxl-type3,bus=swport0,volatile-memdev=vmem1,dc-memdev=vmem2,id=cxl-vmem1,num-dc-regions=2,ldid=1 \Those will be PCI functions at this level so you can't do this until we have more advanced switch support (it has to know about multiple VHs - right now we only support fixed config switches). You could connect them to different switch ports - effectively that will be what it looks like when we do emulate a configurable switch.
Quoted 3 lines
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=24,target=cxl-vmem0,cxl-vmem1")
With this configuration, the same i2c device is handing both LDs and in FMAPI commands we use the LDID specified above.
This effectively becomes a partial implementation of either an MLD or an MH-SLD
To manage the actual memory access, those will almost certainly need a bunch of other shared infrastructure. So I'd ultimately expect thei2c_mctp_cxldevice to target whatever device represents that shared infrastructure - it might be a separate device or a 'lead' type 3 device.So I'm not sure how this will fit together longer term. We need same infrastructure to work for a mailbox CCI on a MH-SLD/MLD as well and in that case there isn't a separate device to which we can provide multiple targets as you've done in your proposal here.
So I think we need to work out how to handle all of (I've probably forgotten something) X marks done or in progress.
X 1)
i2c_mctp_cxlto an SLD (no PCI Mailbox definition for this one)
2)i2c_mctp_cxldirectly to an MLD (your case)
X 3)i2c_mctp_cxlto a fixed config switch (single fixed VH no MLD capable ports)
X 4) PCI mailbox via switch CCI device that fixed config switch (no MLD capable ports)
Even with this simple design some fun things you can do.
5)i2c_mctp_cxlto a configurable switch (probably a separate as yet to be defined management interface - that messes with hotplug)
6) PCI mailbox via switch CCI to configurable switch (again to defined management interface).
7)i2c_mctp_cxlto an MH-SLD - probably to which ever device also has support for
tunneling to the FM owned LD via the PCI mailbox.
X 8) PCI mailbox on MH-SLD tunneling to the FM owned LD.
9)i2c_mctp_cxlto an MH-MLD - similar to above - this one isn't that much more complex than MH-SLD case.
X 10) PCI mailbox to MH-MLD - similar to above.
11) Tunneling via the switch CCI (then over PCI-VDM - though that detail isn't visible in QEMU) to an SLD
12) Tunneling via the switch CCI (then PCI-VDM) to an MH-SLD and on to he FM owned LD.
13) Tunneling via the switch CCI (then over PCI-VDM) to an MLD / MH-MLD
Currenti2c_mctp_cxlcovers 1 and 3
I'm part way through the tunnelling support for (8 and 100) - need to revisit and wire up the switch CCI PoC properly which will give us 4.2 needs MLD support in general which we could maybe make work with a static binding in a switch but that would be odd - so we probably need to emulate a configurable switch for that
5,6 need configurable switch
7 needs same as 2 plus tunneling part similar to 4
9 again probably configurable switch for the MLD part to make sense
11 is fairly straight forward - but not done yet.
12 also not too bad, but needs the MH-SLD part to be fleshed out (some work on going ) 13 needs pretty much everything defined.Trying to get the command line interface and device model right until we have PoC code for a few more cases is going to be at most a draft of what it might look like.
So in short, lots to do. For now feel free to hack whatever you need in to be able to test the FM-API side of things, we can move that towards a clean command line definition once we have one figured out!
Jonathan
Quoted 3 lines
Thanks,
Shesha.
RE: [PATCH v5 14/26] cxl/pci: Store the endpoint's Component Register mappings in struct cxl_dev_state
In reply to Terry BowmanQuoted 8 lines · Terry Bowman
From: Robert Richter <[email protected]>
Same as for ports and dports, also store the endpoint's Component Register mappings, use struct
cxl_dev_statefor that.Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
This really feels like it should fold in the removal of cxlds->component_reg_phys in the same patch. I do not see any reason for cxlds->component_reg_phys and cxlds->comp_map to coexist in the history.
PATCH v5 15/26 cxl/hdm: Use stored Component Register mappings to map HDM decoder capability
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
Now, that the Component Register mappings are stored, use them to enable and map the HDM decoder capabilities. The Component Registers do not need to be probed again for this, remove probing code.
The HDM capability applies to Endpoints, USPs and VH Host Bridges. The Endpoint's component register mappings are located in the cxlds and else in the port's structure. Provide a helper function cxl_port_get_comp_map() to locate the mappings depending on the component's type.
--- drivers/cxl/core/hdm.c | 59 +++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 29 deletions(-)
drivers/cxl/core/hdm.c +30 -29
@@ -81,26 +81,6 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) cxlhdm->interleave_mask |=GENMASK (14 ,12 ); } -static int map_hdm_decoder_regs (struct cxl_port *port,void __iomem *crb, -struct cxl_component_regs *regs) -{ -struct cxl_register_map map = { - .dev = &port->dev, - .resource = port->component_reg_phys, - .base = crb, - .max_size =CXL_COMPONENT_REG_BLOCK_SIZE , - }; - -cxl_probe_component_regs (&port->dev, crb, &map.component_map); -if (!map.component_map.hdm_decoder.valid) { -dev_dbg (&port->dev,"HDM decoder registers not implemented\n" ); -/* unique error code to indicate no HDM decoder capability */ -return -ENODEV ; - } - -return cxl_map_component_regs (&map, regs,BIT (CXL_CM_CAP_CAP_ID_HDM )); -} -static bool should_emulate_decoders (struct cxl_endpoint_dvsec_info *info) {struct cxl_hdm *cxlhdm;
@@ -145,6 +125,22 @@ static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info)return true; } +static struct cxl_register_map *cxl_port_get_comp_map (struct cxl_port *port) +{ +/* +* HDM capability applies to Endpoints, USPs and VH Host +* Bridges. The Endpoint's component register mappings are +* located in the cxlds. +*/ +if (is_cxl_endpoint (port)) { +struct cxl_memdev *memdev =to_cxl_memdev (port->uport); + +return &memdev->cxlds->comp_map; + } + +return &port->comp_map; +} +/** * devm_cxl_setup_hdm - map HDM decoder component registers * @port: cxl_port to map
@@ -155,7 +151,7 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, {struct device *dev = &port->dev;struct cxl_hdm *cxlhdm; -void __iomem *crb; +struct cxl_register_map *comp_map;int rc; cxlhdm =devm_kzalloc (dev,sizeof (*cxlhdm),GFP_KERNEL );
@@ -164,19 +160,24 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, cxlhdm->port = port;dev_set_drvdata (dev, cxlhdm); - crb =ioremap (port->component_reg_phys,CXL_COMPONENT_REG_BLOCK_SIZE ); -if (!crb && info && info->mem_enabled) { - cxlhdm->decoder_count = info->ranges; -return cxlhdm; - }else if (!crb) { + comp_map =cxl_port_get_comp_map (port); + +if (!comp_map->component_map.hdm_decoder.valid) { +dev_dbg (&port->dev,"HDM decoder registers not found\n" ); +if (info && info->mem_enabled) { + cxlhdm->decoder_count = info->ranges; +return cxlhdm; + }dev_err (dev,"No component registers mapped\n" );return ERR_PTR (-ENXIO ); } - rc =map_hdm_decoder_regs (port, crb, &cxlhdm->regs); -iounmap (crb); -if (rc) + rc =cxl_map_component_regs (comp_map, &cxlhdm->regs, +BIT (CXL_CM_CAP_CAP_ID_HDM )); +if (rc) { +dev_dbg (dev,"Failed to map HDM capability.\n" );return ERR_PTR (rc); + }parse_hdm_decoder_caps (cxlhdm);if (cxlhdm->decoder_count ==0 ) {
Signature · 1 line
2.34.1
RE: [PATCH v5 15/26] cxl/hdm: Use stored Component Register mappings to map HDM decoder capability
Quoted 12 lines · Terry Bowman
From: Robert Richter <[email protected]>
Now, that the Component Register mappings are stored, use them to enable and map the HDM decoder capabilities. The Component Registers do not need to be probed again for this, remove probing code.
The HDM capability applies to Endpoints, USPs and VH Host Bridges. The Endpoint's component register mappings are located in the cxlds and else in the port's structure. Provide a helper function
cxl_port_get_comp_map()to locate the mappings depending on the component's type.
Quoted 3 lines
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Looks good.
PATCH v5 16/26 cxl/port: Remove Component Register base address from struct cxl_port
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
--- drivers/cxl/core/port.c | 4 +--- drivers/cxl/cxl.h | 2 -- 2 files changed, 1 insertion(+), 5 deletions(-)
drivers/cxl/core/port.c +1 -3
@@ -615,7 +615,6 @@ static int devm_cxl_link_parent_dport(struct device *host,static struct lock_class_key cxl_port_key;static struct cxl_port *cxl_port_alloc (struct device *uport, -resource_size_t component_reg_phys,struct cxl_dport *parent_dport) {struct cxl_port *port;
@@ -665,7 +664,6 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, }else dev->parent = uport; - port->component_reg_phys = component_reg_phys;ida_init (&port->decoder_ida); port->hdm_end =-1 ; port->commit_end =-1 ;
@@ -724,7 +722,7 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,struct device *dev;int rc; - port =cxl_port_alloc (uport, component_reg_phys, parent_dport); + port =cxl_port_alloc (uport, parent_dport);if (IS_ERR (port))return port;
drivers/cxl/cxl.h +0 -2
@@ -556,7 +556,6 @@ struct cxl_dax_region {* @nr_dports: number of entries in @dports * @hdm_end: track last allocated HDM decoder instance for allocation ordering * @commit_end: cursor to track highest committed decoder for commit ordering -* @component_reg_phys: component register capability base address (optional) * @dead: last ep has been removed, force port re-creation * @depth: How deep this port is relative to the root. depth 0 is the root. * @cdat: Cached CDAT data
@@ -576,7 +575,6 @@ struct cxl_port {int nr_dports;int hdm_end;int commit_end; -resource_size_t component_reg_phys;bool dead;unsigned int depth;struct cxl_cdat {
Signature · 1 line
2.34.1
RE: [PATCH v5 16/26] cxl/port: Remove Component Register base address from struct cxl_port
Quoted 9 lines · Terry Bowman
From: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Looks good.
PATCH v5 17/26 cxl/port: Remove Component Register base address from struct cxl_dport
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
--- drivers/cxl/core/port.c | 1 - drivers/cxl/cxl.h | 2 -- 2 files changed, 3 deletions(-)
drivers/cxl/core/port.c +0 -1
@@ -994,7 +994,6 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, dport->dev = dport_dev; dport->port_id = port_id; - dport->component_reg_phys = component_reg_phys; dport->port = port; rc =cxl_dport_setup_regs (dport, component_reg_phys);
drivers/cxl/cxl.h +0 -2
@@ -601,7 +601,6 @@ struct cxl_rcrb_info {* @port: reference to cxl_port that contains this downstream port * @comp_map: component register capability mappings * @port_id: unique hardware identifier for dport in decoder target list -* @component_reg_phys: downstream port component registers * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @rcrb: Data about the Root Complex Register Block layout */
@@ -610,7 +609,6 @@ struct cxl_dport {struct cxl_port *port;struct cxl_register_map comp_map;int port_id; -resource_size_t component_reg_phys;bool rch;struct cxl_rcrb_info rcrb; };
Signature · 1 line
2.34.1
RE: [PATCH v5 17/26] cxl/port: Remove Component Register base address from struct cxl_dport
Quoted 9 lines · Terry Bowman
From: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
As I mentioned earlier this could have happened several patches back, so looks good, but consider moving it earlier in the series for v6.
PATCH v5 18/26 cxl/pci: Remove Component Register base address from struct cxl_dev_state
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
--- drivers/cxl/cxlmem.h | 2 -- drivers/cxl/mem.c | 4 ++-- drivers/cxl/pci.c | 3 --- 3 files changed, 2 insertions(+), 7 deletions(-)
drivers/cxl/cxlmem.h +0 -2
@@ -287,7 +287,6 @@ struct cxl_poison_state {* @active_persistent_bytes: sum of hard + soft persistent * @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset -* @component_reg_phys: register base of component registers * @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @event: event log driver state
@@ -326,7 +325,6 @@ struct cxl_dev_state {u64 next_volatile_bytes;u64 next_persistent_bytes; -resource_size_t component_reg_phys;u64 serial;struct cxl_event_state event;
drivers/cxl/mem.c +2 -2
@@ -49,7 +49,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, struct cxl_dport *parent_dport) {struct cxl_port *parent_port = parent_dport->port; -struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_port *endpoint, *iter, *down;int rc;
@@ -65,8 +64,9 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, ep->next = down; } +/* The Endpoint's component regs are located in cxlds. */ endpoint =devm_cxl_add_port (host, &cxlmd->dev, - cxlds->component_reg_phys, +CXL_RESOURCE_NONE , parent_dport);if (IS_ERR (endpoint))return PTR_ERR (endpoint);
drivers/cxl/pci.c +0 -3
@@ -661,7 +661,6 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)* If the component registers can't be found, the cxl_pci driver may * still be useful for management functions so don't return an error. */ - cxlds->component_reg_phys =CXL_RESOURCE_NONE ; rc =cxl_pci_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &cxlds->comp_map);if (rc)
@@ -669,8 +668,6 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)else if (!cxlds->comp_map.component_map.ras.valid)dev_dbg (&pdev->dev,"RAS registers not found\n" ); - cxlds->component_reg_phys = cxlds->comp_map.resource; - rc =cxl_map_component_regs (&cxlds->comp_map, &cxlds->regs.component,BIT (CXL_CM_CAP_CAP_ID_RAS ));if (rc)
Signature · 1 line
2.34.1
RE: [PATCH v5 18/26] cxl/pci: Remove Component Register base address from struct cxl_dev_state
Quoted 9 lines · Terry Bowman
From: Robert Richter <[email protected]>
The Component Register base address @component_reg_phys is no longer used after the rework of the Component Register setup which now uses struct member @comp_map instead. Remove the base address.
Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Looks good, if possible fold it in with the introduction of cxlds->comp_map.
PATCH v5 19/26 cxl/pci: Add RCH downstream port AER register discovery
In reply to Terry BowmanRestricted CXL host (RCH) downstream port AER information is not currently logged while in the error state. One problem preventing the error logging is the AER and RAS registers are not accessible. The CXL driver requires changes to find RCH downstream port AER and RAS registers for purpose of error logging.
RCH downstream ports are not enumerated during a PCI bus scan and are instead discovered using system firmware, ACPI in this case.[1] The downstream port is implemented as a Root Complex Register Block (RCRB). The RCRB is a 4k memory block containing PCIe registers based on the PCIe root port.[2] The RCRB includes AER extended capability registers used for reporting errors. Note, the RCH's AER Capability is located in the RCRB memory space instead of PCI configuration space, thus its register access is different. Existing kernel PCIe AER functions can not be used to manage the downstream port AER capabilities and RAS registers because the port was not enumerated during PCI scan and the registers are not PCI config accessible.
Discover RCH downstream port AER extended capability registers. Use MMIO accesses to search for extended AER capability in RCRB register space.
[1] CXL 3.0 Spec, 9.11.2 - System Firmware View of CXL 1.1 Hierarchy
[2] CXL 3.0 Spec, 8.2.1.1 - RCH Downstream Port RCRB
--- drivers/cxl/core/regs.c | 51 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)
drivers/cxl/core/regs.c +51 -0
@@ -408,6 +408,54 @@ int cxl_setup_regs(struct cxl_register_map *map) }EXPORT_SYMBOL_NS_GPL (cxl_setup_regs,CXL ); +static void __iomem *cxl_map_reg (struct device *dev,resource_size_t addr, +resource_size_t length) +{ +struct resource *res; + +if (WARN_ON_ONCE (addr ==CXL_RESOURCE_NONE )) +return NULL ; + + res =request_mem_region (addr, length,dev_name (dev)); +if (!res) +return NULL ; + +return ioremap (addr, length); +} + +static void cxl_unmap_reg (void __iomem *base,resource_size_t addr, +resource_size_t length) +{ +iounmap (base); +release_mem_region (addr, length); +} + +static u16 cxl_rcrb_to_aer (struct device *dev,resource_size_t rcrb) +{ +void __iomem *addr; +u16 offset =0 ; +u32 cap_hdr; + + addr =cxl_map_reg (dev, rcrb,SZ_4K ); +if (!addr) +return 0 ; + + cap_hdr =readl (addr + offset); +while (PCI_EXT_CAP_ID (cap_hdr) !=PCI_EXT_CAP_ID_ERR ) { + offset =PCI_EXT_CAP_NEXT (cap_hdr); +if (!offset) +break ; + cap_hdr =readl (addr + offset); + } + +if (offset) +dev_dbg (dev,"found AER extended capability (0x%x)\n" , offset); + +cxl_unmap_reg (addr, rcrb,SZ_4K ); + +return offset; +} +resource_size_t cxl_probe_rcrb (struct device *dev,resource_size_t rcrb,struct cxl_rcrb_info *ri,enum cxl_rcrb which) {
@@ -471,6 +519,9 @@ resource_size_t cxl_probe_rcrb(struct device *dev, resource_size_t rcrb,if (!IS_ALIGNED (component_reg_phys,CXL_COMPONENT_REG_BLOCK_SIZE ))return CXL_RESOURCE_NONE ; +if (ri) + ri->aer_cap =cxl_rcrb_to_aer (dev, ri->base); +return component_reg_phys; }EXPORT_SYMBOL_NS_GPL (cxl_probe_rcrb,CXL );
Signature · 1 line
2.34.1
RE: [PATCH v5 19/26] cxl/pci: Add RCH downstream port AER register discovery
Quoted 51 lines · Terry Bowman
Restricted CXL host (RCH) downstream port AER information is not currently logged while in the error state. One problem preventing the error logging is the AER and RAS registers are not accessible. The CXL driver requires changes to find RCH downstream port AER and RAS registers for purpose of error logging.
RCH downstream ports are not enumerated during a PCI bus scan and are instead discovered using system firmware, ACPI in this case.[1] The downstream port is implemented as a Root Complex Register Block (RCRB). The RCRB is a 4k memory block containing PCIe registers based on the PCIe root port.[2] The RCRB includes AER extended capability registers used for reporting errors. Note, the RCH's AER Capability is located in the RCRB memory space instead of PCI configuration space, thus its register access is different. Existing kernel PCIe AER functions can not be used to manage the downstream port AER capabilities and RAS registers because the port was not enumerated during PCI scan and the registers are not PCI config accessible.
Discover RCH downstream port AER extended capability registers. Use MMIO accesses to search for extended AER capability in RCRB register space.
[1] CXL 3.0 Spec, 9.11.2 - System Firmware View of CXL 1.1 Hierarchy
[2] CXL 3.0 Spec, 8.2.1.1 - RCH Downstream Port RCRBCo-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/regs.c | 51 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)drivers/cxl/core/regs.c +11
-0@@ -408,6 +408,54 @@ int cxl_setup_regs(struct cxl_register_map *map) }EXPORT_SYMBOL_NS_GPL (cxl_setup_regs,CXL ); +static void __iomem *cxl_map_reg (struct device *dev,resource_size_t addr, +resource_size_t length) +{ +struct resource *res; + +if (WARN_ON_ONCE (addr ==CXL_RESOURCE_NONE )) +return NULL ; + + res =request_mem_region (addr, length,dev_name (dev)); +if (!res) +return NULL ;
drivers/cxl/core/regs.c int cxl_setup_regs(struct cxl_register_map *map) · line 431
Quoted 10 lines
drivers/cxl/core/regs.c +10
-0+ +return ioremap (addr, length); +} + +static void cxl_unmap_reg (void __iomem *base,resource_size_t addr, +resource_size_t length) +{ +iounmap (base); +release_mem_region (addr, length); +}
Why redo the {request,release}_mem_region() and ioremap() vs handling this inside of the existing mapping of the RCRB in this function?
Re: [PATCH v5 19/26] cxl/pci: Add RCH downstream port AER register discovery
Hi Dan,
I added a response inline below.
Quoted 57 lines · Dan Williams
Quoted 56 lines · Terry Bowman
Restricted CXL host (RCH) downstream port AER information is not currently logged while in the error state. One problem preventing the error logging is the AER and RAS registers are not accessible. The CXL driver requires changes to find RCH downstream port AER and RAS registers for purpose of error logging.
RCH downstream ports are not enumerated during a PCI bus scan and are instead discovered using system firmware, ACPI in this case.[1] The downstream port is implemented as a Root Complex Register Block (RCRB). The RCRB is a 4k memory block containing PCIe registers based on the PCIe root port.[2] The RCRB includes AER extended capability registers used for reporting errors. Note, the RCH's AER Capability is located in the RCRB memory space instead of PCI configuration space, thus its register access is different. Existing kernel PCIe AER functions can not be used to manage the downstream port AER capabilities and RAS registers because the port was not enumerated during PCI scan and the registers are not PCI config accessible.
Discover RCH downstream port AER extended capability registers. Use MMIO accesses to search for extended AER capability in RCRB register space.
[1] CXL 3.0 Spec, 9.11.2 - System Firmware View of CXL 1.1 Hierarchy
[2] CXL 3.0 Spec, 8.2.1.1 - RCH Downstream Port RCRBCo-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/regs.c | 51 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)drivers/cxl/core/regs.c +16
-0@@ -408,6 +408,54 @@ int cxl_setup_regs(struct cxl_register_map *map) }EXPORT_SYMBOL_NS_GPL (cxl_setup_regs,CXL ); +static void __iomem *cxl_map_reg (struct device *dev,resource_size_t addr, +resource_size_t length) +{ +struct resource *res; + +if (WARN_ON_ONCE (addr ==CXL_RESOURCE_NONE )) +return NULL ; + + res =request_mem_region (addr, length,dev_name (dev)); +if (!res) +return NULL ; + +return ioremap (addr, length); +} + +static void cxl_unmap_reg(void __iomem *base, resource_size_t addr,
Quoted 8 lines
drivers/cxl/core/regs.c int cxl_setup_regs(struct cxl_register_map *map) · line 431
Quoted 5 lines
drivers/cxl/core/regs.c +5
-0+ resource_size_t length) +{ +iounmap (base); +release_mem_region (addr, length); +}Why redo the {request,release}_mem_region() and
ioremap()vs handling this inside of the existing mapping of the RCRB in this function?
The intention was to follow the same pattern as existing {request,release} functions but doesn't make much sense with only one user in this case. I'll fold the {request,release} logic into cxl_rcrb_to_aer().
Regards,
Terry
PATCH v5 20/26 PCI/AER: Refactor cper_print_aer() for use by CXL driver module
In reply to Terry BowmanThe CXL driver plans to use cper_print_aer() for logging restricted CXL host (RCH) AER errors. cper_print_aer() is not currently exported and therefore not usable by the CXL drivers built as loadable modules. Export the cper_print_aer() function. Use the EXPORT_SYMBOL_NS_GPL() variant to restrict the export to CXL drivers.
The CONFIG_ACPI_APEI_PCIEAER kernel config is currently used to enable cper_print_aer(). cper_print_aer() logs the AER registers and is useful in PCIE AER logging outside of APEI. Remove the CONFIG_ACPI_APEI_PCIEAER dependency to enable cper_print_aer().
The cper_print_aer() function name implies CPER specific use but is useful in non-CPER cases as well. Rename cper_print_aer() to pci_print_aer().
--- drivers/pci/pcie/aer.c | 9 +++++---- include/linux/aer.h | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-)
drivers/pci/pcie/aer.c +5 -4
@@ -771,9 +771,10 @@ int cper_severity_to_aer(int cper_severity) } }EXPORT_SYMBOL_GPL (cper_severity_to_aer); +#endif -void cper_print_aer (struct pci_dev *dev,int aer_severity, -struct aer_capability_regs *aer) +void pci_print_aer (struct pci_dev *dev,int aer_severity, +struct aer_capability_regs *aer) {int layer, agent, tlp_header_valid =0 ;u32 status, mask;
@@ -812,7 +813,7 @@ void cper_print_aer(struct pci_dev *dev, int aer_severity,trace_aer_event (dev_name (&dev->dev), (status & ~mask), aer_severity, tlp_header_valid, &aer->header_log); } -#endif +EXPORT_SYMBOL_NS_GPL (pci_print_aer,CXL );/** * add_error_device - list device to be handled
@@ -1009,7 +1010,7 @@ static void aer_recover_work_func(struct work_struct *work) PCI_SLOT(entry.devfn), PCI_FUNC(entry.devfn));continue ; } -cper_print_aer (pdev, entry.severity, entry.regs); +pci_print_aer (pdev, entry.severity, entry.regs);if (entry.severity ==AER_NONFATAL )pcie_do_recovery (pdev, pci_channel_io_normal, aer_root_reset);
include/linux/aer.h +1 -1
@@ -64,7 +64,7 @@ static inline void pci_save_aer_state(struct pci_dev *dev) {}static inline void pci_restore_aer_state (struct pci_dev *dev) {}#endif -void cper_print_aer (struct pci_dev *dev,int aer_severity, +void pci_print_aer (struct pci_dev *dev,int aer_severity,struct aer_capability_regs *aer);int cper_severity_to_aer (int cper_severity); void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
Signature · 1 line
2.34.1
RE: [PATCH v5 20/26] PCI/AER: Refactor cper_print_aer() for use by CXL driver module
Quoted 14 lines · Terry Bowman
The CXL driver plans to use
cper_print_aer()for logging restricted CXL host (RCH) AER errors.cper_print_aer()is not currently exported and therefore not usable by the CXL drivers built as loadable modules. Export thecper_print_aer()function. Use theEXPORT_SYMBOL_NS_GPL()variant to restrict the export to CXL drivers.The
CONFIG_ACPI_APEI_PCIEAERkernel config is currently used to enablecper_print_aer().cper_print_aer()logs the AER registers and is useful in PCIE AER logging outside of APEI. Remove theCONFIG_ACPI_APEI_PCIEAERdependency to enablecper_print_aer().The
cper_print_aer()function name implies CPER specific use but is useful in non-CPER cases as well. Renamecper_print_aer()topci_print_aer().
Quoted 9 lines
Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Cc Mahesh J Salgaonkar <[email protected]>Cc "Oliver O'Halloran" <[email protected]>Cc Bjorn Helgaas <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>Acked-by Bjorn Helgaas <[email protected]>
Looks good to me.
PATCH v5 21/26 cxl/pci: Update CXL error logging to use RAS register address
In reply to Terry BowmanThe CXL error handler currently only logs endpoint RAS status. The CXL topology includes several components providing RAS details to be logged during error handling.[1] Update the current handler's RAS logging to use a RAS register address. This will allow for adding support to log other CXL component's RAS details in the future.
[1] CXL3.0 Table 8-22 CXL_Capability_ID Assignment
--- drivers/cxl/core/pci.c | 42 ++++++++++++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 12 deletions(-)
drivers/cxl/core/pci.c +30 -12
@@ -665,32 +665,36 @@ void read_cdat_data(struct cxl_port *port) }EXPORT_SYMBOL_NS_GPL (read_cdat_data,CXL ); -void cxl_cor_error_detected (struct pci_dev *pdev) +static void __cxl_log_correctable_ras (struct cxl_dev_state *cxlds, +void __iomem *ras_base) { -struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);void __iomem *addr;u32 status; -if (!cxlds->regs.ras) +if (!ras_base)return ; - addr = cxlds->regs.ras +CXL_RAS_CORRECTABLE_STATUS_OFFSET ; + addr = ras_base +CXL_RAS_CORRECTABLE_STATUS_OFFSET ; status =readl (addr);if (status &CXL_RAS_CORRECTABLE_STATUS_MASK ) {writel (status &CXL_RAS_CORRECTABLE_STATUS_MASK , addr);trace_cxl_aer_correctable_error (cxlds->cxlmd, status); } } -EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL ); + +static void cxl_log_correctable_ras_endpoint (struct cxl_dev_state *cxlds) +{ +return __cxl_log_correctable_ras (cxlds, cxlds->regs.ras); +}/* CXL spec rev3.0 8.2.4.16.1 */ -static void header_log_copy (struct cxl_dev_state *cxlds,u32 *log) +static void header_log_copy (void __iomem *ras_base,u32 *log) {void __iomem *addr;u32 *log_addr;int i, log_u32_size =CXL_HEADERLOG_SIZE /sizeof (u32); - addr = cxlds->regs.ras +CXL_RAS_HEADER_LOG_OFFSET ; + addr = ras_base +CXL_RAS_HEADER_LOG_OFFSET ; log_addr = log;for (i =0 ; i < log_u32_size; i++) {
@@ -704,17 +708,18 @@ static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log)* Log the state of the RAS status registers and prepare them to log the * next error status. Return 1 if reset needed. */ -static bool cxl_report_and_clear (struct cxl_dev_state *cxlds) +static bool __cxl_report_and_clear (struct cxl_dev_state *cxlds, +void __iomem *ras_base) {u32 hl[CXL_HEADERLOG_SIZE_U32 ];void __iomem *addr;u32 status;u32 fe; -if (!cxlds->regs.ras) +if (!ras_base)return false; - addr = cxlds->regs.ras +CXL_RAS_UNCORRECTABLE_STATUS_OFFSET ; + addr = ras_base +CXL_RAS_UNCORRECTABLE_STATUS_OFFSET ; status =readl (addr);if (!(status &CXL_RAS_UNCORRECTABLE_STATUS_MASK ))return false;
@@ -722,7 +727,7 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)/* If multiple errors, log header points to first error from ctrl reg */ if (hweight32 (status) >1 ) {void __iomem *rcc_addr = - cxlds->regs.ras +CXL_RAS_CAP_CONTROL_OFFSET ; + ras_base +CXL_RAS_CAP_CONTROL_OFFSET ; fe =BIT (FIELD_GET (CXL_RAS_CAP_CONTROL_FE_MASK ,readl (rcc_addr)));
@@ -730,13 +735,26 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) fe = status; } -header_log_copy (cxlds, hl); +header_log_copy (ras_base, hl);trace_cxl_aer_uncorrectable_error (cxlds->cxlmd, status, fe, hl);writel (status &CXL_RAS_UNCORRECTABLE_STATUS_MASK , addr);return true; } +static bool cxl_report_and_clear (struct cxl_dev_state *cxlds) +{ +return __cxl_report_and_clear (cxlds, cxlds->regs.ras); +} + +void cxl_cor_error_detected (struct pci_dev *pdev) +{ +struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); + +cxl_log_correctable_ras_endpoint (cxlds); +} +EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL ); +pci_ers_result_t cxl_error_detected (struct pci_dev *pdev,pci_channel_state_t state) {
Signature · 1 line
2.34.1
RE: [PATCH v5 21/26] cxl/pci: Update CXL error logging to use RAS register address
Quoted 6 lines · Terry Bowman
The CXL error handler currently only logs endpoint RAS status. The CXL topology includes several components providing RAS details to be logged during error handling.[1] Update the current handler's RAS logging to use a RAS register address. This will allow for adding support to log other CXL component's RAS details in the future.
Quoted 6 lines
[1] CXL3.0 Table 8-22
CXL_Capability_IDAssignmentCo-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Looks good.
RE: [PATCH v5 21/26] cxl/pci: Update CXL error logging to use RAS register address
In reply to Terry BowmanQuoted 40 lines · Terry Bowman
The CXL error handler currently only logs endpoint RAS status. The CXL topology includes several components providing RAS details to be logged during error handling.[1] Update the current handler's RAS logging to use a RAS register address. This will allow for adding support to log other CXL component's RAS details in the future.
[1] CXL3.0 Table 8-22
CXL_Capability_IDAssignmentCo-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/pci.c | 42 ++++++++++++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 12 deletions(-)drivers/cxl/core/pci.c +4
-4@@ -665,32 +665,36 @@ void read_cdat_data(struct cxl_port *port) }EXPORT_SYMBOL_NS_GPL (read_cdat_data,CXL ); -void cxl_cor_error_detected (struct pci_dev *pdev) +static void __cxl_log_correctable_ras (struct cxl_dev_state *cxlds, +void __iomem *ras_base) { -struct cxl_dev_state *cxlds =pci_get_drvdata (pdev);void __iomem *addr;u32 status; -if (!cxlds->regs.ras) +if (!ras_base)return ; - addr = cxlds->regs.ras +CXL_RAS_CORRECTABLE_STATUS_OFFSET ; + addr = ras_base +CXL_RAS_CORRECTABLE_STATUS_OFFSET ; status =readl (addr);if (status &CXL_RAS_CORRECTABLE_STATUS_MASK ) {
drivers/cxl/core/pci.c void read_cdat_data(struct cxl_port *port) · line 688
Quoted 10 lines
drivers/cxl/core/pci.c +5
-1writel (status &CXL_RAS_CORRECTABLE_STATUS_MASK , addr);trace_cxl_aer_correctable_error (cxlds->cxlmd, status); } } -EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL ); + +static void cxl_log_correctable_ras_endpoint (struct cxl_dev_state *cxlds) +{ +return __cxl_log_correctable_ras (cxlds, cxlds->regs.ras); +}
As I review patch 24 it leads me back here to grumble about some naming choices. We now have:
cxl_cor_error_detected()__cxl_log_correctable_ras()cxl_report_and_clear()
Which of those clear the status? What is the difference between "report" and a "log"? I don't much care what name gets chosen but these three functions at least need to give the impression they were written with a common vision. The term "handle" would not surprise me in the names of functions that emit messages and clear register status bits.
PATCH v5 22/26 cxl/pci: Map RCH downstream AER registers for logging protocol errors
In reply to Terry BowmanThe restricted CXL host (RCH) error handler will log protocol errors using AER and RAS status registers. The AER and RAS registers need to be virtually memory mapped before enabling interrupts. Update __devm_cxl_add_dport() to include RCH RAS and AER mapping.
Add 'struct cxl_regs' to 'struct cxl_dport' for saving a unique copy of the RCH downstream port's mapped registers.
--- drivers/cxl/core/port.c | 38 ++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/regs.c | 1 + drivers/cxl/cxl.h | 11 +++++++++++ 3 files changed, 50 insertions(+)
drivers/cxl/core/port.c +38 -0
@@ -8,6 +8,7 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> +#include <linux/aer.h> #include <cxlmem.h> #include <cxlpci.h> #include <cxl.h>
@@ -947,6 +948,39 @@ static void cxl_dport_unlink(void *data)sysfs_remove_link (&port->dev.kobj, link_name); } +static int cxl_dport_map_rch_aer (struct cxl_dport *dport) +{ +struct cxl_rcrb_info *ri = &dport->rcrb; +resource_size_t aer_phys; +void __iomem *dport_aer; + +if (!dport->rch || !ri->aer_cap) +return -ENODEV ; + + aer_phys = ri->aer_cap + ri->base; + dport_aer =devm_cxl_iomap_block (dport->dev, aer_phys, +sizeof (struct aer_capability_regs )); +if (!dport_aer) +return -ENOMEM ; + + dport->regs.dport_aer = dport_aer; + +return 0 ; +} + +static int cxl_dport_map_regs (struct cxl_dport *dport) +{ +struct cxl_register_map *map = &dport->comp_map; + +if (!map->component_map.ras.valid) +dev_dbg (map->dev,"RAS registers not found\n" ); +else if (cxl_map_component_regs (map, &dport->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ))) +dev_dbg (dport->dev,"Failed to map RAS capability.\n" ); + +return cxl_dport_map_rch_aer (dport); +} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,
@@ -1000,6 +1034,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc); + rc =cxl_dport_map_regs (dport); +if (rc && rc != -ENODEV ) +return ERR_PTR (rc); +cond_cxl_root_lock (port); rc =add_dport (port, dport);cond_cxl_root_unlock (port);
drivers/cxl/core/regs.c +1 -0
@@ -198,6 +198,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct cxl_register_map *map, struct cxl_component_regs *regs,
drivers/cxl/cxl.h +11 -0
@@ -209,6 +209,13 @@ struct cxl_regs {struct_group_tagged (cxl_device_regs, device_regs, void __iomem *status, *mbox, *memdev; ); +/* +* RCH downstream port specific RAS register +* @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB +*/ +struct_group_tagged (cxl_rch_regs, rch_regs, + void __iomem *dport_aer; + ); };struct cxl_reg_map {
@@ -255,6 +262,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);int cxl_map_component_regs (struct cxl_register_map *map,struct cxl_component_regs *regs,unsigned long map_mask);
@@ -603,6 +612,7 @@ struct cxl_rcrb_info {* @port_id: unique hardware identifier for dport in decoder target list * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @rcrb: Data about the Root Complex Register Block layout +* @regs: Dport parsed register blocks */ struct cxl_dport {struct device *dev;
@@ -611,6 +621,7 @@ struct cxl_dport {int port_id;bool rch;struct cxl_rcrb_info rcrb; +struct cxl_regs regs; };/**
Signature · 1 line
2.34.1
RE: [PATCH v5 22/26] cxl/pci: Map RCH downstream AER registers for logging protocol errors
Quoted 35 lines · Terry Bowman
The restricted CXL host (RCH) error handler will log protocol errors using AER and RAS status registers. The AER and RAS registers need to be virtually memory mapped before enabling interrupts. Update
__devm_cxl_add_dport()to include RCH RAS and AER mapping.Add 'struct
cxl_regs' to 'structcxl_dport' for saving a unique copy of the RCH downstream port's mapped registers.Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/port.c | 38 ++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/regs.c | 1 + drivers/cxl/cxl.h | 11 +++++++++++ 3 files changed, 50 insertions(+)drivers/cxl/core/port.c +3
-0@@ -8,6 +8,7 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> +#include <linux/aer.h> #include <cxlmem.h> #include <cxlpci.h> #include <cxl.h> @@ -947,6 +948,39 @@ static void cxl_dport_unlink(void *data)sysfs_remove_link (&port->dev.kobj, link_name); } +static int cxl_dport_map_rch_aer (struct cxl_dport *dport) +{
drivers/cxl/core/port.c static void cxl_dport_unlink(void *data) · line 962
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+struct cxl_rcrb_info *ri = &dport->rcrb; +resource_size_t aer_phys; +void __iomem *dport_aer; + +if (!dport->rch || !ri->aer_cap) +return -ENODEV ; + + aer_phys = ri->aer_cap + ri->base; + dport_aer =devm_cxl_iomap_block (dport->dev, aer_phys, +sizeof (struct aer_capability_regs ));
@dport->dev is not suitable to be the @host argument to devm_cxl_iomap_block(). It needs to match the lifetime of the @dport allocation which means @host needs to be set to @port->dev.
Quoted 25 lines
drivers/cxl/core/port.c +21
-0+if (!dport_aer) +return -ENOMEM ; + + dport->regs.dport_aer = dport_aer; + +return 0 ; +} + +static int cxl_dport_map_regs (struct cxl_dport *dport) +{ +struct cxl_register_map *map = &dport->comp_map; + +if (!map->component_map.ras.valid) +dev_dbg (map->dev,"RAS registers not found\n" ); +else if (cxl_map_component_regs (map, &dport->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ))) +dev_dbg (dport->dev,"Failed to map RAS capability.\n" ); + +return cxl_dport_map_rch_aer (dport); +} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,
drivers/cxl/core/port.c __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, · line 1039
Quoted 6 lines
drivers/cxl/core/port.c +3
-0@@ -1000,6 +1034,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc); + rc =cxl_dport_map_regs (dport); +if (rc && rc != -ENODEV ) +return ERR_PTR (rc);
I'll repeat the previous comment about replacing:
if (rc && rc != -ENODEV)
...with an optional initialization at alloc time.
Quoted 62 lines
drivers/cxl/core/port.c +1
-0+cond_cxl_root_lock (port); rc =add_dport (port, dport);cond_cxl_root_unlock (port);drivers/cxl/core/regs.c +1
-0@@ -198,6 +198,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct cxl_register_map *map, struct cxl_component_regs *regs,drivers/cxl/cxl.h +11
-0@@ -209,6 +209,13 @@ struct cxl_regs {struct_group_tagged (cxl_device_regs, device_regs, void __iomem *status, *mbox, *memdev; ); +/* +* RCH downstream port specific RAS register +* @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB +*/ +struct_group_tagged (cxl_rch_regs, rch_regs, + void __iomem *dport_aer; + ); };struct cxl_reg_map {@@ -255,6 +262,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);int cxl_map_component_regs (struct cxl_register_map *map,struct cxl_component_regs *regs,unsigned long map_mask);@@ -603,6 +612,7 @@ struct cxl_rcrb_info {* @port_id: unique hardware identifier for dport in decoder target list * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @rcrb: Data about the Root Complex Register Block layout +* @regs: Dport parsed register blocks */ struct cxl_dport {struct device *dev;@@ -611,6 +621,7 @@ struct cxl_dport {int port_id;bool rch;struct cxl_rcrb_info rcrb; +struct cxl_regs regs; };/** Signature · 1 line
2.34.1
Re: [PATCH v5 22/26] cxl/pci: Map RCH downstream AER registers for logging protocol errors
Hi Dan,
I'll make the updates you recommended below.
Regards,
Terry
Quoted 154 lines · Dan Williams
Quoted 35 lines · Terry Bowman
The restricted CXL host (RCH) error handler will log protocol errors using AER and RAS status registers. The AER and RAS registers need to be virtually memory mapped before enabling interrupts. Update
__devm_cxl_add_dport()to include RCH RAS and AER mapping.Add 'struct
cxl_regs' to 'structcxl_dport' for saving a unique copy of the RCH downstream port's mapped registers.Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/port.c | 38 ++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/regs.c | 1 + drivers/cxl/cxl.h | 11 +++++++++++ 3 files changed, 50 insertions(+)drivers/cxl/core/port.c +3
-0@@ -8,6 +8,7 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> +#include <linux/aer.h> #include <cxlmem.h> #include <cxlpci.h> #include <cxl.h> @@ -947,6 +948,39 @@ static void cxl_dport_unlink(void *data)sysfs_remove_link (&port->dev.kobj, link_name); } +static int cxl_dport_map_rch_aer (struct cxl_dport *dport) +{drivers/cxl/core/port.c static void cxl_dport_unlink(void *data) · line 962
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+struct cxl_rcrb_info *ri = &dport->rcrb; +resource_size_t aer_phys; +void __iomem *dport_aer; + +if (!dport->rch || !ri->aer_cap) +return -ENODEV ; + + aer_phys = ri->aer_cap + ri->base; + dport_aer =devm_cxl_iomap_block (dport->dev, aer_phys, +sizeof (struct aer_capability_regs ));@dport->dev is not suitable to be the @host argument to
devm_cxl_iomap_block(). It needs to match the lifetime of the @dport allocation which means @host needs to be set to @port->dev.Quoted 25 lines
drivers/cxl/core/port.c +21
-0+if (!dport_aer) +return -ENOMEM ; + + dport->regs.dport_aer = dport_aer; + +return 0 ; +} + +static int cxl_dport_map_regs (struct cxl_dport *dport) +{ +struct cxl_register_map *map = &dport->comp_map; + +if (!map->component_map.ras.valid) +dev_dbg (map->dev,"RAS registers not found\n" ); +else if (cxl_map_component_regs (map, &dport->regs.component, +BIT (CXL_CM_CAP_CAP_ID_RAS ))) +dev_dbg (dport->dev,"Failed to map RAS capability.\n" ); + +return cxl_dport_map_rch_aer (dport); +} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,drivers/cxl/core/port.c __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, · line 1039
Quoted 6 lines
drivers/cxl/core/port.c +3
-0@@ -1000,6 +1034,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc); + rc =cxl_dport_map_regs (dport); +if (rc && rc != -ENODEV ) +return ERR_PTR (rc);I'll repeat the previous comment about replacing:
if (rc && rc != -ENODEV)...with an optional initialization at alloc time.
Quoted 62 lines
drivers/cxl/core/port.c +1
-0+cond_cxl_root_lock (port); rc =add_dport (port, dport);cond_cxl_root_unlock (port);drivers/cxl/core/regs.c +1
-0@@ -198,6 +198,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct cxl_register_map *map, struct cxl_component_regs *regs,drivers/cxl/cxl.h +11
-0@@ -209,6 +209,13 @@ struct cxl_regs {struct_group_tagged (cxl_device_regs, device_regs, void __iomem *status, *mbox, *memdev; ); +/* +* RCH downstream port specific RAS register +* @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB +*/ +struct_group_tagged (cxl_rch_regs, rch_regs, + void __iomem *dport_aer; + ); };struct cxl_reg_map {@@ -255,6 +262,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map);void cxl_probe_device_regs (struct device *dev,void __iomem *base,struct cxl_device_reg_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);int cxl_map_component_regs (struct cxl_register_map *map,struct cxl_component_regs *regs,unsigned long map_mask);@@ -603,6 +612,7 @@ struct cxl_rcrb_info {* @port_id: unique hardware identifier for dport in decoder target list * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @rcrb: Data about the Root Complex Register Block layout +* @regs: Dport parsed register blocks */ struct cxl_dport {struct device *dev;@@ -611,6 +621,7 @@ struct cxl_dport {int port_id;bool rch;struct cxl_rcrb_info rcrb; +struct cxl_regs regs; };/** Signature · 1 line
2.34.1
PATCH v5 23/26 cxl/pci: Disable root port interrupts in RCH mode
In reply to Terry BowmanThe RCH root port contains root command AER registers that should not be enabled.[1] Disable these to prevent root port interrupts.
[1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors
--- drivers/cxl/core/port.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
drivers/cxl/core/port.c +26 -0
@@ -981,6 +981,30 @@ static int cxl_dport_map_regs(struct cxl_dport *dport)return cxl_dport_map_rch_aer (dport); } +static void cxl_disable_rch_root_ints (struct cxl_dport *dport) +{ +void __iomem *aer_base = dport->regs.dport_aer; +u32 aer_cmd_mask, aer_cmd; + +if (!dport->rch || !aer_base) +return ; + +/* +* Disable RCH root port command interrupts. +* CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors +* +* This sequnce may not be necessary. CXL spec states disabling +* the root cmd register's interrupts is required. But, PCI spec +* shows these are disabled by default on reset. +*/ + aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN | +PCI_ERR_ROOT_CMD_NONFATAL_EN | +PCI_ERR_ROOT_CMD_FATAL_EN ); + aer_cmd =readl (aer_base +PCI_ERR_ROOT_COMMAND ); + aer_cmd &= ~aer_cmd_mask; +writel (aer_cmd, aer_base +PCI_ERR_ROOT_COMMAND ); +} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,
@@ -1038,6 +1062,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc); +cxl_disable_rch_root_ints (dport); +cond_cxl_root_lock (port); rc =add_dport (port, dport);cond_cxl_root_unlock (port);
Signature · 1 line
2.34.1
RE: [PATCH v5 23/26] cxl/pci: Disable root port interrupts in RCH mode
Quoted 15 lines · Terry Bowman
The RCH root port contains root command AER registers that should not be enabled.[1] Disable these to prevent root port interrupts.
[1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors
Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/port.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)drivers/cxl/core/port.c +0
-0
drivers/cxl/core/port.c static int cxl_dport_map_regs(struct cxl_dport *dport) · line 990
Quoted 10 lines
drivers/cxl/core/port.c +7
-0@@ -981,6 +981,30 @@ static int cxl_dport_map_regs(struct cxl_dport *dport)return cxl_dport_map_rch_aer (dport); } +static void cxl_disable_rch_root_ints (struct cxl_dport *dport) +{ +void __iomem *aer_base = dport->regs.dport_aer; +u32 aer_cmd_mask, aer_cmd; + +if (!dport->rch || !aer_base) +return ;
Does this need to check ->rch? There is no path that sets ->regs.dport_aer in the VH case, right?
Quoted 5 lines
drivers/cxl/core/port.c +5
-0+ +/* +* Disable RCH root port command interrupts. +* CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors +*
drivers/cxl/core/port.c static int cxl_dport_map_regs(struct cxl_dport *dport) · line 1005
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+* This sequnce may not be necessary. CXL spec states disabling +* the root cmd register's interrupts is required. But, PCI spec +* shows these are disabled by default on reset. +*/ + aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN | +PCI_ERR_ROOT_CMD_NONFATAL_EN | +PCI_ERR_ROOT_CMD_FATAL_EN ); + aer_cmd =readl (aer_base +PCI_ERR_ROOT_COMMAND ); + aer_cmd &= ~aer_cmd_mask; +writel (aer_cmd, aer_base +PCI_ERR_ROOT_COMMAND );
What about the scenario where @host_bridge->native_cxl_error is false? Should the driver unconditionally touch AER registers?
drivers/cxl/core/port.c __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, · line 1066
Quoted 11 lines
drivers/cxl/core/port.c +4
-0+} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,@@ -1038,6 +1062,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc); +cxl_disable_rch_root_ints (dport); +
It feels odd to modify hardware state within an object setup helper. Is there a reason this needs to be settled before __devm_cxl_add_dport() returns? If not this feels like a helper that cxl_acpi should call to change the state of the @dport instance it allocated.
Re: [PATCH v5 23/26] cxl/pci: Disable root port interrupts in RCH mode
Hi Dan,
Thanks for the review.
Quoted 19 lines · Dan Williams
Quoted 18 lines · Terry Bowman
The RCH root port contains root command AER registers that should not be enabled.[1] Disable these to prevent root port interrupts.
[1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors
Signed-off-by Terry Bowman <[email protected]>--- drivers/cxl/core/port.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)drivers/cxl/core/port.c +0
-0@@ -981,6 +981,30 @@ static int cxl_dport_map_regs(struct cxl_dport *dport)return cxl_dport_map_rch_aer (dport); }
Quoted 11 lines
drivers/cxl/core/port.c static int cxl_dport_map_regs(struct cxl_dport *dport) · line 990
Quoted 7 lines
drivers/cxl/core/port.c +7
-0+static void cxl_disable_rch_root_ints (struct cxl_dport *dport) +{ +void __iomem *aer_base = dport->regs.dport_aer; +u32 aer_cmd_mask, aer_cmd; + +if (!dport->rch || !aer_base) +return ;Does this need to check ->rch? There is no path that sets ->regs.dport_aer in the VH case, right?
Correct. This can be simplified to only check !aer_base because aer_base is only set if dport->rch.
Quoted 10 lines
Quoted 10 lines
drivers/cxl/core/port.c +10
-0+ +/* +* Disable RCH root port command interrupts. +* CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors +* +* This sequnce may not be necessary. CXL spec states disabling +* the root cmd register's interrupts is required. But, PCI spec +* shows these are disabled by default on reset. +*/ + aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN |
Quoted 9 lines
drivers/cxl/core/port.c static int cxl_dport_map_regs(struct cxl_dport *dport) · line 1005
Quoted 5 lines
drivers/cxl/core/port.c +5
-0+PCI_ERR_ROOT_CMD_NONFATAL_EN | + PCI_ERR_ROOT_CMD_FATAL_EN); + aer_cmd =readl (aer_base +PCI_ERR_ROOT_COMMAND ); + aer_cmd &= ~aer_cmd_mask; +writel (aer_cmd, aer_base +PCI_ERR_ROOT_COMMAND );What about the scenario where @host_bridge->native_cxl_error is false? Should the driver unconditionally touch AER registers?
Right. The driver should only touch the AER registers if the AER control is OSC negotiated to OS. I'll add check for @host_bridge->native_cxl_error.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/port.c +2
-0+} +static struct cxl_dport * __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id, resource_size_t component_reg_phys,@@ -1038,6 +1062,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,if (rc && rc != -ENODEV )return ERR_PTR (rc);
Quoted 7 lines
drivers/cxl/core/port.c __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, · line 1066
Quoted 2 lines
drivers/cxl/core/port.c +2
-0+cxl_disable_rch_root_ints (dport); +It feels odd to modify hardware state within an object setup helper. Is there a reason this needs to be settled before
__devm_cxl_add_dport()returns? If not this feels like a helper thatcxl_acpishould call to change the state of the @dport instance it allocated.
Sure. This can be moved into the caller, add_host_bridge_dport(), after calling devm_cxl_add_rch_dport().
Regards,
Terry
PATCH v5 24/26 cxl/pci: Add RCH downstream port error logging
In reply to Terry BowmanRCH downstream port error logging is missing in the current CXL driver. The missing AER and RAS error logging is needed for communicating driver error details to userspace. Update the driver to include PCIe AER and CXL RAS error logging.
Add RCH downstream port error handling into the existing RCiEP handler. The downstream port error handler is added to the RCiEP error handler because the downstream port is implemented in a RCRB, is not PCI enumerable, and as a result is not directly accessible to the PCI AER root port driver. The AER root port driver calls the RCiEP handler for handling RCD errors and RCH downstream port protocol errors.
Update existing RCiEP correctable and uncorrectable handlers to also call the RCH handler. The RCH handler will read the RCH AER registers, check for error severity, and if an error exists will log using an existing kernel AER trace routine. The RCH handler will also log downstream port RAS errors if they exist.
--- drivers/cxl/core/pci.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+)
drivers/cxl/core/pci.c +98 -0
@@ -5,6 +5,7 @@#include <linux/delay.h> #include <linux/pci.h> #include <linux/pci-doe.h> +#include <linux/aer.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h>
@@ -747,10 +748,105 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)return __cxl_report_and_clear (cxlds, cxlds->regs.ras); } +#ifdef CONFIG_PCIEAER_CXL + +static void cxl_log_correctable_ras_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_log_correctable_ras (cxlds, dport->regs.ras); +} + +static bool cxl_report_and_clear_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_report_and_clear (cxlds, dport->regs.ras); +} + +/* +* Copy the AER capability registers using 32 bit read accesses. +* This is necessary because RCRB AER capability is MMIO mapped. Clear the +* status after copying. +* +* @aer_base: base address of AER capability block in RCRB +* @aer_regs: destination for copying AER capability +*/ +static bool cxl_rch_get_aer_info (void __iomem *aer_base, +struct aer_capability_regs *aer_regs) +{ +int read_cnt =sizeof (struct aer_capability_regs ) /sizeof (u32); +u32 *aer_regs_buf = (u32 *)aer_regs; +int n; + +if (!aer_base) +return false; + +/* Use readl() to guarantee 32-bit accesses */ +for (n =0 ; n < read_cnt; n++) + aer_regs_buf[n] =readl (aer_base + n *sizeof (u32)); + +writel (aer_regs->uncor_status, aer_base +PCI_ERR_UNCOR_STATUS ); +writel (aer_regs->cor_status, aer_base +PCI_ERR_COR_STATUS ); + +return true; +} + +/* Get AER severity. Return false if there is no error. */ +static bool cxl_rch_get_aer_severity (struct aer_capability_regs *aer_regs, +int *severity) +{ +if (aer_regs->uncor_status & ~aer_regs->uncor_mask) { +if (aer_regs->uncor_status &PCI_ERR_ROOT_FATAL_RCV ) + *severity =AER_FATAL ; +else + *severity =AER_NONFATAL ; +return true; + } + +if (aer_regs->cor_status & ~aer_regs->cor_mask) { + *severity =AER_CORRECTABLE ; +return true; + } + +return false; +} + +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +struct aer_capability_regs aer_regs; +struct cxl_dport *dport; +int severity; + +if (!cxlds->rcd) +return ; + +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return ; + +if (!cxl_rch_get_aer_info (dport->regs.dport_aer, &aer_regs)) +return ; + +if (!cxl_rch_get_aer_severity (&aer_regs, &severity)) +return ; + +pci_print_aer (pdev, severity, &aer_regs); + +if (severity ==AER_CORRECTABLE ) +cxl_log_correctable_ras_dport (cxlds, dport); +else +cxl_report_and_clear_dport (cxlds, dport); +} + +#else +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) { } +#endif +void cxl_cor_error_detected (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +cxl_handle_rch_dport_errors (cxlds); +cxl_log_correctable_ras_endpoint (cxlds); }EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL );
@@ -763,6 +859,8 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,struct device *dev = &cxlmd->dev;bool ue; +cxl_handle_rch_dport_errors (cxlds); +/* * A frozen channel indicates an impending reset which is fatal to * CXL.mem operation, and will likely crash the system. On the off
Signature · 1 line
2.34.1
RE: [PATCH v5 24/26] cxl/pci: Add RCH downstream port error logging
Quoted 39 lines · Terry Bowman
RCH downstream port error logging is missing in the current CXL driver. The missing AER and RAS error logging is needed for communicating driver error details to userspace. Update the driver to include PCIe AER and CXL RAS error logging.
Add RCH downstream port error handling into the existing RCiEP handler. The downstream port error handler is added to the RCiEP error handler because the downstream port is implemented in a RCRB, is not PCI enumerable, and as a result is not directly accessible to the PCI AER root port driver. The AER root port driver calls the RCiEP handler for handling RCD errors and RCH downstream port protocol errors.
Update existing RCiEP correctable and uncorrectable handlers to also call the RCH handler. The RCH handler will read the RCH AER registers, check for error severity, and if an error exists will log using an existing kernel AER trace routine. The RCH handler will also log downstream port RAS errors if they exist.
Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/pci.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+)drivers/cxl/core/pci.c +1
-0@@ -5,6 +5,7 @@#include <linux/delay.h> #include <linux/pci.h> #include <linux/pci-doe.h> +#include <linux/aer.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h>
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 751
Quoted 4 lines
drivers/cxl/core/pci.c +1
-0@@ -747,10 +748,105 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)return __cxl_report_and_clear (cxlds, cxlds->regs.ras); } +#ifdef CONFIG_PCIEAER_CXL
A general reaction to the "ifdef in a .c file" style recommendation. Maybe this section could move to a drivers/cxl/core/aer.c file, and be optionally compiled by config in the Makefile? I.e. similar to:
cxl_core-$(CONFIG_TRACING) += trace.o
cxl_core-$(CONFIG_CXL_REGION) += region.o
...it is borderline just big enough, but I'll leave it up to you.
Quoted 60 lines
drivers/cxl/core/pci.c +60
-0+ +static void cxl_log_correctable_ras_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_log_correctable_ras (cxlds, dport->regs.ras); +} + +static bool cxl_report_and_clear_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_report_and_clear (cxlds, dport->regs.ras); +} + +/* +* Copy the AER capability registers using 32 bit read accesses. +* This is necessary because RCRB AER capability is MMIO mapped. Clear the +* status after copying. +* +* @aer_base: base address of AER capability block in RCRB +* @aer_regs: destination for copying AER capability +*/ +static bool cxl_rch_get_aer_info (void __iomem *aer_base, +struct aer_capability_regs *aer_regs) +{ +int read_cnt =sizeof (struct aer_capability_regs ) /sizeof (u32); +u32 *aer_regs_buf = (u32 *)aer_regs; +int n; + +if (!aer_base) +return false; + +/* Use readl() to guarantee 32-bit accesses */ +for (n =0 ; n < read_cnt; n++) + aer_regs_buf[n] =readl (aer_base + n *sizeof (u32)); + +writel (aer_regs->uncor_status, aer_base +PCI_ERR_UNCOR_STATUS ); +writel (aer_regs->cor_status, aer_base +PCI_ERR_COR_STATUS ); + +return true; +} + +/* Get AER severity. Return false if there is no error. */ +static bool cxl_rch_get_aer_severity (struct aer_capability_regs *aer_regs, +int *severity) +{ +if (aer_regs->uncor_status & ~aer_regs->uncor_mask) { +if (aer_regs->uncor_status &PCI_ERR_ROOT_FATAL_RCV ) + *severity =AER_FATAL ; +else + *severity =AER_NONFATAL ; +return true; + } + +if (aer_regs->cor_status & ~aer_regs->cor_mask) { + *severity =AER_CORRECTABLE ; +return true; + } + +return false; +}
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 821
Quoted 10 lines
drivers/cxl/core/pci.c +10
-0+ +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +struct aer_capability_regs aer_regs; +struct cxl_dport *dport; +int severity; + +if (!cxlds->rcd) +return ;
Small quibble, but I think this check belongs in the caller.
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 824
Quoted 3 lines
drivers/cxl/core/pci.c +3
-0+ +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return ;
The reference for the @port return from cxl_pci_find_port() is leaked here.
How can dport->rch be false while cxlds->rcd is true? Is that check required?
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 837
Quoted 13 lines
drivers/cxl/core/pci.c +13
-0+ +if (!cxl_rch_get_aer_info (dport->regs.dport_aer, &aer_regs)) +return ; + +if (!cxl_rch_get_aer_severity (&aer_regs, &severity)) +return ; + +pci_print_aer (pdev, severity, &aer_regs); + +if (severity ==AER_CORRECTABLE ) +cxl_log_correctable_ras_dport (cxlds, dport); +else +cxl_report_and_clear_dport (cxlds, dport);
This is the code that made me go back and have heartburn about the naming choices. I.e. would a casual reader assume that correctable errors are not cleared, and that reporting is different than logging?
Quoted 16 lines
drivers/cxl/core/pci.c +8
-0+} + +#else +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) { } +#endif +void cxl_cor_error_detected (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +cxl_handle_rch_dport_errors (cxlds); +cxl_log_correctable_ras_endpoint (cxlds); }EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL );
drivers/cxl/core/pci.c pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, · line 862
Quoted 4 lines
drivers/cxl/core/pci.c +1
-0@@ -763,6 +859,8 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,struct device *dev = &cxlmd->dev;bool ue; +cxl_handle_rch_dport_errors (cxlds);
Per above comment on "cxlds->rcd" conditional, it is mildly surprising that even the VH path calls this helper.
Re: [PATCH v5 24/26] cxl/pci: Add RCH downstream port error logging
Hi Dan,
I added responses below.
Quoted 49 lines · Dan Williams
Quoted 39 lines · Terry Bowman
RCH downstream port error logging is missing in the current CXL driver. The missing AER and RAS error logging is needed for communicating driver error details to userspace. Update the driver to include PCIe AER and CXL RAS error logging.
Add RCH downstream port error handling into the existing RCiEP handler. The downstream port error handler is added to the RCiEP error handler because the downstream port is implemented in a RCRB, is not PCI enumerable, and as a result is not directly accessible to the PCI AER root port driver. The AER root port driver calls the RCiEP handler for handling RCD errors and RCH downstream port protocol errors.
Update existing RCiEP correctable and uncorrectable handlers to also call the RCH handler. The RCH handler will read the RCH AER registers, check for error severity, and if an error exists will log using an existing kernel AER trace routine. The RCH handler will also log downstream port RAS errors if they exist.
Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/pci.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+)drivers/cxl/core/pci.c +1
-0@@ -5,6 +5,7 @@#include <linux/delay.h> #include <linux/pci.h> #include <linux/pci-doe.h> +#include <linux/aer.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h> drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 751
Quoted 4 lines
drivers/cxl/core/pci.c +1
-0@@ -747,10 +748,105 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)return __cxl_report_and_clear (cxlds, cxlds->regs.ras); } +#ifdef CONFIG_PCIEAER_CXL A general reaction to the "ifdef in a .c file" style recommendation. Maybe this section could move to a
drivers/cxl/core/aer.cfile, and be optionally compiled by config in theMakefile? I.e. similar to:
Quoted 5 lines
cxl_core-$(CONFIG_TRACING) += trace.o
cxl_core-$(CONFIG_CXL_REGION) += region.o...it is borderline just big enough, but I'll leave it up to you.
I'll take a look at this. We have most of the patchset requests implplemented and will give me time to look at this.
Quoted 64 lines
Quoted 64 lines
drivers/cxl/core/pci.c +64
-0+ +static void cxl_log_correctable_ras_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_log_correctable_ras (cxlds, dport->regs.ras); +} + +static bool cxl_report_and_clear_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_report_and_clear (cxlds, dport->regs.ras); +} + +/* +* Copy the AER capability registers using 32 bit read accesses. +* This is necessary because RCRB AER capability is MMIO mapped. Clear the +* status after copying. +* +* @aer_base: base address of AER capability block in RCRB +* @aer_regs: destination for copying AER capability +*/ +static bool cxl_rch_get_aer_info (void __iomem *aer_base, +struct aer_capability_regs *aer_regs) +{ +int read_cnt =sizeof (struct aer_capability_regs ) /sizeof (u32); +u32 *aer_regs_buf = (u32 *)aer_regs; +int n; + +if (!aer_base) +return false; + +/* Use readl() to guarantee 32-bit accesses */ +for (n =0 ; n < read_cnt; n++) + aer_regs_buf[n] =readl (aer_base + n *sizeof (u32)); + +writel (aer_regs->uncor_status, aer_base +PCI_ERR_UNCOR_STATUS ); +writel (aer_regs->cor_status, aer_base +PCI_ERR_COR_STATUS ); + +return true; +} + +/* Get AER severity. Return false if there is no error. */ +static bool cxl_rch_get_aer_severity (struct aer_capability_regs *aer_regs, +int *severity) +{ +if (aer_regs->uncor_status & ~aer_regs->uncor_mask) { +if (aer_regs->uncor_status &PCI_ERR_ROOT_FATAL_RCV ) + *severity =AER_FATAL ; +else + *severity =AER_NONFATAL ; +return true; + } + +if (aer_regs->cor_status & ~aer_regs->cor_mask) { + *severity =AER_CORRECTABLE ; +return true; + } + +return false; +} + +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev);
Quoted 9 lines
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 821
Quoted 6 lines
drivers/cxl/core/pci.c +6
-0+struct aer_capability_regs aer_regs; +struct cxl_dport *dport; +int severity; + +if (!cxlds->rcd) +return ;Small quibble, but I think this check belongs in the caller.
Ok.
Quoted 10 lines
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 824
Quoted 3 lines
drivers/cxl/core/pci.c +3
-0+ +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return ;The reference for the @port return from
cxl_pci_find_port()is leaked here.How can
dport->rchbe false whilecxlds->rcdis true? Is that check required?
I will remove the rch check.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/pci.c +9
-0+ +if (!cxl_rch_get_aer_info (dport->regs.dport_aer, &aer_regs)) +return ; + +if (!cxl_rch_get_aer_severity (&aer_regs, &severity)) +return ; + +pci_print_aer (pdev, severity, &aer_regs); +
Quoted 9 lines
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 837
Quoted 4 lines
drivers/cxl/core/pci.c +4
-0+if (severity ==AER_CORRECTABLE ) +cxl_log_correctable_ras_dport (cxlds, dport); +else +cxl_report_and_clear_dport (cxlds, dport);This is the code that made me go back and have heartburn about the naming choices. I.e. would a casual reader assume that correctable errors are not cleared, and that reporting is different than logging?
Yes, the names are ready for reworking. I have updated the functions to use consistent naming in the v6 patchset.
Quoted 19 lines
Quoted 19 lines
drivers/cxl/core/pci.c +8
-0+} + +#else +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) { } +#endif +void cxl_cor_error_detected (struct pci_dev *pdev) {struct cxl_dev_state *cxlds =pci_get_drvdata (pdev); +cxl_handle_rch_dport_errors (cxlds); +cxl_log_correctable_ras_endpoint (cxlds); }EXPORT_SYMBOL_NS_GPL (cxl_cor_error_detected,CXL );@@ -763,6 +859,8 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,struct device *dev = &cxlmd->dev;bool ue;
Quoted 4 lines
drivers/cxl/core/pci.c pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, · line 862
Quoted 1 line
drivers/cxl/core/pci.c +1
-0+cxl_handle_rch_dport_errors (cxlds);Per above comment on "
cxlds->rcd" conditional, it is mildly surprising that even the VH path calls this helper.
The 'if (cxlds->rcd)' will be moved here per your above request. Strictly speaking, this is still in the VH path but an improvement. This is really an endpoint path for RCH(RCD) and VH endpoints.
An alternative solution we considered was using a separate RCH dport error handler but that requires further AER port driver plumbing rework (for only CXL) or changing the assigned error handlers depending on RCH-VH mode at runtime. I spent time implementing and testing these options and we found it added significant complexity for a limited use case.
Regards,
Terry
Re: [PATCH v5 24/26] cxl/pci: Add RCH downstream port error logging
Hi Dan,
Quoted 138 lines · Terry Bowman
Hi Dan,
I added responses below.
Quoted 49 lines · Dan Williams
Quoted 39 lines · Terry Bowman
RCH downstream port error logging is missing in the current CXL driver. The missing AER and RAS error logging is needed for communicating driver error details to userspace. Update the driver to include PCIe AER and CXL RAS error logging.
Add RCH downstream port error handling into the existing RCiEP handler. The downstream port error handler is added to the RCiEP error handler because the downstream port is implemented in a RCRB, is not PCI enumerable, and as a result is not directly accessible to the PCI AER root port driver. The AER root port driver calls the RCiEP handler for handling RCD errors and RCH downstream port protocol errors.
Update existing RCiEP correctable and uncorrectable handlers to also call the RCH handler. The RCH handler will read the RCH AER registers, check for error severity, and if an error exists will log using an existing kernel AER trace routine. The RCH handler will also log downstream port RAS errors if they exist.
Co-developed-by Robert Richter <[email protected]>Signed-off-by Robert Richter <[email protected]>Signed-off-by Terry Bowman <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/cxl/core/pci.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+)drivers/cxl/core/pci.c +1
-0@@ -5,6 +5,7 @@#include <linux/delay.h> #include <linux/pci.h> #include <linux/pci-doe.h> +#include <linux/aer.h> #include <cxlpci.h> #include <cxlmem.h> #include <cxl.h> drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 751
Quoted 4 lines
drivers/cxl/core/pci.c +1
-0@@ -747,10 +748,105 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)return __cxl_report_and_clear (cxlds, cxlds->regs.ras); } +#ifdef CONFIG_PCIEAER_CXL A general reaction to the "ifdef in a .c file" style recommendation. Maybe this section could move to a
drivers/cxl/core/aer.cfile, and be optionally compiled by config in theMakefile? I.e. similar to:Quoted 5 lines
cxl_core-$(CONFIG_TRACING) += trace.o
cxl_core-$(CONFIG_CXL_REGION) += region.o...it is borderline just big enough, but I'll leave it up to you.
I'll take a look at this. We have most of the patchset requests implplemented and will give me time to look at this.
Quoted 64 lines
Quoted 64 lines
drivers/cxl/core/pci.c +64
-0+ +static void cxl_log_correctable_ras_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_log_correctable_ras (cxlds, dport->regs.ras); +} + +static bool cxl_report_and_clear_dport (struct cxl_dev_state *cxlds, +struct cxl_dport *dport) +{ +return __cxl_report_and_clear (cxlds, dport->regs.ras); +} + +/* +* Copy the AER capability registers using 32 bit read accesses. +* This is necessary because RCRB AER capability is MMIO mapped. Clear the +* status after copying. +* +* @aer_base: base address of AER capability block in RCRB +* @aer_regs: destination for copying AER capability +*/ +static bool cxl_rch_get_aer_info (void __iomem *aer_base, +struct aer_capability_regs *aer_regs) +{ +int read_cnt =sizeof (struct aer_capability_regs ) /sizeof (u32); +u32 *aer_regs_buf = (u32 *)aer_regs; +int n; + +if (!aer_base) +return false; + +/* Use readl() to guarantee 32-bit accesses */ +for (n =0 ; n < read_cnt; n++) + aer_regs_buf[n] =readl (aer_base + n *sizeof (u32)); + +writel (aer_regs->uncor_status, aer_base +PCI_ERR_UNCOR_STATUS ); +writel (aer_regs->cor_status, aer_base +PCI_ERR_COR_STATUS ); + +return true; +} + +/* Get AER severity. Return false if there is no error. */ +static bool cxl_rch_get_aer_severity (struct aer_capability_regs *aer_regs, +int *severity) +{ +if (aer_regs->uncor_status & ~aer_regs->uncor_mask) { +if (aer_regs->uncor_status &PCI_ERR_ROOT_FATAL_RCV ) + *severity =AER_FATAL ; +else + *severity =AER_NONFATAL ; +return true; + } + +if (aer_regs->cor_status & ~aer_regs->cor_mask) { + *severity =AER_CORRECTABLE ; +return true; + } + +return false; +} + +static void cxl_handle_rch_dport_errors (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev);Quoted 9 lines
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 821
Quoted 6 lines
drivers/cxl/core/pci.c +6
-0+struct aer_capability_regs aer_regs; +struct cxl_dport *dport; +int severity; + +if (!cxlds->rcd) +return ;Small quibble, but I think this check belongs in the caller.
Quoted 9 lines
Ok.
Quoted 7 lines
drivers/cxl/core/pci.c static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) · line 824
Quoted 3 lines
drivers/cxl/core/pci.c +3
-0+ +if (!cxl_pci_find_port (pdev, &dport) || !dport->rch) +return ;The reference for the @port return from
cxl_pci_find_port()is leaked here.
I will address this as well. Thanks for pointing this out.
Regards,
Terry
PATCH v5 25/26 PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
In Restricted CXL Device (RCD) mode a CXL device is exposed as an RCiEP, but CXL downstream and upstream ports are not enumerated and not visible in the PCIe hierarchy. [1] Protocol and link errors from these non-enumerated ports are signaled as internal AER errors, either Uncorrectable Internal Error (UIE) or Corrected Internal Errors (CIE) via an RCEC.
Restricted CXL host (RCH) downstream port-detected errors have the Requster ID of the RCEC set in the RCEC's AER Error Source ID register. A CXL handler must then inspect the error status in various CXL registers residing in the dport's component register space (CXL RAS capability) or the dport's RCRB (PCIe AER extended capability). [2]
Errors showing up in the RCEC's error handler must be handled and connected to the CXL subsystem. Implement this by forwarding the error to all CXL devices below the RCEC. Since the entire CXL device is controlled only using PCIe Configuration Space of device 0, function 0, only pass it there [3]. The error handling is limited to currently supported devices with the Memory Device class code set (CXL Type 3 Device, PCI_CLASS_MEMORY_CXL, 502h), handle downstream port errors in the device's cxl_pci driver. Support for other CXL Device Types (e.g. a CXL.cache Device) can be added later.
To handle downstream port errors in addition to errors directed to the CXL endpoint device, a handler must also inspect the CXL RAS and PCIe AER capabilities of the CXL downstream port the device is connected to.
Since CXL downstream port errors are signaled using internal errors, the handler requires those errors to be unmasked. This is subject of a follow-on patch.
The reason for choosing this implementation is that the AER service driver claims the RCEC device, but does not allow it to register a custom specific handler to support CXL. Connecting the RCEC hard-wired with a CXL handler does not work, as the CXL subsystem might not be present all the time. The alternative to add an implementation to the portdrv to allow the registration of a custom RCEC error handler isn't worth doing it as CXL would be its only user. Instead, just check for an CXL RCEC and pass it down to the connected CXL device's error handler. With this approach the code can entirely be implemented in the PCIe AER driver and is independent of the CXL subsystem. The CXL driver only provides the handler.
[1] CXL 3.0 spec: 9.11.8 CXL Devices Attached to an RCH
[2] CXL 3.0 spec, 12.2.1.1 RCH Downstream Port-detected Errors
[3] CXL 3.0 spec, 8.1.3 PCIe DVSEC for CXL Devices
--- drivers/pci/pcie/Kconfig | 12 +++++ drivers/pci/pcie/aer.c | 96 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 2 deletions(-)
drivers/pci/pcie/Kconfig +12 -0
@@ -49,6 +49,18 @@ config PCIEAER_INJECT gotten from: https://git.kernel.org/cgit/linux/kernel/git/gong.chen/aer-inject.git/ +config PCIEAER_CXL + bool "PCI Express CXL RAS support for Restricted Hosts (RCH)" + default y + depends on PCIEAER && CXL_PCI + help + Enables error handling of downstream ports of a CXL host + that is operating in RCD mode (Restricted CXL Host, RCH). + The downstream port reports AER errors to a given RCEC. + Errors are handled by the CXL memory device driver. + + If unsure, say Y. + # # PCI Express ECRC #
drivers/pci/pcie/aer.c +94 -2
114 lines not on this page · raw message
Signature · 1 line
2.34.1
kernel test robot
build botRe: [PATCH v5 25/26] PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler
kernel test robot noticed the following build errors:
Hi Terry,
kernel test robot noticed the following build errors:
[auto build test ERROR on a70fc4ed20a6118837b0aecbbf789074935f473b]
url: https://github.com/intel-lab-lkp/linux/commits/Terry-Bowman/cxl-acpi-Probe-RCRB-later-during-RCH-downstream-port-creation/20230608-062818 base: a70fc4ed20a6118837b0aecbbf789074935f473b patch link: https://lore.kernel.org/r/20230607221651.2454764-26-terry.bowman%40amd.com patch subject: [PATCH v5 25/26] PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler config: x86_64-randconfig-r005-20230607 (https://download.01.org/0day-ci/archive/20230609/[email protected]/config) compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce (this is a W=1 build):
mkdir -p ~/bin
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
git checkout a70fc4ed20a6118837b0aecbbf789074935f473b
b4 shazam https://lore.kernel.org/r/[email protected]
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang ~/bin/make.cross W=1 O=build_dir ARCH=x86_64 olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang ~/bin/make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: module cxl_core uses symbol pci_print_aer from namespace CXL, but does not import it.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RE: [PATCH v5 25/26] PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler
In reply to Terry BowmanQuoted 104 lines · Terry Bowman
From: Robert Richter <[email protected]>
In Restricted CXL Device (RCD) mode a CXL device is exposed as an RCiEP, but CXL downstream and upstream ports are not enumerated and not visible in the PCIe hierarchy. [1] Protocol and link errors from these non-enumerated ports are signaled as internal AER errors, either Uncorrectable Internal Error (UIE) or Corrected Internal Errors (CIE) via an RCEC.
Restricted CXL host (RCH) downstream port-detected errors have the Requster ID of the RCEC set in the RCEC's AER Error Source ID register. A CXL handler must then inspect the error status in various CXL registers residing in the dport's component register space (CXL RAS capability) or the dport's RCRB (PCIe AER extended capability). [2]
Errors showing up in the RCEC's error handler must be handled and connected to the CXL subsystem. Implement this by forwarding the error to all CXL devices below the RCEC. Since the entire CXL device is controlled only using PCIe Configuration Space of device 0, function 0, only pass it there [3]. The error handling is limited to currently supported devices with the Memory Device class code set (CXL Type 3 Device,
PCI_CLASS_MEMORY_CXL, 502h), handle downstream port errors in the device'scxl_pcidriver. Support for other CXL Device Types (e.g. a CXL.cache Device) can be added later.To handle downstream port errors in addition to errors directed to the CXL endpoint device, a handler must also inspect the CXL RAS and PCIe AER capabilities of the CXL downstream port the device is connected to.
Since CXL downstream port errors are signaled using internal errors, the handler requires those errors to be unmasked. This is subject of a follow-on patch.
The reason for choosing this implementation is that the AER service driver claims the RCEC device, but does not allow it to register a custom specific handler to support CXL. Connecting the RCEC hard-wired with a CXL handler does not work, as the CXL subsystem might not be present all the time. The alternative to add an implementation to the portdrv to allow the registration of a custom RCEC error handler isn't worth doing it as CXL would be its only user. Instead, just check for an CXL RCEC and pass it down to the connected CXL device's error handler. With this approach the code can entirely be implemented in the PCIe AER driver and is independent of the CXL subsystem. The CXL driver only provides the handler.
[1] CXL 3.0 spec: 9.11.8 CXL Devices Attached to an RCH
[2] CXL 3.0 spec, 12.2.1.1 RCH Downstream Port-detected Errors
[3] CXL 3.0 spec, 8.1.3 PCIe DVSEC for CXL DevicesCo-developed-by Terry Bowman <[email protected]>Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>Cc "Oliver O'Halloran" <[email protected]>Cc Bjorn Helgaas <[email protected]>Acked-by Bjorn Helgaas <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>--- drivers/pci/pcie/Kconfig | 12 +++++ drivers/pci/pcie/aer.c | 96 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 2 deletions(-)drivers/pci/pcie/Kconfig +12
-0@@ -49,6 +49,18 @@ config PCIEAER_INJECT gotten from: https://git.kernel.org/cgit/linux/kernel/git/gong.chen/aer-inject.git/ +config PCIEAER_CXL + bool "PCI Express CXL RAS support for Restricted Hosts (RCH)" + default y + depends on PCIEAER && CXL_PCI + help + Enables error handling of downstream ports of a CXL host + that is operating in RCD mode (Restricted CXL Host, RCH). + The downstream port reports AER errors to a given RCEC. + Errors are handled by the CXL memory device driver. + + If unsure, say Y. + # # PCI Express ECRC #drivers/pci/pcie/aer.c +8
-0@@ -946,14 +946,100 @@ static bool find_source_device(struct pci_dev *parent,return true; } +#ifdef CONFIG_PCIEAER_CXL + +static bool is_cxl_mem_dev (struct pci_dev *dev) +{ +/* +* The capability, status, and control fields in Device 0, +* Function 0 DVSEC control the CXL functionality of the +* entire device (CXL 3.0, 8.1.3).
drivers/pci/pcie/aer.c static bool find_source_device(struct pci_dev *parent, · line 966
Quoted 10 lines
drivers/pci/pcie/aer.c +10
-0+*/ +if (dev->devfn !=PCI_DEVFN (0 ,0 )) +return false; + +/* +* CXL Memory Devices must have the 502h class code set (CXL +* 3.0, 8.1.12.1). +*/ +if ((dev->class >>8 ) !=PCI_CLASS_MEMORY_CXL ) +return false;
I think this is ok for now, but I expect it will want to be something like dev->is_cxl in the future where that is the cached result of:
dev->is_cxl = !!pci_find_dvsec_capability(dev, PCI_DVSEC_VENDOR_ID_CXL,
CXL_DVSEC_PCIE_DEVICE);Quoted 51 lines
drivers/pci/pcie/aer.c +51
-0+ +return true; +} + +static bool cxl_error_is_native (struct pci_dev *dev) +{ +struct pci_host_bridge *host =pci_find_host_bridge (dev->bus); + +if (pcie_ports_native) +return true; + +return host->native_aer && host->native_cxl_error; +} + +static bool is_internal_error (struct aer_err_info *info) +{ +if (info->severity ==AER_CORRECTABLE ) +return info->status &PCI_ERR_COR_INTERNAL ; + +return info->status &PCI_ERR_UNC_INTN ; +} + +static int cxl_rch_handle_error_iter (struct pci_dev *dev,void *data) +{ +struct aer_err_info *info = (struct aer_err_info *)data; +const struct pci_error_handlers *err_handler; + +if (!is_cxl_mem_dev (dev) || !cxl_error_is_native (dev)) +return 0 ; + +/* protect dev->driver */ +device_lock (&dev->dev); + + err_handler = dev->driver ? dev->driver->err_handler :NULL ; +if (!err_handler) + goto out; + +if (info->severity ==AER_CORRECTABLE ) { +if (err_handler->cor_error_detected) + err_handler->cor_error_detected (dev); + }else if (err_handler->error_detected) { +if (info->severity ==AER_NONFATAL ) + err_handler->error_detected (dev, pci_channel_io_normal); +else if (info->severity ==AER_FATAL ) + err_handler->error_detected (dev, pci_channel_io_frozen); + } +out: +device_unlock (&dev->dev); +return 0 ; +} +
drivers/pci/pcie/aer.c static bool find_source_device(struct pci_dev *parent, · line 1027
Quoted 10 lines
drivers/pci/pcie/aer.c +10
-0+static void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) +{ +/* +* Internal errors of an RCEC indicate an AER error in an +* RCH's downstream port. Check and handle them in the CXL.mem +* device driver. +*/ +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC && +is_internal_error (info)) +pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info);
If I understand the RDPAS implementation note correctly,
"Probe all CXL Downstream Ports and determine whether they have logged an error in the CXL.io or CXL.cachemem status registers."
A VH topology would want to do something similar for root port error events that get routed to an event collector. Now I do not think the VH case needs to be solved in this patchset, but I believe this function can be called cxl_rcec_handle_error(), and not need to be specific about the topologies it is handling.
Other than that naming quibble, this patch looks good to me.
PATCH v5 26/26 PCI/AER: Unmask RCEC internal errors to enable RCH downstream port error handling
In reply to Terry BowmanFrom: Robert Richter <[email protected]>
AER corrected and uncorrectable internal errors (CIE/UIE) are masked in their corresponding mask registers per default once in power-up state. [1][2] Enable internal errors for RCECs to receive CXL downstream port errors of Restricted CXL Hosts (RCHs).
[1] CXL 3.0 Spec, 12.2.1.1 - RCH Downstream Port Detected Errors
[2] PCIe Base Spec r6.0, 7.8.4.3 Uncorrectable Error Mask Register, 7.8.4.6 Correctable Error Mask Register
--- drivers/pci/pcie/aer.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
drivers/pci/pcie/aer.c +57 -0
@@ -948,6 +948,30 @@ static bool find_source_device(struct pci_dev *parent,#ifdef CONFIG_PCIEAER_CXL +/** +* pci_aer_unmask_internal_errors - unmask internal errors +* @dev: pointer to the pcie_dev data structure +* +* Unmasks internal errors in the Uncorrectable and Correctable Error +* Mask registers. +* +* Note: AER must be enabled and supported by the device which must be +* checked in advance, e.g. with pcie_aer_is_native(). +*/ +static void pci_aer_unmask_internal_errors (struct pci_dev *dev) +{ +int aer = dev->aer_cap; +u32 mask; + +pci_read_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , &mask); + mask &= ~PCI_ERR_UNC_INTN ; +pci_write_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , mask); + +pci_read_config_dword (dev, aer +PCI_ERR_COR_MASK , &mask); + mask &= ~PCI_ERR_COR_INTERNAL ; +pci_write_config_dword (dev, aer +PCI_ERR_COR_MASK , mask); +} +static bool is_cxl_mem_dev (struct pci_dev *dev) {/*
@@ -1027,7 +1051,39 @@ static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); } +static int handles_cxl_error_iter (struct pci_dev *dev,void *data) +{ +int *handles_cxl = data; + +if (!*handles_cxl) + *handles_cxl =is_cxl_mem_dev (dev) &&cxl_error_is_native (dev); + +/* Non-zero terminates iteration */ +return *handles_cxl; +} + +static bool handles_cxl_errors (struct pci_dev *rcec) +{ +int handles_cxl =0 ; + +if (pci_pcie_type (rcec) ==PCI_EXP_TYPE_RC_EC && +pcie_aer_is_native (rcec)) +pcie_walk_rcec (rcec, handles_cxl_error_iter, &handles_cxl); + +return !!handles_cxl; +} + +static void cxl_rch_enable_rcec (struct pci_dev *rcec) +{ +if (!handles_cxl_errors (rcec)) +return ; + +pci_aer_unmask_internal_errors (rcec); +pci_info (rcec,"CXL: Internal errors unmasked" ); +} +#else +static inline void cxl_rch_enable_rcec (struct pci_dev *dev) { }static inline void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) { }#endif
@@ -1428,6 +1484,7 @@ static int aer_probe(struct pcie_device *dev)return status; } +cxl_rch_enable_rcec (port);aer_enable_rootport (rpc);pci_info (port,"enabled with IRQ %d\n" , dev->irq);return 0 ;
Signature · 1 line
2.34.1
Re: [PATCH v5 26/26] PCI/AER: Unmask RCEC internal errors to enable RCH downstream port error handling
Quoted 7 lines · Terry Bowman
From: Robert Richter <[email protected]>
AER corrected and uncorrectable internal errors (CIE/UIE) are masked in their corresponding mask registers per default once in power-up state. [1][2] Enable internal errors for RCECs to receive CXL downstream port errors of Restricted CXL Hosts (RCHs).
Quoted 7 lines
[1] CXL 3.0 Spec, 12.2.1.1 - RCH Downstream Port Detected Errors
[2] PCIe Base Spec r6.0, 7.8.4.3 Uncorrectable Error Mask Register, 7.8.4.6 Correctable Error Mask RegisterCo-developed-by Terry Bowman <[email protected]>Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>
Quoted 90 lines
--- drivers/pci/pcie/aer.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)drivers/pci/pcie/aer.c +57
-0@@ -948,6 +948,30 @@ static bool find_source_device(struct pci_dev *parent,#ifdef CONFIG_PCIEAER_CXL +/** +* pci_aer_unmask_internal_errors - unmask internal errors +* @dev: pointer to the pcie_dev data structure +* +* Unmasks internal errors in the Uncorrectable and Correctable Error +* Mask registers. +* +* Note: AER must be enabled and supported by the device which must be +* checked in advance, e.g. with pcie_aer_is_native(). +*/ +static void pci_aer_unmask_internal_errors (struct pci_dev *dev) +{ +int aer = dev->aer_cap; +u32 mask; + +pci_read_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , &mask); + mask &= ~PCI_ERR_UNC_INTN ; +pci_write_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , mask); + +pci_read_config_dword (dev, aer +PCI_ERR_COR_MASK , &mask); + mask &= ~PCI_ERR_COR_INTERNAL ; +pci_write_config_dword (dev, aer +PCI_ERR_COR_MASK , mask); +} +static bool is_cxl_mem_dev (struct pci_dev *dev) {/* @@ -1027,7 +1051,39 @@ static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); } +static int handles_cxl_error_iter (struct pci_dev *dev,void *data) +{ +int *handles_cxl = data; + +if (!*handles_cxl) + *handles_cxl =is_cxl_mem_dev (dev) &&cxl_error_is_native (dev); + +/* Non-zero terminates iteration */ +return *handles_cxl; +} + +static bool handles_cxl_errors (struct pci_dev *rcec) +{ +int handles_cxl =0 ; + +if (pci_pcie_type (rcec) ==PCI_EXP_TYPE_RC_EC && +pcie_aer_is_native (rcec)) +pcie_walk_rcec (rcec, handles_cxl_error_iter, &handles_cxl); + +return !!handles_cxl; +} + +static void cxl_rch_enable_rcec (struct pci_dev *rcec) +{ +if (!handles_cxl_errors (rcec)) +return ; + +pci_aer_unmask_internal_errors (rcec); +pci_info (rcec,"CXL: Internal errors unmasked" ); +} +#else +static inline void cxl_rch_enable_rcec (struct pci_dev *dev) { }static inline void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) { }#endif @@ -1428,6 +1484,7 @@ static int aer_probe(struct pcie_device *dev)return status; } +cxl_rch_enable_rcec (port);aer_enable_rootport (rpc);pci_info (port,"enabled with IRQ %d\n" , dev->irq);return 0 ;Signature · 1 line
2.34.1
RE: [PATCH v5 26/26] PCI/AER: Unmask RCEC internal errors to enable RCH downstream port error handling
In reply to Terry BowmanQuoted 94 lines · Terry Bowman
From: Robert Richter <[email protected]>
AER corrected and uncorrectable internal errors (CIE/UIE) are masked in their corresponding mask registers per default once in power-up state. [1][2] Enable internal errors for RCECs to receive CXL downstream port errors of Restricted CXL Hosts (RCHs).
[1] CXL 3.0 Spec, 12.2.1.1 - RCH Downstream Port Detected Errors
[2] PCIe Base Spec r6.0, 7.8.4.3 Uncorrectable Error Mask Register, 7.8.4.6 Correctable Error Mask RegisterCo-developed-by Terry Bowman <[email protected]>Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>--- drivers/pci/pcie/aer.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)drivers/pci/pcie/aer.c +56
-0@@ -948,6 +948,30 @@ static bool find_source_device(struct pci_dev *parent,#ifdef CONFIG_PCIEAER_CXL +/** +* pci_aer_unmask_internal_errors - unmask internal errors +* @dev: pointer to the pcie_dev data structure +* +* Unmasks internal errors in the Uncorrectable and Correctable Error +* Mask registers. +* +* Note: AER must be enabled and supported by the device which must be +* checked in advance, e.g. with pcie_aer_is_native(). +*/ +static void pci_aer_unmask_internal_errors (struct pci_dev *dev) +{ +int aer = dev->aer_cap; +u32 mask; + +pci_read_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , &mask); + mask &= ~PCI_ERR_UNC_INTN ; +pci_write_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , mask); + +pci_read_config_dword (dev, aer +PCI_ERR_COR_MASK , &mask); + mask &= ~PCI_ERR_COR_INTERNAL ; +pci_write_config_dword (dev, aer +PCI_ERR_COR_MASK , mask); +} +static bool is_cxl_mem_dev (struct pci_dev *dev) {/* @@ -1027,7 +1051,39 @@ static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)pcie_walk_rcec (dev, cxl_rch_handle_error_iter, info); } +static int handles_cxl_error_iter (struct pci_dev *dev,void *data) +{ +int *handles_cxl = data; + +if (!*handles_cxl) + *handles_cxl =is_cxl_mem_dev (dev) &&cxl_error_is_native (dev); + +/* Non-zero terminates iteration */ +return *handles_cxl; +} + +static bool handles_cxl_errors (struct pci_dev *rcec) +{ +int handles_cxl =0 ; + +if (pci_pcie_type (rcec) ==PCI_EXP_TYPE_RC_EC && +pcie_aer_is_native (rcec)) +pcie_walk_rcec (rcec, handles_cxl_error_iter, &handles_cxl); + +return !!handles_cxl; +} + +static void cxl_rch_enable_rcec (struct pci_dev *rcec) +{ +if (!handles_cxl_errors (rcec)) +return ; + +pci_aer_unmask_internal_errors (rcec); +pci_info (rcec,"CXL: Internal errors unmasked" ); +} +#else +static inline void cxl_rch_enable_rcec (struct pci_dev *dev) { }static inline void cxl_rch_handle_error (struct pci_dev *dev,struct aer_err_info *info) { }#endif
drivers/pci/pcie/aer.c static int aer_probe(struct pcie_device *dev) · line 1487
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-0@@ -1428,6 +1484,7 @@ static int aer_probe(struct pcie_device *dev)return status; } +cxl_rch_enable_rcec (port);
Similar to the last patch, I wonder if it is sufficient to call this cxl_enable_rcec() in anticipation of VH support and the fact that this only depends on RCiEPs not necessarily anything RCH specific like RCRB shenanigans.
Shouldn't there also be a corresponding cxl_disable_rcec() in aer_remove()?