cxl: Add support for Restricted CXL hosts (RCD mode)
v2 cover cxl: Add support for Restricted CXL hosts (RCD mode)
In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1) the PCIe enumeration hierarchy is different from CXL VH Enumeration (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11 and 9.12, [1]). This series adds support for RCD mode. It implements the detection of Restricted CXL Hosts (RCHs) and its corresponding Restricted CXL Devices (RCDs). It does the necessary enumeration of ports and connects the endpoints. With all the plumbing an RCH/RCD pair is registered at the Linux CXL bus and becomes visible in sysfs in the same way as CXL VH hosts and devices do already. RCDs are brought up as CXL endpoints and bound to subsequent drivers such as cxl_mem.
For CXL VH the host driver (cxl_acpi) starts host bridge discovery once the ACPI0017 CXL root device is detected and then searches for ACPI0016 host bridges to enable CXL. This implementation requires that in RCD mode an ACPI0017 device exists, the host bridge must have the ACPI0016 ID set. This implementation also expects the host's downstream and upstream port RCRBs base address being reported by firmware using the optional CEDT CHBS entry of the host bridge (see CXL spec 3.0, 9.17.1.2).
RCD mode does not support hot-plug, so host discovery is at boot time only.
Patches #1 to #5 are prerequisites of the series with fixes needed and a rework of debug messages for port enumeration. Those are general patches and could be applied earlier and independently from the rest assuming there are no objections with them. Patches #6 to #12 contain the actual implementation of RCD mode support.
[1] https://www.computeexpresslink.org/spec-landing
---
v2:
* Reworked series to use add_host_bridge_dport() and add_host_bridge_uport(). There is a single cxl root device (ACPI0017) also for RCHs (must have a ACPI0016 id). (Dan)
* Rebased onto 6.1-rc1.
* Added a WARN_ON_ONCE() to CXL_RESOURCE_NONE check. Updated patch description with an example case. (Dan, Jonathan)
* Added wrapper functions to devm_cxl_add_port() and devm_cxl_add_dport(). (Dan)
* Dropped "PCI/ACPI: Link host bridge to its ACPI fw node" patch.
* Updated spec refs to use 3.0. Added PCIe base spec refs. (Jonathan)
* Reused UID detect code. (Dan)
* Dropped "cxl/acpi: Specify module load order dependency for the cxl_acpi module" patch. Return -EINVAL if host not yet ready. (Dan)
* Minor other changes.
* Note: I haven't included most of the received Reviewed-by tags due to the major rework. In any case, thanks to all here.
Robert Richter (12):
cxl/core: Remove duplicate declaration of devm_cxl_iomap_block() cxl/core: Check physical address before mapping it in
devm_cxl_iomap_block()
cxl: Unify debug messages when calling devm_cxl_add_port()
cxl: Unify debug messages when calling devm_cxl_add_dport() cxl/acpi: Improve debug messages in cxl_acpi_probe() cxl/acpi: Extract component registers of restricted hosts from RCRB
cxl: Remove dev_is_cxl_root_child() check in
devm_cxl_enumerate_ports()
cxl: Factor out code in devm_cxl_enumerate_ports() to
find_port_attach_ep()
cxl: Extend devm_cxl_enumerate_ports() to support restricted devices
(RCDs)
cxl: Do not ignore PCI config read errors in match_add_dports()
cxl: Factor out code in match_add_dports() to pci_dev_add_dport()
cxl: Extend devm_cxl_port_enumerate_dports() to support restricted
hosts (RCH)
drivers/cxl/acpi.c | 98 ++++++++++++---- drivers/cxl/core/pci.c | 76 +++++++++--- drivers/cxl/core/port.c | 219 +++++++++++++++++++++++++---------- drivers/cxl/core/regs.c | 5 + drivers/cxl/cxl.h | 2 - tools/testing/cxl/test/cxl.c | 8 +- 6 files changed, 305 insertions(+), 103 deletions(-)
base-commit: 9abf2313adc1ca1b6180c508c25f22f9395cc780
Signature · 1 line
2.30.2
PATCH v2 1/12 cxl/core: Remove duplicate declaration of devm_cxl_iomap_block()
The function devm_cxl_iomap_block() is only used in the core code. There are two declarations in header files of it, in drivers/cxl/core/core.h and drivers/cxl/cxl.h. Remove its unused declaration in drivers/cxl/cxl.h.
Fixing build error in regs.c found by kernel test robot by including "core.h" there.
--- drivers/cxl/core/regs.c | 2 ++ drivers/cxl/cxl.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/cxl/core/regs.c +2 -0
@@ -7,6 +7,8 @@#include <cxlmem.h> #include <cxlpci.h> +#include "core.h" +/** * DOC: cxl registers *
drivers/cxl/cxl.h +0 -2
@@ -218,8 +218,6 @@ int cxl_map_device_regs(struct pci_dev *pdev,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); -void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, -resource_size_t length);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20
Signature · 1 line
2.30.2
RE: [PATCH v2 01/12] cxl/core: Remove duplicate declaration of devm_cxl_iomap_block()
Quoted 7 lines · Robert Richter
The function
devm_cxl_iomap_block()is only used in the core code. There are two declarations in header files of it, indrivers/cxl/core/core.handdrivers/cxl/cxl.h. Remove its unused declaration indrivers/cxl/cxl.h.Fixing build error in
regs.cfound by kernel test robot by including "core.h" there.
Fixes a build error only with the new code in this series, or do you have a config where this fails with current mainline?
Assuming the former, this is applied for v6.2.
Re: [PATCH v2 01/12] cxl/core: Remove duplicate declaration of devm_cxl_iomap_block()
Quoted 6 lines · Dan Williams
Quoted 5 lines · Robert Richter
The function
devm_cxl_iomap_block()is only used in the core code. There are two declarations in header files of it, indrivers/cxl/core/core.handdrivers/cxl/cxl.h. Remove its unused declaration indrivers/cxl/cxl.h.
Quoted 7 lines
Quoted 2 lines
Fixing build error in
regs.cfound by kernel test robot by including "core.h" there.Fixes a build error only with the new code in this series, or do you have a config where this fails with current mainline?
Assuming the former, this is applied for v6.2.
Yes, a build error in v1.
Thanks,
-Robert
PATCH v2 2/12 cxl/core: Check physical address before mapping it in devm_cxl_iomap_block()
In reply to Robert RichterThe physical base address of a CXL range can be invalid and is then set to CXL_RESOURCE_NONE. In general software shall prevent such situations, but it is hard to proof this may never happen. E.g. in add_port_attach_ep() there this the following:
component_reg_phys = find_component_registers(uport_dev);
port = devm_cxl_add_port(&parent_port->dev, uport_dev,
component_reg_phys, parent_dport);find_component_registers() and subsequent functions (e.g. cxl_regmap_to_base()) may return CXL_RESOURCE_NONE. But it is written to port without any further check in cxl_port_alloc():
port->component_reg_phys = component_reg_phys;
It is then later directly used in devm_cxl_setup_hdm() to map io ranges with devm_cxl_iomap_block(). Just an example...
Check this condition. Also do not fail silently like an ioremap() failure, use a WARN_ON_ONCE() for it.
--- drivers/cxl/core/regs.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/cxl/core/regs.c +3 -0
@@ -167,6 +167,9 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,void __iomem *ret_val;struct resource *res; +if (WARN_ON_ONCE (addr ==CXL_RESOURCE_NONE )) +return NULL ; + res =devm_request_mem_region (dev, addr, length,dev_name (dev));if (!res) {resource_size_t end = addr + length -1 ;
Signature · 1 line
2.30.2
RE: [PATCH v2 02/12] cxl/core: Check physical address before mapping it in devm_cxl_iomap_block()
Quoted 15 lines · Robert Richter
The physical base address of a CXL range can be invalid and is then set to
CXL_RESOURCE_NONE. In general software shall prevent such situations, but it is hard to proof this may never happen. E.g. inadd_port_attach_ep()there this the following:component_reg_phys = find_component_registers(uport_dev); port = devm_cxl_add_port(&parent_port->dev, uport_dev, component_reg_phys, parent_dport);
find_component_registers()and subsequent functions (e.g.cxl_regmap_to_base()) may returnCXL_RESOURCE_NONE. But it is written to port without any further check incxl_port_alloc():port->component_reg_phys = component_reg_phys;
Quoted 7 lines
It is then later directly used in
devm_cxl_setup_hdm()to map io ranges withdevm_cxl_iomap_block(). Just an example...Check this condition. Also do not fail silently like an
ioremap()failure, use aWARN_ON_ONCE()for it.Signed-off-by Robert Richter <[email protected]>
Looks good to me, applied for v6.2
PATCH v2 3/12 cxl: Unify debug messages when calling devm_cxl_add_port()
In reply to Robert RichterCXL ports are added in a couple of code paths using devm_cxl_add_ port(). Debug messages are individually generated, but are incomplete and inconsistent. Change this by moving its generation to devm_cxl_ add_port(). This unifies the messages and reduces code duplication. Also, generate messages on failure. Use a __devm_cxl_add_port() wrapper to keep the readability of the error exits.
--- drivers/cxl/acpi.c | 2 -- drivers/cxl/core/port.c | 51 +++++++++++++++++++++++++++++++---------- 2 files changed, 39 insertions(+), 14 deletions(-)
drivers/cxl/acpi.c +0 -2
@@ -220,7 +220,6 @@ static int add_host_bridge_uport(struct device *match, void *arg) port =devm_cxl_add_port (host, match, dport->component_reg_phys, dport);if (IS_ERR (port))return PTR_ERR (port); -dev_dbg (host,"%s: add: %s\n" ,dev_name (match),dev_name (&port->dev));return 0 ; }
@@ -466,7 +465,6 @@ static int cxl_acpi_probe(struct platform_device *pdev) root_port =devm_cxl_add_port (host, host,CXL_RESOURCE_NONE ,NULL );if (IS_ERR (root_port))return PTR_ERR (root_port); -dev_dbg (host,"add: %s\n" ,dev_name (&root_port->dev)); rc =bus_for_each_dev (adev->dev.bus,NULL , root_port, add_host_bridge_dport);
drivers/cxl/core/port.c +39 -12
69 lines not on this page · raw message
Signature · 1 line
2.30.2
RE: [PATCH v2 03/12] cxl: Unify debug messages when calling devm_cxl_add_port()
Quoted 2 lines · Robert Richter
CXL ports are added in a couple of code paths using
devm_cxl_add_port(). Debug messages are individually generated, but are incomplete
Hmm, devm_cxl_add_port() got wrapped in a few places here. I fixed it up locally, but maybe take a look at your word wrap to keep function symbols unbroken in future changelogs.
Quoted 4 lines
and inconsistent. Change this by moving its generation to
devm_cxl_add_port(). This unifies the messages and reduces code duplication. Also, generate messages on failure. Use a__devm_cxl_add_port()wrapper to keep the readability of the error exits.
Otherwise looks good, applied for v6.2.
Re: [PATCH v2 03/12] cxl: Unify debug messages when calling devm_cxl_add_port()
Quoted 7 lines · Dan Williams
Quoted 2 lines · Robert Richter
CXL ports are added in a couple of code paths using
devm_cxl_add_port(). Debug messages are individually generated, but are incompleteHmm,
devm_cxl_add_port()got wrapped in a few places here. I fixed it up locally, but maybe take a look at your word wrap to keep function symbols unbroken in future changelogs.
Ok, will do. Thanks for the note.
Quoted 7 lines
Quoted 5 lines
and inconsistent. Change this by moving its generation to
devm_cxl_add_port(). This unifies the messages and reduces code duplication. Also, generate messages on failure. Use a__devm_cxl_add_port()wrapper to keep the readability of the error exits.Otherwise looks good, applied for v6.2.
Thanks,
-Robert
PATCH v2 4/12 cxl: Unify debug messages when calling devm_cxl_add_dport()
In reply to Robert RichterCXL dports are added in a couple of code paths using devm_cxl_add_ dport(). Debug messages are individually generated, but are incomplete and inconsistent. Change this by moving its generation to devm_cxl_ add_dport(). This unifies the messages and reduces code duplication. Also, generate messages on failure. Use a __devm_cxl_add_dport() wrapper to keep the readability of the error exits.
--- drivers/cxl/acpi.c | 7 ++---- drivers/cxl/core/pci.c | 2 -- drivers/cxl/core/port.c | 48 +++++++++++++++++++++++++----------- tools/testing/cxl/test/cxl.c | 8 +----- 4 files changed, 37 insertions(+), 28 deletions(-)
drivers/cxl/acpi.c +2 -5
@@ -282,12 +282,9 @@ static int add_host_bridge_dport(struct device *match, void *arg) } dport =devm_cxl_add_dport (root_port, match, uid, ctx.chbcr); -if (IS_ERR (dport)) { -dev_err (host,"failed to add downstream port: %s\n" , -dev_name (match)); +if (IS_ERR (dport))return PTR_ERR (dport); - } -dev_dbg (host,"add dport%llu: %s\n" , uid,dev_name (match)); +return 0 ; }
drivers/cxl/core/pci.c +0 -2
@@ -62,8 +62,6 @@ static int match_add_dports(struct pci_dev *pdev, void *data) } ctx->count++; -dev_dbg (&port->dev,"add dport%d: %s\n" , port_num,dev_name (&pdev->dev)); -return 0 ; }
drivers/cxl/core/port.c +34 -14
60 lines not on this page · raw message
tools/testing/cxl/test/cxl.c +1 -7
@@ -582,14 +582,8 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port) dport =devm_cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE ); -if (IS_ERR (dport)) { -dev_err (dev,"failed to add dport: %s (%ld)\n" , -dev_name (&pdev->dev),PTR_ERR (dport)); +if (IS_ERR (dport))return PTR_ERR (dport); - } - -dev_dbg (dev,"add dport%d: %s\n" , pdev->id, -dev_name (&pdev->dev)); }return 0 ;
Signature · 1 line
2.30.2
RE: [PATCH v2 04/12] cxl: Unify debug messages when calling devm_cxl_add_dport()
Quoted 6 lines · Robert Richter
CXL dports are added in a couple of code paths using
devm_cxl_add_dport(). Debug messages are individually generated, but are incomplete and inconsistent. Change this by moving its generation todevm_cxl_add_dport(). This unifies the messages and reduces code duplication. Also, generate messages on failure. Use a__devm_cxl_add_dport()wrapper to keep the readability of the error exits.
Looks good, I fixed up the wrapping of devm_cxl_add_dport() and one more fixup below, but applied for v6.2.
Quoted 9 lines
Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 7 ++---- drivers/cxl/core/pci.c | 2 -- drivers/cxl/core/port.c | 48 +++++++++++++++++++++++++----------- tools/testing/cxl/test/cxl.c | 8 +----- 4 files changed, 37 insertions(+), 28 deletions(-)
[..]
Quoted 7 lines
tools/testing/cxl/test/cxl.c +0
-0@@ -582,14 +582,8 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port) dport =devm_cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE );
tools/testing/cxl/test/cxl.c static int mock_cxl_port_enumerate_dports(struct cxl_port *port) · line 592
Quoted 10 lines
tools/testing/cxl/test/cxl.c +1
-7-if (IS_ERR (dport)) { -dev_err (dev,"failed to add dport: %s (%ld)\n" , -dev_name (&pdev->dev),PTR_ERR (dport)); +if (IS_ERR (dport))return PTR_ERR (dport); - } - -dev_dbg (dev,"add dport%d: %s\n" , pdev->id, -dev_name (&pdev->dev));
This hunk causes this new compile error:
tools/testing/cxl/test/cxl.c: In function ‘mock_cxl_port_enumerate_dports’:tools/testing/cxl/test/cxl.c:559:24: error: unused variable ‘dev’ [-Werror=unused-variable]
559 | struct device *dev = &port->dev;
| ^~~
cc1: all warnings being treated as errors
I fixed it up locally, but just double check that you are building cxl_test if you touch tools/testing/cxl/:
make M=tools/testing/cxl
Re: [PATCH v2 04/12] cxl: Unify debug messages when calling devm_cxl_add_dport()
Quoted 28 lines · Dan Williams
Quoted 7 lines · Robert Richter
tools/testing/cxl/test/cxl.c +0
-0@@ -582,14 +582,8 @@ static int mock_cxl_port_enumerate_dports(struct cxl_port *port) dport =devm_cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE );tools/testing/cxl/test/cxl.c static int mock_cxl_port_enumerate_dports(struct cxl_port *port) · line 592
Quoted 10 lines
tools/testing/cxl/test/cxl.c +1
-7-if (IS_ERR (dport)) { -dev_err (dev,"failed to add dport: %s (%ld)\n" , -dev_name (&pdev->dev),PTR_ERR (dport)); +if (IS_ERR (dport))return PTR_ERR (dport); - } - -dev_dbg (dev,"add dport%d: %s\n" , pdev->id, -dev_name (&pdev->dev));This hunk causes this new compile error:
tools/testing/cxl/test/cxl.c: In function ‘mock_cxl_port_enumerate_dports’:tools/testing/cxl/test/cxl.c:559:24: error: unused variable ‘dev’ [-Werror=unused-variable]
559 | struct device *dev =&port->dev;
| ^~~
cc1: all warnings being treated as errors
Quoted 4 lines
I fixed it up locally, but just double check that you are building
cxl_testif you touchtools/testing/cxl/:make M=tools/testing/cxl
Right. Thanks for fixing. Will build test in the future.
-Robert
PATCH v2 5/12 cxl/acpi: Improve debug messages in cxl_acpi_probe()
In reply to Robert RichterIn cxl_acpi_probe() the iterator bus_for_each_dev() walks through all CXL hosts. Since all dev_*() debug messages point to the ACPI0017 device which is the CXL root for all hosts, the device information is pointless as it is always the same device. Change this to use the host device for this instead.
Also, add additional host specific information such as CXL support, UID and CHBCR.
This is an example log:
acpi ACPI0016:00: UID found: 4
acpi ACPI0016:00: CHBCR found: 0x28090000000
acpi ACPI0016:00: dport added to root0
acpi ACPI0016:00: host-bridge: ACPI0016:00
pci0000:7f: host supports CXL
--- drivers/cxl/acpi.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/cxl/acpi.c +8 -4
@@ -221,6 +221,8 @@ static int add_host_bridge_uport(struct device *match, void *arg)if (IS_ERR (port))return PTR_ERR (port); +dev_info (pci_root->bus->bridge,"host supports CXL\n" ); +return 0 ; }
@@ -264,11 +266,12 @@ static int add_host_bridge_dport(struct device *match, void *arg) status =acpi_evaluate_integer (bridge->handle,METHOD_NAME__UID ,NULL , &uid);if (status !=AE_OK ) { -dev_err (host,"unable to retrieve _UID of %s\n" , -dev_name (match)); +dev_err (match,"unable to retrieve _UID\n" );return -ENODEV ; } +dev_dbg (match,"UID found: %lld\n" , uid); + ctx = (struct cxl_chbs_context ) { .dev = host, .uid = uid,
@@ -276,11 +279,12 @@ static int add_host_bridge_dport(struct device *match, void *arg)acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbcr, &ctx);if (ctx.chbcr ==0 ) { -dev_warn (host,"No CHBS found for Host Bridge: %s\n" , -dev_name (match)); +dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , uid);return 0 ; } +dev_dbg (match,"CHBCR found: 0x%08llx\n" , (u64 )ctx.chbcr); + dport =devm_cxl_add_dport (root_port, match, uid, ctx.chbcr);if (IS_ERR (dport))return PTR_ERR (dport);
Signature · 1 line
2.30.2
RE: [PATCH v2 05/12] cxl/acpi: Improve debug messages in cxl_acpi_probe()
Quoted 27 lines · Robert Richter
In
cxl_acpi_probe()the iteratorbus_for_each_dev()walks through all CXL hosts. Since all dev_*() debug messages point to the ACPI0017 device which is the CXL root for all hosts, the device information is pointless as it is always the same device. Change this to use the host device for this instead.Also, add additional host specific information such as CXL support, UID and CHBCR.
This is an example log:
acpi ACPI0016:00: UID found: 4
acpi ACPI0016:00: CHBCR found:0x28090000000
acpi ACPI0016:00: dport added to root0
acpi ACPI0016:00: host-bridge: ACPI0016:00
pci0000:7f: host supports CXLSigned-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)drivers/cxl/acpi.c +0
-0
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 225
Quoted 5 lines
drivers/cxl/acpi.c +2
-0@@ -221,6 +221,8 @@ static int add_host_bridge_uport(struct device *match, void *arg)if (IS_ERR (port))return PTR_ERR (port); +dev_info (pci_root->bus->bridge,"host supports CXL\n" ); +
I generally advocate for drivers to be silent, but this is a fundamental message in a place that is helpful to indicate whether platform-firmware is doing its job correctly, and it's only once per host bridge.
Applied for v6.2.
PATCH 1/1 tools/testing/cxl: Add bridge mocking support
In reply to Robert RichterIn preparation for cxl_acpi walking pci_root->bus->bridge, add that association to the mock pci_root instances.
Note that the missing 3rd entry in mock_pci_root[] was not noticed until now given that the test version of to_cxl_host_bridge() (tools/testing/cxl/mock_acpi.c), obviated the need for that entry. However, "cxl/acpi: Improve debug messages in cxl_acpi_probe()" [1] needs pci_root->bus->bridge to be populated.
--- tools/testing/cxl/test/cxl.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
tools/testing/cxl/test/cxl.c +8 -2
@@ -439,14 +439,18 @@ mock_acpi_evaluate_integer(acpi_handle handle, acpi_string pathname,return AE_OK ; } -static struct pci_bus mock_pci_bus[NR_CXL_HOST_BRIDGES ]; -static struct acpi_pci_root mock_pci_root[NR_CXL_HOST_BRIDGES ] = { +static struct pci_bus mock_pci_bus[NR_CXL_HOST_BRIDGES +NR_CXL_SINGLE_HOST ]; +static struct acpi_pci_root mock_pci_root[ARRAY_SIZE (mock_pci_bus)] = { [0 ] = { .bus = &mock_pci_bus[0 ], }, [1 ] = { .bus = &mock_pci_bus[1 ], }, + [2 ] = { + .bus = &mock_pci_bus[2 ], + }, + };static bool is_mock_bus (struct pci_bus *bus)
@@ -744,6 +748,7 @@ static __init int cxl_single_init(void) } cxl_hb_single[i] = pdev; + mock_pci_bus[i +NR_CXL_HOST_BRIDGES ].bridge = &pdev->dev; rc =sysfs_create_link (&pdev->dev.kobj, &pdev->dev.kobj,"physical_node" );if (rc)
@@ -910,6 +915,7 @@ static __init int cxl_test_init(void) } cxl_host_bridge[i] = pdev; + mock_pci_bus[i].bridge = &pdev->dev; rc =sysfs_create_link (&pdev->dev.kobj, &pdev->dev.kobj,"physical_node" );if (rc)
Re: [PATCH] tools/testing/cxl: Add bridge mocking support
Quoted 8 lines · Dan Williams
In preparation for
cxl_acpiwalkingpci_root->bus->bridge, add that association to the mockpci_rootinstances.Note that the missing 3rd entry in mock_pci_root[] was not noticed until now given that the test version of
to_cxl_host_bridge()(tools/testing/cxl/mock_acpi.c), obviated the need for that entry. However, "cxl/acpi: Improve debug messages incxl_acpi_probe()" [1] needspci_root->bus->bridgeto be populated.
Patch looks good to me, though we might want to move the mock_pci_root[] fix into e41c8452b9b2 or on top as a separate change.
For patch logistics, could you merge that changes into the rch tree as a new base? Right now I have cherry picked e41c8452b9b2 and this one as pre-requisites of my series.
-Robert
Quoted 4 lines
Cc Robert Richter <[email protected]>Signed-off-by Dan Williams <[email protected]>
Re: [PATCH] tools/testing/cxl: Add bridge mocking support
Quoted 10 lines · Robert Richter
Quoted 8 lines · Dan Williams
In preparation for
cxl_acpiwalkingpci_root->bus->bridge, add that association to the mockpci_rootinstances.Note that the missing 3rd entry in mock_pci_root[] was not noticed until now given that the test version of
to_cxl_host_bridge()(tools/testing/cxl/mock_acpi.c), obviated the need for that entry. However, "cxl/acpi: Improve debug messages incxl_acpi_probe()" [1] needspci_root->bus->bridgeto be populated.
Quoted 6 lines
Patch looks good to me, though we might want to move the mock_pci_root[] fix into
e41c8452b9b2or on top as a separate change.For patch logistics, could you merge that changes into the rch tree as a new base? Right now I have cherry picked
e41c8452b9b2and this one as pre-requisites of my series.
Yes, I re-rolled the series on -rc4. Will post a new baseline once I have the unit test topology ready later today.
Re: [PATCH] tools/testing/cxl: Add bridge mocking support
In reply to Dan WilliamsQuoted 28 lines · On Mon, 14 Nov 2022 12:11:13 -0800 Dan Williams <[email protected]>…
In preparation for
cxl_acpiwalkingpci_root->bus->bridge, add that association to the mockpci_rootinstances.Note that the missing 3rd entry in mock_pci_root[] was not noticed until now given that the test version of
to_cxl_host_bridge()(tools/testing/cxl/mock_acpi.c), obviated the need for that entry. However, "cxl/acpi: Improve debug messages incxl_acpi_probe()" [1] needspci_root->bus->bridgeto be populated.Cc Robert Richter <[email protected]>Signed-off-by Dan Williams <[email protected]>--- tools/testing/cxl/test/cxl.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)tools/testing/cxl/test/cxl.c +2
-2@@ -439,14 +439,18 @@ mock_acpi_evaluate_integer(acpi_handle handle, acpi_string pathname,return AE_OK ; } -static struct pci_bus mock_pci_bus[NR_CXL_HOST_BRIDGES ]; -static struct acpi_pci_root mock_pci_root[NR_CXL_HOST_BRIDGES ] = { +static struct pci_bus mock_pci_bus[NR_CXL_HOST_BRIDGES +NR_CXL_SINGLE_HOST ]; +static struct acpi_pci_root mock_pci_root[ARRAY_SIZE (mock_pci_bus)] = {
tools/testing/cxl/test/cxl.c mock_acpi_evaluate_integer(acpi_handle handle, acpi_string pathname, · line 453
Quoted 10 lines
tools/testing/cxl/test/cxl.c +4
-0[0] = { .bus = &mock_pci_bus[0], }, [1] = { .bus = &mock_pci_bus[1], }, + [2] = { + .bus = &mock_pci_bus[2], + }, +
Ahah. Here's the stray space that gets cleaned up in the RCH series. Guessing you don't want to rebase to fix it here?
Quoted 1 line
tools/testing/cxl/test/cxl.c +0
-0};
PATCH v2 6/12 cxl/acpi: Extract component registers of restricted hosts from RCRB
In reply to Robert RichterA downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)
drivers/cxl/acpi.c +69 -10
@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;
@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{ +struct acpi_cedt_chbs *chbs = &ctx->chbs; +resource_size_t component_reg_phys, rcrb; +u32 bar0, bar1; +void *addr; + +if (!chbs->base) +return CXL_RESOURCE_NONE ; + +if (chbs->cxl_version !=ACPI_CEDT_CHBS_VERSION_CXL11 ) +return chbs->base; + +/* Extract RCRB */ + +if (chbs->length !=CXL_RCRB_SIZE ) +return CXL_RESOURCE_NONE ; + + rcrb = chbs->base; + +dev_dbg (ctx->dev,"RCRB found for UID %lld: 0x%08llx\n" , + ctx->uid, (u64 )rcrb); + +/* +* RCRB's BAR[0..1] point to component block containing CXL +* subsystem component registers. MEMBAR extraction follows +* the PCI Base spec here, esp. 64 bit extraction and memory +* ranges alignment (6.0, 7.5.1.2.1). +*/ + addr =ioremap (rcrb,PCI_BASE_ADDRESS_0 +SZ_8 ); + bar0 =readl (addr +PCI_BASE_ADDRESS_0 ); + bar1 =readl (addr +PCI_BASE_ADDRESS_1 ); +iounmap (addr); + +/* sanity check */ +if (bar0 & (PCI_BASE_ADDRESS_MEM_TYPE_1M |PCI_BASE_ADDRESS_SPACE_IO )) +return CXL_RESOURCE_NONE ; + + component_reg_phys = bar0 &PCI_BASE_ADDRESS_MEM_MASK ; +if (bar0 &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + component_reg_phys |= ((u64 )bar1) <<32 ; + +if (!component_reg_phys) +return CXL_RESOURCE_NONE ; + +/* +* Must be 8k aligned (size of combined CXL 1.1 Downstream and +* Upstream Port RCRBs). +*/ +if (component_reg_phys & (CXL_RCRB_SIZE -1 )) +return CXL_RESOURCE_NONE ; + +return component_reg_phys; +} +static int add_host_bridge_dport (struct device *match,void *arg) {acpi_status status;
@@ -259,6 +316,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)struct cxl_port *root_port = arg;struct device *host = root_port->dev.parent;struct acpi_device *bridge =to_cxl_host_bridge (host, match); +resource_size_t component_reg_phys;if (!bridge)return 0 ;
@@ -273,19 +331,20 @@ static int add_host_bridge_dport(struct device *match, void *arg)dev_dbg (match,"UID found: %lld\n" , uid); ctx = (struct cxl_chbs_context ) { - .dev = host, + .dev = match, .uid = uid, }; -acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbcr, &ctx); +acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs, &ctx); -if (ctx.chbcr ==0 ) { + component_reg_phys =cxl_get_chbcr (&ctx); +if (component_reg_phys ==CXL_RESOURCE_NONE ) {dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , uid);return 0 ; } -dev_dbg (match,"CHBCR found: 0x%08llx\n" , (u64 )ctx.chbcr); +dev_dbg (match,"CHBCR found: 0x%08llx\n" , (u64 )component_reg_phys); - dport =devm_cxl_add_dport (root_port, match, uid, ctx.chbcr); + dport =devm_cxl_add_dport (root_port, match, uid, component_reg_phys);if (IS_ERR (dport))return PTR_ERR (dport);
Signature · 1 line
2.30.2
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 11 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Quoted 5 lines
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>
What does this S-o-B mean? If this person is your co-developer, you need to add a Co-developed-by tag to clarify that.
Quoted 143 lines
Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)drivers/cxl/acpi.c +69
-11@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{ +struct acpi_cedt_chbs *chbs = &ctx->chbs; +resource_size_t component_reg_phys, rcrb; +u32 bar0, bar1; +void *addr; + +if (!chbs->base) +return CXL_RESOURCE_NONE ; + +if (chbs->cxl_version !=ACPI_CEDT_CHBS_VERSION_CXL11 ) +return chbs->base; + +/* Extract RCRB */ + +if (chbs->length !=CXL_RCRB_SIZE ) +return CXL_RESOURCE_NONE ; + + rcrb = chbs->base; + +dev_dbg (ctx->dev,"RCRB found for UID %lld: 0x%08llx\n" , + ctx->uid, (u64 )rcrb); + +/* +* RCRB's BAR[0..1] point to component block containing CXL +* subsystem component registers. MEMBAR extraction follows +* the PCI Base spec here, esp. 64 bit extraction and memory +* ranges alignment (6.0, 7.5.1.2.1). +*/ + addr =ioremap (rcrb,PCI_BASE_ADDRESS_0 +SZ_8 ); + bar0 =readl (addr +PCI_BASE_ADDRESS_0 ); + bar1 =readl (addr +PCI_BASE_ADDRESS_1 ); +iounmap (addr); + +/* sanity check */ +if (bar0 & (PCI_BASE_ADDRESS_MEM_TYPE_1M |PCI_BASE_ADDRESS_SPACE_IO )) +return CXL_RESOURCE_NONE ; + + component_reg_phys = bar0 &PCI_BASE_ADDRESS_MEM_MASK ; +if (bar0 &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + component_reg_phys |= ((u64 )bar1) <<32 ; + +if (!component_reg_phys) +return CXL_RESOURCE_NONE ; + +/* +* Must be 8k aligned (size of combined CXL 1.1 Downstream and +* Upstream Port RCRBs). +*/ +if (component_reg_phys & (CXL_RCRB_SIZE -1 )) +return CXL_RESOURCE_NONE ; + +return component_reg_phys; +} +static int add_host_bridge_dport (struct device *match,void *arg) {acpi_status status;@@ -259,6 +316,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)struct cxl_port *root_port = arg;struct device *host = root_port->dev.parent;struct acpi_device *bridge =to_cxl_host_bridge (host, match); +resource_size_t component_reg_phys;if (!bridge)return 0 ;@@ -273,19 +331,20 @@ static int add_host_bridge_dport(struct device *match, void *arg)dev_dbg (match,"UID found: %lld\n" , uid); ctx = (struct cxl_chbs_context ) { - .dev = host, + .dev = match, .uid = uid, }; -acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbcr, &ctx); +acpi_table_parse_cedt (ACPI_CEDT_TYPE_CHBS , cxl_get_chbs, &ctx); -if (ctx.chbcr ==0 ) { + component_reg_phys =cxl_get_chbcr (&ctx); +if (component_reg_phys ==CXL_RESOURCE_NONE ) {dev_warn (match,"No CHBS found for Host Bridge (UID %lld)\n" , uid);return 0 ; } -dev_dbg (match,"CHBCR found: 0x%08llx\n" , (u64 )ctx.chbcr); +dev_dbg (match,"CHBCR found: 0x%08llx\n" , (u64 )component_reg_phys); - dport =devm_cxl_add_dport (root_port, match, uid, ctx.chbcr); + dport =devm_cxl_add_dport (root_port, match, uid, component_reg_phys);if (IS_ERR (dport))return PTR_ERR (dport); --2.30.2
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 12 lines · Rafael J. Wysocki
Quoted 11 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Quoted 10 lines
Quoted 5 lines
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>What does this S-o-B mean? If this person is your co-developer, you need to add a Co-developed-by tag to clarify that.
Quoted 1 line
Signed-off-by Robert Richter <[email protected]>
I picked up an early patch and modified it significantly, so I just left the S-o-B. I could change this to a Co-developed-by tag. IMO, the S-o-B is ok, but could be wrong here.
-Robert
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 20 lines · Robert Richter
Quoted 18 lines · Rafael J. Wysocki
Quoted 16 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>
Quoted 7 lines
Quoted 4 lines
What does this S-o-B mean? If this person is your co-developer, you need to add a Co-developed-by tag to clarify that.
Quoted 1 line
Signed-off-by Robert Richter <[email protected]>I picked up an early patch and modified it significantly, so I just left the S-o-B.
In that case the right thing to do is to mention the original author in the changelog instead of retaining the S-o-b.
Quoted 1 line
I could change this to a Co-developed-by tag.
Co-developed-by should be used in addition to and not instead of S-o-b when one of the authors is sending a patch. However, all of the authors need to be familiar with the patch in the form in which it is being sent then.
Quoted 1 line
IMO, the S-o-B is ok, but could be wrong here.
It isn't, at least not without a Co-developed-by tag.
There are 3 cases in which S-o-b is OK AFAICS:
1. When it matches the From: address.
2. When there is a matching Co-developed-by.
3. When maintainers pick up patches and add their own S-o-b.
This case is none of the above.
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 43 lines · Rafael J. Wysocki
Quoted 20 lines · Robert Richter
Quoted 18 lines · Rafael J. Wysocki
Quoted 16 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>Quoted 7 lines
Quoted 4 lines
What does this S-o-B mean? If this person is your co-developer, you need to add a Co-developed-by tag to clarify that.
Quoted 1 line
Signed-off-by Robert Richter <[email protected]>I picked up an early patch and modified it significantly, so I just left the S-o-B.
In that case the right thing to do is to mention the original author in the changelog instead of retaining the S-o-b.
Quoted 1 line
I could change this to a Co-developed-by tag.
Co-developed-by should be used in addition to and not instead of S-o-b when one of the authors is sending a patch. However, all of the authors need to be familiar with the patch in the form in which it is being sent then.
Quoted 1 line
IMO, the S-o-B is ok, but could be wrong here.
It isn't, at least not without a Co-developed-by tag.
Quoted 7 lines
There are 3 cases in which S-o-b is OK AFAICS:
1. When it matches the From: address.
2. When there is a matching Co-developed-by.
3. When maintainers pick up patches and add their own S-o-b.This case is none of the above.
Will add a Co-developed-by tag in my next version. Thanks for pointing that out.
-Robert
RE: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
In reply to Robert RichterQuoted 55 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)drivers/cxl/acpi.c +6
-4@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 256
Quoted 10 lines
drivers/cxl/acpi.c +3
-1if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{
The core logic of this looks good, but this wants to be shared with the upstream port component register discovery.
Full disclosure I am reconciling these patches with an attempt that Dave Jiang made at this topic. Since your series hit the list first I will let it take the lead, but then fill it in with comments and learnings from Dave's effort.
So in this case Dave moved this into the drivers/cxl/core/regs.c with a function signature like:
enum cxl_rcrb {
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
resource_size_t cxl_rcrb_to_component(struct device *dev,
resource_size_t rcrb_base, int len, enum cxl_rcrb which);
...where @which alternates when called by cxl_acpi for the downstream case, or cxl_mem for the upstream case.
Quoted 18 lines
drivers/cxl/acpi.c +18
-0+struct acpi_cedt_chbs *chbs = &ctx->chbs; +resource_size_t component_reg_phys, rcrb; +u32 bar0, bar1; +void *addr; + +if (!chbs->base) +return CXL_RESOURCE_NONE ; + +if (chbs->cxl_version !=ACPI_CEDT_CHBS_VERSION_CXL11 ) +return chbs->base; + +/* Extract RCRB */ + +if (chbs->length !=CXL_RCRB_SIZE ) +return CXL_RESOURCE_NONE ; + + rcrb = chbs->base; +
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 284
Quoted 10 lines
drivers/cxl/acpi.c +10
-0+dev_dbg (ctx->dev,"RCRB found for UID %lld: 0x%08llx\n" , + ctx->uid, (u64 )rcrb); + +/* +* RCRB's BAR[0..1] point to component block containing CXL +* subsystem component registers. MEMBAR extraction follows +* the PCI Base spec here, esp. 64 bit extraction and memory +* ranges alignment (6.0, 7.5.1.2.1). +*/ + addr =ioremap (rcrb,PCI_BASE_ADDRESS_0 +SZ_8 );
No failure check? This also only needs to map 4K at a time.
Quoted 11 lines
drivers/cxl/acpi.c +11
-0+ bar0 =readl (addr +PCI_BASE_ADDRESS_0 ); + bar1 =readl (addr +PCI_BASE_ADDRESS_1 ); +iounmap (addr); + +/* sanity check */ +if (bar0 & (PCI_BASE_ADDRESS_MEM_TYPE_1M |PCI_BASE_ADDRESS_SPACE_IO )) +return CXL_RESOURCE_NONE ; + + component_reg_phys = bar0 &PCI_BASE_ADDRESS_MEM_MASK ; +if (bar0 &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + component_reg_phys |= ((u64 )bar1) <<32 ;
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 305
Quoted 10 lines
drivers/cxl/acpi.c +10
-0+ +if (!component_reg_phys) +return CXL_RESOURCE_NONE ; + +/* +* Must be 8k aligned (size of combined CXL 1.1 Downstream and +* Upstream Port RCRBs). +*/ +if (component_reg_phys & (CXL_RCRB_SIZE -1 )) +return CXL_RESOURCE_NONE ;
This is open-coding the IS_ALIGNED() macro. More importantly, why is it using RCRB size for the component register block alignment? The component lock is 64K, and at least for CXL 2.0 devices it is 64K aligned (8.1.9.1 Register Block Offset Low), so I am not sure what this check is for?
---
Given that there are actual CXL RCH platforms in the wild I want this topic branch to be the first thing queued for v6.2. To help us coordinate I pushed:
https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=rch
...with the patches from this set accepted so far. You can use that as the baseline for the next spin.
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 83 lines · Dan Williams
Quoted 55 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)drivers/cxl/acpi.c +6
-4@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 256
Quoted 10 lines
drivers/cxl/acpi.c +3
-1if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{The core logic of this looks good, but this wants to be shared with the upstream port component register discovery.
Full disclosure I am reconciling these patches with an attempt that Dave Jiang made at this topic. Since your series hit the list first I will let it take the lead, but then fill it in with comments and learnings from Dave's effort.
So in this case Dave moved this into the
drivers/cxl/core/regs.cwith a function signature like:enum
cxl_rcrb{
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
Quoted 6 lines
resource_size_tcxl_rcrb_to_component(struct device *dev,
resource_size_trcrb_base, int len, enumcxl_rcrbwhich);...where @which alternates when called by
cxl_acpifor the downstream case, orcxl_memfor the upstream case.
Ok, I see where to go here. Could you point me to Dave's postings you are referring to? I checked linux-cxl and could not find anything related to RCRB or that changes regs.c.
Quoted 24 lines
Quoted 24 lines
drivers/cxl/acpi.c +22
-0+struct acpi_cedt_chbs *chbs = &ctx->chbs; +resource_size_t component_reg_phys, rcrb; +u32 bar0, bar1; +void *addr; + +if (!chbs->base) +return CXL_RESOURCE_NONE ; + +if (chbs->cxl_version !=ACPI_CEDT_CHBS_VERSION_CXL11 ) +return chbs->base; + +/* Extract RCRB */ + +if (chbs->length !=CXL_RCRB_SIZE ) +return CXL_RESOURCE_NONE ; + + rcrb = chbs->base; + +dev_dbg (ctx->dev,"RCRB found for UID %lld: 0x%08llx\n" , + ctx->uid, (u64 )rcrb); + +/*
Quoted 8 lines
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 284
Quoted 6 lines
drivers/cxl/acpi.c +6
-0+* RCRB's BAR[0..1] point to component block containing CXL +* subsystem component registers. MEMBAR extraction follows +* the PCI Base spec here, esp. 64 bit extraction and memory +* ranges alignment (6.0, 7.5.1.2.1). +*/ + addr =ioremap (rcrb,PCI_BASE_ADDRESS_0 +SZ_8 );No failure check? This also only needs to map 4K at a time.
Right, will add that.
Quoted 20 lines
Quoted 20 lines
drivers/cxl/acpi.c +19
-0+ bar0 =readl (addr +PCI_BASE_ADDRESS_0 ); + bar1 =readl (addr +PCI_BASE_ADDRESS_1 ); +iounmap (addr); + +/* sanity check */ +if (bar0 & (PCI_BASE_ADDRESS_MEM_TYPE_1M |PCI_BASE_ADDRESS_SPACE_IO )) +return CXL_RESOURCE_NONE ; + + component_reg_phys = bar0 &PCI_BASE_ADDRESS_MEM_MASK ; +if (bar0 &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + component_reg_phys |= ((u64 )bar1) <<32 ; + +if (!component_reg_phys) +return CXL_RESOURCE_NONE ; + +/* +* Must be 8k aligned (size of combined CXL 1.1 Downstream and +* Upstream Port RCRBs). +*/
Quoted 8 lines
drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 305
Quoted 2 lines
drivers/cxl/acpi.c +2
-0+if (component_reg_phys & (CXL_RCRB_SIZE -1 )) +return CXL_RESOURCE_NONE ;This is open-coding the
IS_ALIGNED()macro. More importantly, why is it using RCRB size for the component register block alignment? The component lock is 64K, and at least for CXL 2.0 devices it is 64K aligned (8.1.9.1 Register Block Offset Low), so I am not sure what this check is for?
True, this is a mistake and needs to be corrected. It is the component reg range which is 64k. Will also use IS_ALIGNED().
Quoted 11 lines
---
Given that there are actual CXL RCH platforms in the wild I want this topic branch to be the first thing queued for v6.2. To help us coordinate I pushed:
https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=rch
...with the patches from this set accepted so far. You can use that as the baseline for the next spin.
Yes, thanks for that branch and applying the first part.
-Robert
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 88 lines · Robert Richter
Quoted 87 lines · Dan Williams
Quoted 55 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)drivers/cxl/acpi.c +6
-4@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 256
Quoted 10 lines
drivers/cxl/acpi.c +3
-1if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{The core logic of this looks good, but this wants to be shared with the upstream port component register discovery.
Full disclosure I am reconciling these patches with an attempt that Dave Jiang made at this topic. Since your series hit the list first I will let it take the lead, but then fill it in with comments and learnings from Dave's effort.
So in this case Dave moved this into the
drivers/cxl/core/regs.cwith a function signature like:enum
cxl_rcrb{
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
resource_size_tcxl_rcrb_to_component(struct device *dev,
resource_size_trcrb_base, int len, enumcxl_rcrbwhich);
Quoted 6 lines
Quoted 2 lines
...where @which alternates when called by
cxl_acpifor the downstream case, orcxl_memfor the upstream case.Ok, I see where to go here. Could you point me to Dave's postings you are referring to? I checked linux-cxl and could not find anything related to RCRB or that changes
regs.c.
He was in the middle of tidying them when you posted your series, but I think it would not hurt to push them to a git tree so you can grab the bits and pieces you want.
Dave?
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 96 lines · Dan Williams
Quoted 88 lines · Robert Richter
Quoted 87 lines · Dan Williams
Quoted 55 lines · Robert Richter
A downstream port must be connected to a component register block. For restricted hosts the base address is determined from the RCRB. The RCRB is provided by the host's CEDT CHBS entry. Rework CEDT parser to get the RCRB and add code to extract the component register block from it.
RCRB's BAR[0..1] point to the component block containing CXL subsystem component registers. MEMBAR extraction follows the PCI base spec here, esp. 64 bit extraction and memory range alignment (6.0, 7.5.1.2.1).
Note: Right now the component register block is used for HDM decoder capability only which is optional for RCDs. If unsupported by the RCD, the HDM init will fail. It is future work to bypass it in this case.
Signed-off-by Terry Bowman <[email protected]>Signed-off-by Robert Richter <[email protected]>--- drivers/cxl/acpi.c | 79 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)drivers/cxl/acpi.c +6
-4@@ -9,6 +9,8 @@#include "cxlpci.h" #include "cxl.h" +#define CXL_RCRB_SIZE SZ_8K +static unsigned long cfmws_to_decoder_flags (int restrictions) {unsigned long flags =CXL_DECODER_F_ENABLE ;@@ -229,27 +231,82 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct cxl_chbs_context {struct device *dev;unsigned long long uid; -resource_size_t chbcr; +struct acpi_cedt_chbs chbs; }; -static int cxl_get_chbcr (union acpi_subtable_headers *header,void *arg, -const unsigned long end) +static int cxl_get_chbs (union acpi_subtable_headers *header,void *arg, +const unsigned long end) {struct cxl_chbs_context *ctx = arg;struct acpi_cedt_chbs *chbs; -if (ctx->chbcr) +if (ctx->chbs.base)return 0 ; chbs = (struct acpi_cedt_chbs *) header;drivers/cxl/acpi.c static int add_host_bridge_uport(struct device *match, void *arg) · line 256
Quoted 10 lines
drivers/cxl/acpi.c +3
-1if (ctx->uid != chbs->uid)return 0 ; - ctx->chbcr = chbs->base; + ctx->chbs = *chbs;return 0 ; } +static resource_size_t cxl_get_chbcr (struct cxl_chbs_context *ctx) +{The core logic of this looks good, but this wants to be shared with the upstream port component register discovery.
Full disclosure I am reconciling these patches with an attempt that Dave Jiang made at this topic. Since your series hit the list first I will let it take the lead, but then fill it in with comments and learnings from Dave's effort.
So in this case Dave moved this into the
drivers/cxl/core/regs.cwith a function signature like:enum
cxl_rcrb{
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
resource_size_tcxl_rcrb_to_component(struct device *dev,
resource_size_trcrb_base, int len, enumcxl_rcrbwhich);Quoted 6 lines
Quoted 2 lines
...where @which alternates when called by
cxl_acpifor the downstream case, orcxl_memfor the upstream case.Ok, I see where to go here. Could you point me to Dave's postings you are referring to? I checked linux-cxl and could not find anything related to RCRB or that changes
regs.c.
Quoted 5 lines
He was in the middle of tidying them when you posted your series, but I think it would not hurt to push them to a git tree so you can grab the bits and pieces you want.
Dave?
Looks like the list delivery is backed up, so I added Dave to the Cc:.
He pushed:
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch
...which was his original attempt and:
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch-robert
...which was an attempt to rebase on top of your bits.
The common RCRB mapping function is here:
I think the path forward is to build on that common RCRB code, fix cxl_acpi to register the pci host bridge device instead of the APCI device as the dport device, and then rely on a flag to skip over devm_enumerate_cxl_ports() in favor of just calling cxl_mem_find_port() directly in the RCIEP / RCH case. Then we can figure out what to do about RCDs that choose not to implement the HDM decoder capability which was forbidden in CXL 2.0, but now allowed in CXL 3.0.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 22 lines · Dave Jiang
On 10/24/2022 3:37 PM, Dan Williams wrote:
Dan Williams wrote:
Robert Richter wrote:Ok, I see where to go here. Could you point me to Dave's postings you are referring to? I checked linux-cxl and could not find anything related to RCRB or that changes
regs.c.He was in the middle of tidying them when you posted your series, but I think it would not hurt to push them to a git tree so you can grab the bits and pieces you want.
Dave?
Looks like the list delivery is backed up, so I added Dave to the Cc:.
He pushed:
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch
...which was his original attempt and:
Quoted 7 lines
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch-robert
...which was an attempt to rebase on top of your bits.
The common RCRB mapping function is here:
Thanks for the pointers.
Quoted 6 lines
I think the path forward is to build on that common RCRB code, fix
cxl_acpito register the pci host bridge device instead of the APCI device as the dport device, and then rely on a flag to skip overdevm_enumerate_cxl_ports()in favor of just callingcxl_mem_find_port()directly in the RCIEP / RCH case.
Yes, we can completely skip devm_enumerate_cxl_ports() now. Though, I am not convinced on using the pci host bridge as dport_dev as RCD and non-RCD mode will diverge too much then. Looking into details here.
Quoted 10 lines
Then we can figure out what to do
about RCDs that choose not to implement the HDM decoder capability which was forbidden in CXL 2.0, but now allowed in CXL 3.0.Hi Robert. As follow on to your work, I'm also working on reworking the hdm decoder enumeration. I have this table from Dan where rr - range register exist but not setup, RR - range register exist and setup, hdm - HDM decoder exist but not programmed, HDM - HDM decoders exist and programmed. And I'm trying to refactor the current code to cover all those scenarios:
Quoted 7 lines
rr RR rr hdm rr HDM RR hdm RR HDM D2 unsupported emulate RR enable HDM keep HDM enable HDM keep HDM D1 unsupported emulate RR enable HDM keep HDM enable HDM keep HDMThe current test device I have that's attached to RCH host, I'm seeing the RR has setup a single range, but none of the HDM decoders are programmed.
Right, HDM decoder init need to be changed next.
Thanks,
-Robert
Re: [PATCH v2 06/12] cxl/acpi: Extract component registers of restricted hosts from RCRB
Quoted 40 lines · Robert Richter
Quoted 22 lines · Dave Jiang
On 10/24/2022 3:37 PM, Dan Williams wrote:
Dan Williams wrote:
Robert Richter wrote:Ok, I see where to go here. Could you point me to Dave's postings you are referring to? I checked linux-cxl and could not find anything related to RCRB or that changes
regs.c.He was in the middle of tidying them when you posted your series, but I think it would not hurt to push them to a git tree so you can grab the bits and pieces you want.
Dave?
Looks like the list delivery is backed up, so I added Dave to the Cc:.
He pushed:
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch
...which was his original attempt and:
Quoted 7 lines
https://git.kernel.org/pub/scm/linux/kernel/git/djiang/linux.git/log/?h=cxl-rch-robert
...which was an attempt to rebase on top of your bits.
The common RCRB mapping function is here:
Thanks for the pointers.
Quoted 6 lines
I think the path forward is to build on that common RCRB code, fix
cxl_acpito register the pci host bridge device instead of the APCI device as the dport device, and then rely on a flag to skip overdevm_enumerate_cxl_ports()in favor of just callingcxl_mem_find_port()directly in the RCIEP / RCH case.
Quoted 3 lines
Yes, we can completely skip
devm_enumerate_cxl_ports()now. Though, I am not convinced on using the pci host bridge asdport_devas RCD and non-RCD mode will diverge too much then. Looking into details here.
Oh, I disagree with the initial implementation Dave had here. Both cases should be specifying the bridge device as the dport. That's a fixup that can go in now even without the RCD support.
As it is the tooling needs to jump through the physical_node attribute to provide the useful information in cxl list:
# cxl list -BTu -b ACPI.CXL
{
"bus":"root0",
"provider":"ACPI.CXL",
"nr_dports":1,
"dports":[
{
"dport":"ACPI0016:00",
"alias":"pci0000:34",
"id":"0x34"
}
]
}...and I think that should just swap to this in all cases:
# cxl list -BTu -b ACPI.CXL
{
"bus":"root0",
"provider":"ACPI.CXL",
"nr_dports":1,
"dports":[
{
"dport":"pci0000:34",
"alias":"ACPI0016:00",
"id":"0x34"
}
]
}PATCH v2 7/12 cxl: Remove dev_is_cxl_root_child() check in devm_cxl_enumerate_ports()
In reply to Robert RichterThe dev_is_cxl_root_child() check adds complexity to the control flow of the iterator loop in devm_cxl_enumerate_ports(). This check is unnecessary and can safely be removed: If the port of a dport_dev is connected to the cxl root port, the grandparent of dport_dev will be null. This is checked early in the iterator loop and the function is left in this case.
Drop this check to ease the control flow in devm_cxl_enumerate_ ports().
This change is a prerequisite of factoring out parts of the loop.
--- drivers/cxl/core/port.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/cxl/core/port.c +3 -10
@@ -1395,6 +1395,7 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)"found already registered port %s:%s\n" , dev_name(&port->dev), dev_name(port->uport)); rc =cxl_add_ep (dport, &cxlmd->dev); +put_device (&port->dev);/* * If the endpoint already exists in the port's list,
@@ -1403,19 +1404,11 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)* the parent_port lock as the current port may be being * reaped. */ -if (rc && rc != -EBUSY ) { -put_device (&port->dev); +if (rc && rc != -EBUSY )return rc; - }/* Any more ports to add between this one and the root? */ -if (!dev_is_cxl_root_child (&port->dev)) { -put_device (&port->dev); -continue ; - } - -put_device (&port->dev); -return 0 ; +continue ; } rc =add_port_attach_ep (cxlmd, uport_dev, dport_dev);
Signature · 1 line
2.30.2
RE: [PATCH v2 07/12] cxl: Remove dev_is_cxl_root_child() check in devm_cxl_enumerate_ports()
Quoted 11 lines · Robert Richter
The
dev_is_cxl_root_child()check adds complexity to the control flow of the iterator loop indevm_cxl_enumerate_ports(). This check is unnecessary and can safely be removed: If the port of adport_devis connected to the cxl root port, the grandparent ofdport_devwill be null. This is checked early in the iterator loop and the function is left in this case.Drop this check to ease the control flow in
devm_cxl_enumerate_ports().This change is a prerequisite of factoring out parts of the loop.
Ok, so this seems to be where we diverge on how an RCH topology maps into the CXL subsystem object hierarchy. The main observation going through this with Dave before this set came out is that devm_cxl_enumerate_ports() is a nop and should be skipped in the RCH case. devm_cxl_enumerate_ports() is only for discovering intermediate ports between a host bridge and an endpoint.
In a CXL VH topology a direct attached endpoint is downstream of a host-bridge's root port. In the CXL RCH topology the Root-Complex-Integrated-Endpoint is a *peer* of a host-bridge's root port. So one level of the hierarchy is removed and devm_cxl_enumerate_ports() can be skipped.
The proposal is to have cxl_mem_probe() do something like:
if (!cxlds->is_rcd) {
rc = devm_cxl_enumerate_ports(cxlmd);
if (rc)
return rc;
}The existing:
parent_port = cxl_mem_find_port(cxlmd, &dport);
...should do the right thing as long as cxl_acpi registers the host bridge as a dport with this change:
- dport = devm_cxl_add_dport(root_port, match, uid, ...
+ dport = devm_cxl_add_dport(root_port, pci_root->bus->bridge, uid, ...
That way the dport device is not the ACPI device but the 'struct pci_dev' for the host-bridge.
With that scheme in place and some cxl-cli fixups from Vishal we are seeing:
# cxl list -BEMPTu
{
"bus":"root0",
"provider":"ACPI.CXL",
"nr_dports":1,
"dports":[
{
"dport":"pci0000:38",
"id":"0x31"
}
],
"endpoints:root0":[
{
"endpoint":"endpoint1",
"host":"mem0",
"depth":1,
"memdev":{
"memdev":"mem0",
"pmem_size":0,
"ram_size":"16.00 GiB (17.18 GB)",
"serial":"0",
"numa_node":0,
"host":"0000:38:00.0"
}
}
]
}Does that make sense? I think this patchset gets a lot simpler if it does not try to make devm_cxl_enumerate_ports() understand the RCH topology.
Verma, Vishal L
maintainerRe: [PATCH v2 07/12] cxl: Remove dev_is_cxl_root_child() check in devm_cxl_enumerate_ports()
Quoted 2 lines · Dan Williams
Quoted 1 line · Robert Richter
<..>
Quoted 23 lines
With that scheme in place and some cxl-cli fixups from Vishal we are seeing:
# cxl list -BEMPTu { "bus":"root0", "provider":"ACPI.CXL", "nr_dports":1, "dports":[ { "dport":"pci0000:38", "id":"0x31" } ], "endpoints:root0":[ { "endpoint":"endpoint1", "host":"mem0", "depth":1, "memdev":{ "memdev":"mem0", "pmem_size":0,
Quoted 8 lines
"ram_size":"16.00 GiB (17.18 GB)", "serial":"0", "numa_node":0, "host":"0000:38:00.0" } } ] }
I was waiting to post the cxl-cli patches for this until the kernel discussion settles - but maybe this topology layout is settled enough?
In the meanwhile I've pushed a branch with these fixups here:
https://github.com/pmem/ndctl/tree/vv/rch-support
Quoted 4 lines
Does that make sense? I think this patchset gets a lot simpler if it does not try to make
devm_cxl_enumerate_ports()understand the RCH topology.
Re: [PATCH v2 07/12] cxl: Remove dev_is_cxl_root_child() check in devm_cxl_enumerate_ports()
Vishal,
Quoted 5 lines · Vishal L
I was waiting to post the cxl-cli patches for this until the kernel discussion settles - but maybe this topology layout is settled enough?
In the meanwhile I've pushed a branch with these fixups here:
https://github.com/pmem/ndctl/tree/vv/rch-support
I sent out the v3 series today, patch #1 contains the change of the port's reference device. You need to adopt the ndctl patches for this. To find the alias in sysfs, you need to lookup the firmware_node instead of the physical_node. I would suggest to change the implemenation to just check for both entries.
Thanks,
-Robert
PATCH v2 8/12 cxl: Factor out code in devm_cxl_enumerate_ports() to find_port_attach_ep()
In reply to Robert RichterFactor out the code to attach an EP to an existing port. It will be reused to implement RCH discovery.
--- drivers/cxl/core/port.c | 66 +++++++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 25 deletions(-)
drivers/cxl/core/port.c +41 -25
@@ -1286,6 +1286,36 @@ static resource_size_t find_component_registers(struct device *dev)return cxl_regmap_to_base (pdev, &map); } +static int find_port_attach_ep (struct cxl_memdev *cxlmd, +struct device *uport_dev, +struct device *dport_dev, +struct device *origin) +{ +struct device *dev = &cxlmd->dev; +struct cxl_dport *dport; +struct cxl_port *port; +int rc; + +dev_dbg (dev,"scan: origin: %s dport_dev: %s parent: %s\n" , +dev_name (origin),dev_name (dport_dev), + uport_dev ?dev_name (uport_dev) :"(null)" ); + + port =find_cxl_port (dport_dev, &dport); +if (!port) +return 0 ; + +dev_dbg (dev,"found already registered port %s:%s\n" , +dev_name (&port->dev),dev_name (port->uport)); + + rc =cxl_add_ep (dport, &cxlmd->dev); +put_device (&port->dev); +if (rc) +return rc; + +/* Continue to add more ports between this one and the root. */ +return 1 ; +} +static int add_port_attach_ep (struct cxl_memdev *cxlmd,struct device *uport_dev,struct device *dport_dev)
@@ -1373,8 +1403,6 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)for (iter = dev; iter; iter =grandparent (iter)) {struct device *dport_dev =grandparent (iter);struct device *uport_dev; -struct cxl_dport *dport; -struct cxl_port *port;if (!dport_dev)return 0 ;
@@ -1386,30 +1414,18 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)return -ENXIO ; } -dev_dbg (dev,"scan: iter: %s dport_dev: %s parent: %s\n" , -dev_name (iter),dev_name (dport_dev), -dev_name (uport_dev)); - port =find_cxl_port (dport_dev, &dport); -if (port) { -dev_dbg (&cxlmd->dev, -"found already registered port %s:%s\n" , -dev_name (&port->dev),dev_name (port->uport)); - rc =cxl_add_ep (dport, &cxlmd->dev); -put_device (&port->dev); - -/* -* If the endpoint already exists in the port's list, -* that's ok, it was added on a previous pass. -* Otherwise, retry in add_port_attach_ep() after taking -* the parent_port lock as the current port may be being -* reaped. -*/ -if (rc && rc != -EBUSY ) -return rc; - -/* Any more ports to add between this one and the root? */ + rc =find_port_attach_ep (cxlmd, uport_dev, dport_dev, iter); +/* +* If the endpoint already exists in the port's list, +* that's ok, it was added on a previous pass. +* Otherwise, retry in add_port_attach_ep() after taking +* the parent_port lock as the current port may be being +* reaped. +*/ +if (rc >0 || rc == -EBUSY )continue ; - } +if (rc) +return rc; rc =add_port_attach_ep (cxlmd, uport_dev, dport_dev);/* port missing, try to add parent */
Signature · 1 line
2.30.2
PATCH v2 9/12 cxl: Extend devm_cxl_enumerate_ports() to support restricted devices (RCDs)
In reply to Robert RichterThe PCIe Software View of an RCH and RCD is different to VH mode. An RCD is paired with an RCH and shows up as RCiEP. Its downstream and upstream ports are hidden to the PCI hierarchy. This different PCI topology requires a different handling of RCHs.
Extend devm_cxl_enumerate_ports() to support restricted devices (RCDs). If an RCD is detected all to do is to search its corresponding RCH's port and attach the EP to it. Update cxl_mem_find_port() for proper removal of the EP in delete_endpoint().
--- drivers/cxl/core/port.c | 45 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-)
drivers/cxl/core/port.c +44 -1
@@ -1384,16 +1384,56 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd,return rc; } +static inline bool is_cxl_restricted (struct cxl_memdev *cxlmd) +{ +struct device *parent = cxlmd->dev.parent; +if (!dev_is_pci (parent)) +return false; +return pci_pcie_type (to_pci_dev (parent)) ==PCI_EXP_TYPE_RC_END ; +} + +static int restricted_host_enumerate_port (struct cxl_memdev *cxlmd) +{ +struct device *dev, *dport_dev, *uport_dev; +int count; + +if (!is_cxl_restricted (cxlmd)) +return 0 ; + +/* +* The cxlmd is an RCD, the dport_dev of it is the PCI device +* and the uport_dev is its host bridge which is the parent of +* the PCI device. +*/ + dev = &cxlmd->dev;/* cxlmd */ + dport_dev = dev->parent;/* pci_dev */ + uport_dev = dev->parent->parent;/* pci_host_bridge */ + + count =find_port_attach_ep (cxlmd, uport_dev, dport_dev, dev); + +/* If missing the host is not yet ready. */ +if (!count) +return -EAGAIN ; + +return count; +} +int devm_cxl_enumerate_ports (struct cxl_memdev *cxlmd) {struct device *dev = &cxlmd->dev;struct device *iter; -int rc; +int count, rc; rc =devm_add_action_or_reset (&cxlmd->dev, cxl_detach_ep, cxlmd);if (rc)return rc; + count =restricted_host_enumerate_port (cxlmd); +if (count <0 ) +return count; +if (count) +return 0 ; +/* * Scan for and add all cxl_ports in this device's ancestry. * Repeat until no more ports are added. Abort if a port add
@@ -1445,6 +1485,9 @@ EXPORT_SYMBOL_NS_GPL(devm_cxl_enumerate_ports, CXL);struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd,struct cxl_dport **dport) { +if (is_cxl_restricted (cxlmd)) +return find_cxl_port (cxlmd->dev.parent, dport); +return find_cxl_port (grandparent (&cxlmd->dev), dport); }EXPORT_SYMBOL_NS_GPL (cxl_mem_find_port,CXL );
Signature · 1 line
2.30.2
PATCH v2 10/12 cxl: Do not ignore PCI config read errors in match_add_dports()
In reply to Robert RichterThe link capabilities of a PCI device are read when enumerating its dports. This is done by reading the PCI config space. If that fails port enumeration ignores that error. However, reading the PCI config space should reliably work.
To reduce some complexity to the code flow when factoring out parts of the code in match_add_dports() for later reuse, change this to throw an error.
--- drivers/cxl/core/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/pci.c +1 -1
@@ -47,7 +47,7 @@ static int match_add_dports(struct pci_dev *pdev, void *data)return 0 ;if (pci_read_config_dword (pdev,pci_pcie_cap (pdev) +PCI_EXP_LNKCAP , &lnkcap)) -return 0 ; +return -ENXIO ; rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map);if (rc)
Signature · 1 line
2.30.2
PATCH v2 11/12 cxl: Factor out code in match_add_dports() to pci_dev_add_dport()
In reply to Robert RichterFactor out the code to register a PCI device's dport to a port. It will be reused to implement RCD mode.
--- drivers/cxl/core/pci.c | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-)
drivers/cxl/core/pci.c +26 -11
@@ -29,14 +29,32 @@ struct cxl_walk_context {int count; }; +static int pci_dev_add_dport (struct pci_dev *pdev,struct cxl_port *port, +resource_size_t component_reg_phys) +{ +struct cxl_dport *dport; +u32 lnkcap, port_num; + +if (pci_read_config_dword (pdev,pci_pcie_cap (pdev) +PCI_EXP_LNKCAP , + &lnkcap)) +return -ENXIO ; + + port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); + dport =devm_cxl_add_dport (port, &pdev->dev, port_num, + component_reg_phys); +if (IS_ERR (dport)) +return PTR_ERR (dport); + +return 0 ; +} +static int match_add_dports (struct pci_dev *pdev,void *data) {struct cxl_walk_context *ctx = data;struct cxl_port *port = ctx->port;int type =pci_pcie_type (pdev);struct cxl_register_map map; -struct cxl_dport *dport; -u32 lnkcap, port_num; +resource_size_t component_reg_phys;int rc;if (pdev->bus != ctx->bus)
@@ -45,21 +63,18 @@ static int match_add_dports(struct pci_dev *pdev, void *data)return 0 ;if (type != ctx->type)return 0 ; -if (pci_read_config_dword (pdev,pci_pcie_cap (pdev) +PCI_EXP_LNKCAP , - &lnkcap)) -return -ENXIO ; rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map);if (rc)dev_dbg (&port->dev,"failed to find component registers\n" ); - port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - dport =devm_cxl_add_dport (port, &pdev->dev, port_num, -cxl_regmap_to_base (pdev, &map)); -if (IS_ERR (dport)) { - ctx->error =PTR_ERR (dport); -return PTR_ERR (dport); + component_reg_phys =cxl_regmap_to_base (pdev, &map); + rc =pci_dev_add_dport (pdev, port, component_reg_phys); +if (rc) { + ctx->error = rc; +return rc; } + ctx->count++;return 0 ;
Signature · 1 line
2.30.2
PATCH v2 12/12 cxl: Extend devm_cxl_port_enumerate_dports() to support restricted hosts (RCH)
In reply to Robert RichterThe PCIe Software View of an RCH and RCD is different to VH mode. An RCD is paired with an RCH and shows up as RCiEP. Its downstream and upstream ports are hidden to the PCI hierarchy. This different PCI topology requires a different handling of RCHs.
Extend devm_cxl_port_enumerate_dports() to support restricted hosts (RCH). If an RCH is detected, register its port as dport to the device. An RCH is found if the host's dev 0 func 0 devices is an RCiEP with an existing PCIe DVSEC for CXL Devices (ID 0000h).
--- drivers/cxl/core/pci.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
drivers/cxl/core/pci.c +36 -1
@@ -48,6 +48,37 @@ static int pci_dev_add_dport(struct pci_dev *pdev, struct cxl_port *port,return 0 ; } +static int restricted_host_enumerate_dport (struct cxl_port *port, +struct pci_bus *bus) +{ +struct pci_dev *pdev; +bool is_restricted_host; +int rc; + +/* Check CXL DVSEC of dev 0 func 0 */ + pdev =pci_get_slot (bus,PCI_DEVFN (0 ,0 )); + + is_restricted_host = pdev + && (pci_pcie_type (pdev) ==PCI_EXP_TYPE_RC_END ) + &&pci_find_dvsec_capability (pdev, +PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (is_restricted_host) + rc =pci_dev_add_dport (pdev, port,CXL_RESOURCE_NONE ); + +pci_dev_put (pdev); + +if (!is_restricted_host) +return 0 ; + +dev_dbg (bus->bridge,"CXL restricted host found\n" ); + +if (rc) +return rc; + +return 1 ; +} +static int match_add_dports (struct pci_dev *pdev,void *data) {struct cxl_walk_context *ctx = data;
@@ -91,11 +122,15 @@ int devm_cxl_port_enumerate_dports(struct cxl_port *port) {struct pci_bus *bus =cxl_port_to_pci_bus (port);struct cxl_walk_context ctx; -int type; +int type, count;if (!bus)return -ENXIO ; + count =restricted_host_enumerate_dport (port, bus); +if (count) +return count; +if (pci_is_root_bus (bus)) type =PCI_EXP_TYPE_ROOT_PORT ;else
Signature · 1 line
2.30.2