Add drivers for CXL ports and mem devices
Subsystems (3)
cover Add drivers for CXL ports and mem devices
This is the first set of patches from the RFC [1] for region creation. The patches enable port enumeration for endpoint devices, and enumeration of decoder resources for ports. In the RFC [1], I felt it necessary to post the consumer of this work, the region driver, so that it was clear why these patches were necessary. Because the region patches patches are less baked, and received no review in the RFC, they are excluded here. If you find yourself unclear about why these patches are interesting, go look at the RFC [1].
Each patch contains the list of changes from RFCv2. IMHO the following are the high level most important changes:
1. Rework cxl_pci to fix mailbox handling and allow for wait media ready.
2. DVSEC range information is passed from cxl_pci and checked
linux-pci is on the Cc since CXL lives in a parallel universe to PCI and some PCI mechanisms are reused here. Feedback from experts in that domain is very welcome.
What was requested and not changed:
1. Dropping global list of root ports.
2. Improving find_parent_cxl_port()
---
Summary
=======
Two new drivers are introduced to support Compute Express Link 2.0 [2] HDM decoder enumeration. While the existing cxl_acpi and cxl_pci drivers already create some of the necessary devices, they did not do full enumeration of decoders, and they did not do port enumeration for switches. Additionally, CXL 2.0 Root Port component registers are now handled as well.
cxl_port
========
The cxl_port driver is implemented within the cxl_port module. While loading of this module is optional, the other new drivers depend, and cxl_acpi depend on it for complete enumeration. The port driver is responsible for all activities around HDM decoder enumeration and programming. Introduced earlier, the concept of a port is an abstraction over CXL components with an upstream port, every host bridge, switch, and endpoint.
cxl_mem
=======
The cxl_mem driver's main job is to walk up the hierarchy to make the determination if it is CXL.mem routed, meaning, all components above it in the hierarchy are participating in the CXL.mem protocol. It is implemented within the cxl_mem module. As the host bridge ports are added by a platform specific driver, such as cxl_acpi, the scope of the mem driver can be reduced to scan for switches and ask cxl_core to work on enumerating them. With this done, the determination as to whether a device is CXL.mem routed can be done simply by checking if the struct device has a driver bound to it.
Results
=======
Running these patches should yield new devices and new drivers under /sys/bus/cxl/devices and /sys/bus/cxl/drivers. For example, in a standard QEMU run, using run_qemu [3]
/sys/bus/cxl/devices (new):
# The host bridge CHBS decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder1.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/decoder1.0 # mem0's decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder2.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2/decoder2.0 # mem1's decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder3.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3/decoder3.0 # mem0's port
lrwxrwxrwx 1 root root 0 Nov 19 15:23 port2 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2 # mem1's port
lrwxrwxrwx 1 root root 0 Nov 19 15:23 port3 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3
/sys/bus/cxl/drivers:
drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_mem
drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_port
---
[1]: https://lore.kernel.org/linux-cxl/[email protected]/T/#t
[2]: https://www.computeexpresslink.org/download-the-specification
[3]: https://github.com/pmem/run_qemu/
Ben Widawsky (23):
cxl: Rename CXL_MEM to CXL_PCI
cxl: Flesh out register names
cxl/pci: Extract device status check cxl/pci: Implement Interface Ready Timeout cxl/pci: Don't poll doorbell for mailbox access cxl/pci: Don't check media status for mbox access cxl/pci: Add new DVSEC definitions cxl/acpi: Map component registers for Root Ports
cxl: Introduce module_cxl_driver
cxl/core: Convert decoder range to resource cxl/core: Document and tighten up decoder APIs
cxl: Introduce endpoint decoders
cxl/core: Move target population locking to caller
cxl: Introduce topology host registration cxl/core: Store global list of root ports cxl/pci: Cache device DVSEC offset
cxl: Cache and pass DVSEC ranges
cxl/pci: Implement wait for media active cxl/pci: Store component register base in cxlds cxl/port: Introduce a port driver
cxl: Unify port enumeration for decoders cxl/mem: Introduce cxl_mem driver
cxl/mem: Disable switch hierarchies for now
.../driver-api/cxl/memory-devices.rst | 14 + drivers/cxl/Kconfig | 54 ++- drivers/cxl/Makefile | 6 +- drivers/cxl/acpi.c | 103 ++-- drivers/cxl/core/Makefile | 1 + drivers/cxl/core/bus.c | 439 ++++++++++++++++-- drivers/cxl/core/core.h | 3 + drivers/cxl/core/memdev.c | 2 +- drivers/cxl/core/pci.c | 119 +++++ drivers/cxl/core/regs.c | 60 ++- drivers/cxl/cxl.h | 73 ++- drivers/cxl/cxlmem.h | 27 ++ drivers/cxl/mem.c | 197 ++++++++ drivers/cxl/pci.c | 341 ++++++++++---- drivers/cxl/pci.h | 53 ++- drivers/cxl/port.c | 383 +++++++++++++++ tools/testing/cxl/Kbuild | 1 + tools/testing/cxl/mock_acpi.c | 4 +- 18 files changed, 1666 insertions(+), 214 deletions(-) create mode 100644 drivers/cxl/core/pci.c create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/port.c
base-commit: 53989fad1286e652ea3655ae3367ba698da8d2ff
Signature · 1 line
2.34.0
PATCH 1/23 cxl: Rename CXL_MEM to CXL_PCI
The cxl_mem module was renamed cxl_pci in commit 21e9f76733a8 ("cxl:
Rename mem to pci"). In preparation for adding an ancillary driver for cxl_memdev devices (registered on the cxl bus by cxl_pci), go ahead and rename CONFIG_CXL_MEM to CONFIG_CXL_PCI. Free up the CXL_MEM name for that new driver to manage CXL.mem endpoint operations.
---
Changes since RFCv2:
- Reword commit message (Dan)
- Reword Kconfig description (Dan)
--- drivers/cxl/Kconfig | 23 ++++++++++++----------- drivers/cxl/Makefile | 2 +- 2 files changed, 13 insertions(+), 12 deletions(-)
drivers/cxl/Kconfig +12 -11
@@ -13,25 +13,26 @@ menuconfig CXL_BUS if CXL_BUS -config CXL_MEM - tristate "CXL.mem: Memory Devices" +config CXL_PCI + tristate "PCI manageability" default CXL_BUS help - The CXL.mem protocol allows a device to act as a provider of - "System RAM" and/or "Persistent Memory" that is fully coherent - as if the memory was attached to the typical CPU memory - controller. + The CXL specification defines a "CXL memory device" sub-class in the + PCI "memory controller" base class of devices. Device's identified by + this class code provide support for volatile and / or persistent + memory to be mapped into the system address map (Host-managed Device + Memory (HDM)). - Say 'y/m' to enable a driver that will attach to CXL.mem devices for - configuration and management primarily via the mailbox interface. See - Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification for more - details. + Say 'y/m' to enable a driver that will attach to CXL memory expander + devices enumerated by the memory device class code for configuration + and management primarily via the mailbox interface. See Chapter 2.3 + Type 3 CXL Device in the CXL 2.0 specification for more details. If unsure say 'm'. config CXL_MEM_RAW_COMMANDS bool "RAW Command Interface for Memory Devices" - depends on CXL_MEM + depends on CXL_PCI help Enable CXL RAW command interface.
drivers/cxl/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += core/ -obj-$(CONFIG_CXL_MEM) += cxl_pci.o +obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
Signature · 1 line
2.34.0
Re: [PATCH 01/23] cxl: Rename CXL_MEM to CXL_PCI
Quoted 8 lines · On Fri, 19 Nov 2021 16:02:28 -0800 Ben Widawsky <[email protected]> wrote:
The
cxl_memmodule was renamedcxl_pciin commit21e9f76733a8("cxl:
Rename mem to pci"). In preparation for adding an ancillary driver forcxl_memdevdevices (registered on the cxl bus bycxl_pci), go ahead and renameCONFIG_CXL_MEMtoCONFIG_CXL_PCI. Free up theCXL_MEMname for that new driver to manage CXL.mem endpoint operations.Suggested-by Dan Williams <[email protected]>Signed-off-by Ben Widawsky <[email protected]>
Makes sense to me, particularly as it brings it inline with the file name.
Quoted 64 lines
---
Changes since RFCv2:
- Reword commit message (Dan)
- RewordKconfigdescription (Dan)--- drivers/cxl/Kconfig | 23 ++++++++++++----------- drivers/cxl/Makefile | 2 +- 2 files changed, 13 insertions(+), 12 deletions(-)drivers/cxl/Kconfig +12
-11@@ -13,25 +13,26 @@ menuconfig CXL_BUS if CXL_BUS -config CXL_MEM - tristate "CXL.mem: Memory Devices" +config CXL_PCI + tristate "PCI manageability" default CXL_BUS help - The CXL.mem protocol allows a device to act as a provider of - "System RAM" and/or "Persistent Memory" that is fully coherent - as if the memory was attached to the typical CPU memory - controller. + The CXL specification defines a "CXL memory device" sub-class in the + PCI "memory controller" base class of devices. Device's identified by + this class code provide support for volatile and / or persistent + memory to be mapped into the system address map (Host-managed Device + Memory (HDM)). - Say 'y/m' to enable a driver that will attach to CXL.mem devices for - configuration and management primarily via the mailbox interface. See - Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification for more - details. + Say 'y/m' to enable a driver that will attach to CXL memory expander + devices enumerated by the memory device class code for configuration + and management primarily via the mailbox interface. See Chapter 2.3 + Type 3 CXL Device in the CXL 2.0 specification for more details. If unsure say 'm'. config CXL_MEM_RAW_COMMANDS bool "RAW Command Interface for Memory Devices" - depends on CXL_MEM + depends on CXL_PCI help Enable CXL RAW command interface.drivers/cxl/Makefile +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += core/ -obj-$(CONFIG_CXL_MEM) += cxl_pci.o +obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
Re: [PATCH 01/23] cxl: Rename CXL_MEM to CXL_PCI
In reply to Ben WidawskyQuoted 8 lines · Ben Widawsky
The
cxl_memmodule was renamedcxl_pciin commit21e9f76733a8("cxl:
Rename mem to pci"). In preparation for adding an ancillary driver forcxl_memdevdevices (registered on the cxl bus bycxl_pci), go ahead and renameCONFIG_CXL_MEMtoCONFIG_CXL_PCI. Free up theCXL_MEMname for that new driver to manage CXL.mem endpoint operations.Suggested-by Dan Williams <[email protected]>
Quoted 68 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Reword commit message (Dan)
- RewordKconfigdescription (Dan)--- drivers/cxl/Kconfig | 23 ++++++++++++----------- drivers/cxl/Makefile | 2 +- 2 files changed, 13 insertions(+), 12 deletions(-)drivers/cxl/Kconfig +12
-11@@ -13,25 +13,26 @@ menuconfig CXL_BUS if CXL_BUS -config CXL_MEM - tristate "CXL.mem: Memory Devices" +config CXL_PCI + tristate "PCI manageability" default CXL_BUS help - The CXL.mem protocol allows a device to act as a provider of - "System RAM" and/or "Persistent Memory" that is fully coherent - as if the memory was attached to the typical CPU memory - controller. + The CXL specification defines a "CXL memory device" sub-class in the + PCI "memory controller" base class of devices. Device's identified by + this class code provide support for volatile and / or persistent + memory to be mapped into the system address map (Host-managed Device + Memory (HDM)). - Say 'y/m' to enable a driver that will attach to CXL.mem devices for - configuration and management primarily via the mailbox interface. See - Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification for more - details. + Say 'y/m' to enable a driver that will attach to CXL memory expander + devices enumerated by the memory device class code for configuration + and management primarily via the mailbox interface. See Chapter 2.3 + Type 3 CXL Device in the CXL 2.0 specification for more details. If unsure say 'm'. config CXL_MEM_RAW_COMMANDS bool "RAW Command Interface for Memory Devices" - depends on CXL_MEM + depends on CXL_PCI help Enable CXL RAW command interface.drivers/cxl/Makefile +1
-2@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_BUS) += core/ -obj-$(CONFIG_CXL_MEM) += cxl_pci.o +obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o --2.34.0
PATCH 2/23 cxl: Flesh out register names
In reply to Ben WidawskyGet a better naming scheme in place for upcoming additions.
---
Changes since RFCv2:
Use some abbreviations (Jonathan)
Prefix everything with CXL (Jonathan)
Remove new additions (Dan)
Original discussion motivating this occurred here:
https://lore.kernel.org/linux-pci/[email protected]/
--- drivers/cxl/pci.c | 14 +++++++------- drivers/cxl/pci.h | 19 ++++++++++--------- 2 files changed, 17 insertions(+), 16 deletions(-)
drivers/cxl/pci.c +7 -7
@@ -403,10 +403,10 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mastatic void cxl_decode_regblock (u32 reg_lo,u32 reg_hi,struct cxl_register_map *map) { - map->block_offset = - ((u64 )reg_hi <<32 ) | (reg_lo &CXL_REGLOC_ADDR_MASK ); - map->barno =FIELD_GET (CXL_REGLOC_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_REGLOC_RBI_MASK , reg_lo); + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); }/**
@@ -427,15 +427,15 @@ static int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,int regloc, i; regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID ); +CXL_DVSEC_REG_LOCATOR );if (!regloc)return -ENXIO ;pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - regloc +=PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ; - regblocks = (regloc_size -PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ) /8 ; + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ;for (i =0 ; i < regblocks; i++, regloc +=8 ) {u32 reg_lo, reg_hi;
drivers/cxl/pci.h +10 -9
@@ -7,17 +7,21 @@/* * See section 8.1 Configuration Space Registers in the CXL 2.0 -* Specification +* Specification. Names are taken straight from the specification with "CXL" and +* "DVSEC" redundancies removed. When obvious, abbreviations may be used. */ #define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20)#define PCI_DVSEC_VENDOR_ID_CXL 0x1E98 -#define PCI_DVSEC_ID_CXL 0x0 -#define PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID 0x8 -#define PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET 0xC +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ +#define CXL_DVSEC_PCIE_DEVICE 0 -/* BAR Indicator Register (BIR) */ -#define CXL_REGLOC_BIR_MASK GENMASK(2, 0) +/* CXL 2.0 8.1.9: Register Locator DVSEC */ +#define CXL_DVSEC_REG_LOCATOR 8 +#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC +#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16)/* Register Block Identifier (RBI) */ enum cxl_regloc_type {
@@ -28,7 +32,4 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; -#define CXL_REGLOC_RBI_MASK GENMASK(15, 8) -#define CXL_REGLOC_ADDR_MASK GENMASK(31, 16) -#endif /* __CXL_PCI_H__ */
Signature · 1 line
2.34.0
Re: [PATCH 02/23] cxl: Flesh out register names
Quoted 3 lines · On Fri, 19 Nov 2021 16:02:29 -0800 Ben Widawsky <[email protected]> wrote:
Get a better naming scheme in place for upcoming additions.
Signed-off-by Ben Widawsky <[email protected]>
Seems like a good balance to me between the different directions this sort of naming gets dragged in.
Quoted 91 lines
---
Changes since RFCv2:
Use some abbreviations (Jonathan)
Prefix everything with CXL (Jonathan)
Remove new additions (Dan)Original discussion motivating this occurred here:
https://lore.kernel.org/linux-pci/[email protected]/--- drivers/cxl/pci.c | 14 +++++++------- drivers/cxl/pci.h | 19 ++++++++++--------- 2 files changed, 17 insertions(+), 16 deletions(-)drivers/cxl/pci.c +7
-7@@ -403,10 +403,10 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mastatic void cxl_decode_regblock (u32 reg_lo,u32 reg_hi,struct cxl_register_map *map) { - map->block_offset = - ((u64 )reg_hi <<32 ) | (reg_lo &CXL_REGLOC_ADDR_MASK ); - map->barno =FIELD_GET (CXL_REGLOC_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_REGLOC_RBI_MASK , reg_lo); + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); }/** @@ -427,15 +427,15 @@ static int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,int regloc, i; regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID ); +CXL_DVSEC_REG_LOCATOR );if (!regloc)return -ENXIO ;pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - regloc +=PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ; - regblocks = (regloc_size -PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ) /8 ; + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ;for (i =0 ; i < regblocks; i++, regloc +=8 ) {u32 reg_lo, reg_hi;drivers/cxl/pci.h +10
-9@@ -7,17 +7,21 @@/* * See section 8.1 Configuration Space Registers in the CXL 2.0 -* Specification +* Specification. Names are taken straight from the specification with "CXL" and +* "DVSEC" redundancies removed. When obvious, abbreviations may be used. */ #define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20)#define PCI_DVSEC_VENDOR_ID_CXL 0x1E98 -#define PCI_DVSEC_ID_CXL 0x0 -#define PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID 0x8 -#define PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET 0xC +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ +#define CXL_DVSEC_PCIE_DEVICE 0 -/* BAR Indicator Register (BIR) */ -#define CXL_REGLOC_BIR_MASK GENMASK(2, 0) +/* CXL 2.0 8.1.9: Register Locator DVSEC */ +#define CXL_DVSEC_REG_LOCATOR 8 +#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC +#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16)/* Register Block Identifier (RBI) */ enum cxl_regloc_type {@@ -28,7 +32,4 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; -#define CXL_REGLOC_RBI_MASK GENMASK(15, 8) -#define CXL_REGLOC_ADDR_MASK GENMASK(31, 16) -#endif /* __CXL_PCI_H__ */
Re: [PATCH 02/23] cxl: Flesh out register names
In reply to Ben WidawskyQuoted 2 lines · Ben Widawsky
Get a better naming scheme in place for upcoming additions.
I prefer that subject lines and changelog have at least one concrete detail. Writing that out would identify that "rename REGLOC to REG_LOCATOR", is separate from "drop the unused PCI_DVSEC_ID_CXL definition", is different from "drop redundant prefixes of 'PCI' and 'DVSEC' from defines".
With some concrete details added to the changelog you can add:
Quoted 96 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
Use some abbreviations (Jonathan)
Prefix everything with CXL (Jonathan)
Remove new additions (Dan)Original discussion motivating this occurred here:
https://lore.kernel.org/linux-pci/[email protected]/--- drivers/cxl/pci.c | 14 +++++++------- drivers/cxl/pci.h | 19 ++++++++++--------- 2 files changed, 17 insertions(+), 16 deletions(-)drivers/cxl/pci.c +7
-7@@ -403,10 +403,10 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mastatic void cxl_decode_regblock (u32 reg_lo,u32 reg_hi,struct cxl_register_map *map) { - map->block_offset = - ((u64 )reg_hi <<32 ) | (reg_lo &CXL_REGLOC_ADDR_MASK ); - map->barno =FIELD_GET (CXL_REGLOC_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_REGLOC_RBI_MASK , reg_lo); + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); }/** @@ -427,15 +427,15 @@ static int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,int regloc, i; regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID ); +CXL_DVSEC_REG_LOCATOR );if (!regloc)return -ENXIO ;pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - regloc +=PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ; - regblocks = (regloc_size -PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET ) /8 ; + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ;for (i =0 ; i < regblocks; i++, regloc +=8 ) {u32 reg_lo, reg_hi;drivers/cxl/pci.h +10
-10@@ -7,17 +7,21 @@/* * See section 8.1 Configuration Space Registers in the CXL 2.0 -* Specification +* Specification. Names are taken straight from the specification with "CXL" and +* "DVSEC" redundancies removed. When obvious, abbreviations may be used. */ #define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20)#define PCI_DVSEC_VENDOR_ID_CXL 0x1E98 -#define PCI_DVSEC_ID_CXL 0x0 -#define PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID 0x8 -#define PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET 0xC +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ +#define CXL_DVSEC_PCIE_DEVICE 0 -/* BAR Indicator Register (BIR) */ -#define CXL_REGLOC_BIR_MASK GENMASK(2, 0) +/* CXL 2.0 8.1.9: Register Locator DVSEC */ +#define CXL_DVSEC_REG_LOCATOR 8 +#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC +#define CXL_DVSEC_REG_LOCATOR_BIR_MASK GENMASK(2, 0) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) +#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16)/* Register Block Identifier (RBI) */ enum cxl_regloc_type {@@ -28,7 +32,4 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; -#define CXL_REGLOC_RBI_MASK GENMASK(15, 8) -#define CXL_REGLOC_ADDR_MASK GENMASK(31, 16) -#endif /* __CXL_PCI_H__ */ --2.34.0
PATCH 3/23 cxl/pci: Extract device status check
In reply to Ben WidawskyThe Memory Device Status register is inspected in the same way for at least two flows in the CXL Type 3 Memory Device Software Guide (Revision: 1.0): 2.13.9 Device discovery and mailbox ready sequence, and 2.13.10 Media ready sequence. Extract this common functionality for use by both.
---
This patch did not exist in RFCv2
--- drivers/cxl/pci.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-)
drivers/cxl/pci.c +25 -8
@@ -182,6 +182,27 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,return 0 ; } +/* +* Implements roughly the bottom half of Figure 42 of the CXL Type 3 Memory +* Device Software Guide +*/ +static int check_device_status (struct cxl_dev_state *cxlds) +{ +const u64 md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); + +if (md_status &CXLMDEV_DEV_FATAL ) { +dev_err (cxlds->dev,"Fatal: replace device\n" ); +return -EIO ; + } + +if (md_status &CXLMDEV_FW_HALT ) { +dev_err (cxlds->dev,"FWHalt: reset or replace device\n" ); +return -EBUSY ; + } + +return 0 ; +} +/** * cxl_pci_mbox_get() - Acquire exclusive access to the mailbox. * @cxlds: The device state to gain access to.
@@ -231,17 +252,13 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* Hardware shouldn't allow a ready status but also have failure bits * set. Spit out an error, this should be a bug report */ - rc = -EFAULT ; -if (md_status &CXLMDEV_DEV_FATAL ) { -dev_err (dev,"mbox: reported ready, but fatal\n" ); + rc =check_device_status (cxlds); +if (rc) goto out; - } -if (md_status &CXLMDEV_FW_HALT ) { -dev_err (dev,"mbox: reported ready, but halted\n" ); - goto out; - } +if (CXLMDEV_RESET_NEEDED (md_status)) {dev_err (dev,"mbox: reported ready, but reset needed\n" ); + rc = -EFAULT ; goto out; }
Signature · 1 line
2.34.0
Re: [PATCH 03/23] cxl/pci: Extract device status check
Quoted 7 lines · On Fri, 19 Nov 2021 16:02:30 -0800 Ben Widawsky <[email protected]> wrote:
The Memory Device Status register is inspected in the same way for at least two flows in the CXL Type 3 Memory Device Software Guide (Revision: 1.0): 2.13.9 Device discovery and mailbox ready sequence, and 2.13.10 Media ready sequence. Extract this common functionality for use by both.
Signed-off-by Ben Widawsky <[email protected]>
Quoted 60 lines
---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-)drivers/cxl/pci.c +25
-8@@ -182,6 +182,27 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,return 0 ; } +/* +* Implements roughly the bottom half of Figure 42 of the CXL Type 3 Memory +* Device Software Guide +*/ +static int check_device_status (struct cxl_dev_state *cxlds) +{ +const u64 md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); + +if (md_status &CXLMDEV_DEV_FATAL ) { +dev_err (cxlds->dev,"Fatal: replace device\n" ); +return -EIO ; + } + +if (md_status &CXLMDEV_FW_HALT ) { +dev_err (cxlds->dev,"FWHalt: reset or replace device\n" ); +return -EBUSY ; + } + +return 0 ; +} +/** * cxl_pci_mbox_get() - Acquire exclusive access to the mailbox. * @cxlds: The device state to gain access to. @@ -231,17 +252,13 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* Hardware shouldn't allow a ready status but also have failure bits * set. Spit out an error, this should be a bug report */ - rc = -EFAULT ; -if (md_status &CXLMDEV_DEV_FATAL ) { -dev_err (dev,"mbox: reported ready, but fatal\n" ); + rc =check_device_status (cxlds); +if (rc) goto out; - } -if (md_status &CXLMDEV_FW_HALT ) { -dev_err (dev,"mbox: reported ready, but halted\n" ); - goto out; - } +if (CXLMDEV_RESET_NEEDED (md_status)) {dev_err (dev,"mbox: reported ready, but reset needed\n" ); + rc = -EFAULT ; goto out; }
Re: [PATCH 03/23] cxl/pci: Extract device status check
In reply to Ben WidawskyQuoted 6 lines · Ben Widawsky
The Memory Device Status register is inspected in the same way for at least two flows in the CXL Type 3 Memory Device Software Guide (Revision: 1.0): 2.13.9 Device discovery and mailbox ready sequence, and 2.13.10 Media ready sequence. Extract this common functionality for use by both.
Can you translate this into CXL specification terms? See below for the rationale...
Quoted 13 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-)drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, · line 187
Quoted 6 lines
drivers/cxl/pci.c +3
-0@@ -182,6 +182,27 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds,return 0 ; } +/* +* Implements roughly the bottom half of Figure 42 of the CXL Type 3 Memory +* Device Software Guide
I do appreciate that document for working through some of the concerns that system software might have for various CXL flows, but at the same time it's not authoritative. I.e. it is not a specification itself and it depends on the CXL specification as the "source of truth". So for Linux commentary I would translate the guide's recommendations back into the base truth from the CXL specification.
There will be places where Linux goes a different direction than the software guide so I do not want to set any expectations that those excursions are a bug, or otherwise require someone to consult a specific hardware vendor's software guide.
Especially in this case when the logic is simply "check a couple fatal status flags", the base specification is sufficient and the original code made no reference to the guide.
drivers/cxl/pci.c static int __cxl_pci_mbox_send_cmd(struct cxl_dev_state *cxlds, · line 194
Quoted 7 lines
drivers/cxl/pci.c +7
-0+*/ +static int check_device_status (struct cxl_dev_state *cxlds) +{ +const u64 md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); + +if (md_status &CXLMDEV_DEV_FATAL ) { +dev_err (cxlds->dev,"Fatal: replace device\n" );
The specification says "replace device", I disagree that the kernel should be recommending that the device by replaced. Just report what the driver does, and that's probably easier if the error messages are left to the caller.
const u64 md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
if (md_status & (CXLMDEV_DEV_FATAL | CXLMDEV_FW_HALT)) {
dev_err(dev, "mbox: failed to acquire, device state:%s%s\n",
md_status & CXLMDEV_DEV_FATAL ? " fatal" : "",
md_status & CXLMDEV_FW_HALT ? " firmware-halt" : "");
return -EIO;
}...i.e. it's not clear to me the helper helps.
Quoted 21 lines
drivers/cxl/pci.c +11
-3+return -EIO ; + } + +if (md_status &CXLMDEV_FW_HALT ) { +dev_err (cxlds->dev,"FWHalt: reset or replace device\n" ); +return -EBUSY ; + } + +return 0 ; +} +/** * cxl_pci_mbox_get() - Acquire exclusive access to the mailbox. * @cxlds: The device state to gain access to. @@ -231,17 +252,13 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* Hardware shouldn't allow a ready status but also have failure bits * set. Spit out an error, this should be a bug report */ - rc = -EFAULT ; -if (md_status &CXLMDEV_DEV_FATAL ) { -dev_err (dev,"mbox: reported ready, but fatal\n" );
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 259
Quoted 10 lines
drivers/cxl/pci.c +3
-5+ rc =check_device_status (cxlds); +if (rc) goto out; - } -if (md_status &CXLMDEV_FW_HALT ) { -dev_err (dev,"mbox: reported ready, but halted\n" ); - goto out; - } +if (CXLMDEV_RESET_NEEDED (md_status)) {
I think this check needs to go. If the reset is needed because of one of the above failure statuses then the function will have already error exited. If the reset is needed because media is disabled that should not be fatal for mailbox operations. It could be useful to do some interrogation of *why* media is disabled.
Quoted 8 lines
drivers/cxl/pci.c +1
-1dev_err (dev,"mbox: reported ready, but reset needed\n" ); + rc = -EFAULT ; goto out; } --2.34.0
PATCH 4/23 cxl/pci: Implement Interface Ready Timeout
In reply to Ben WidawskyThe original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in 8adaf747c9f0 ("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
---
This patch did not exist in RFCv2
--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/cxl/pci.c +29 -0
@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h>
@@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds) {const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented +* prior to the ECN, wait the max amount of time no matter what the +* device says. +*/ + timeout = jiffies +256 *HZ ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + +do { + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (md_status &CXLMDEV_MBOX_IF_READY ) +break ; +if (msleep_interruptible (100 )) +break ; + }while (!time_after (jiffies, timeout)); + +/* It's assumed that once the interface is ready, it will remain ready. */ +if (!(md_status &CXLMDEV_MBOX_IF_READY )) +return -EIO ; cxlds->mbox_send = cxl_pci_mbox_send; cxlds->payload_size =
Signature · 1 line
2.34.0
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 11 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)
drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented
I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Jonathan
Quoted 23 lines
drivers/cxl/pci.c +20
-0+* prior to the ECN, wait the max amount of time no matter what the +* device says. +*/ + timeout = jiffies +256 *HZ ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + +do { + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (md_status &CXLMDEV_MBOX_IF_READY ) +break ; +if (msleep_interruptible (100 )) +break ; + }while (!time_after (jiffies, timeout)); + +/* It's assumed that once the interface is ready, it will remain ready. */ +if (!(md_status &CXLMDEV_MBOX_IF_READY )) +return -EIO ; cxlds->mbox_send = cxl_pci_mbox_send; cxlds->payload_size =
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.
Quoted 8 lines
Quoted 5 lines
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Reasonable to assume is certainly not the same as "in reality". I can soften this wording.
Quoted 38 lines
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Quoted 2 lines
Jonathan
Let me pose it a different way, if a device advertises 1s, but for whatever takes 4s to come up, should we penalize it over the device advertising 256s? The way this field is defined in the spec would [IMHO] lead vendors to simply put the max field in there to game the driver, so why not start off with just insisting they don't?
Quoted 24 lines
Quoted 23 lines
drivers/cxl/pci.c +20
-0+* prior to the ECN, wait the max amount of time no matter what the +* device says. +*/ + timeout = jiffies +256 *HZ ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + +do { + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (md_status &CXLMDEV_MBOX_IF_READY ) +break ; +if (msleep_interruptible (100 )) +break ; + }while (!time_after (jiffies, timeout)); + +/* It's assumed that once the interface is ready, it will remain ready. */ +if (!(md_status &CXLMDEV_MBOX_IF_READY )) +return -EIO ; cxlds->mbox_send = cxl_pci_mbox_send; cxlds->payload_size =
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 60 lines · On Mon, 22 Nov 2021 09:17:31 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.Quoted 8 lines
Quoted 5 lines
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Reasonable to assume is certainly not the same as "in reality". I can soften this wording.
Quoted 38 lines
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Quoted 5 lines
Quoted 2 lines
Jonathan
Let me pose it a different way, if a device advertises 1s, but for whatever takes 4s to come up, should we penalize it over the device advertising 256s?
Yes, because it is buggy. A compliance test should have failed on this anyway.
Quoted 4 lines
The
way this field is defined in the spec would [IMHO] lead vendors to simply put the max field in there to game the driver, so why not start off with just insisting they don't?
Given reading this value and getting a big number gives the implication that the device is meant to be really slow to initialize, I'd expect that to push vendors a little in the directly of putting realistic values in).
Maybe we should print the value in the log to make them look silly ;)
Jonathan
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
drivers/cxl/pci.c +20
-0+* prior to the ECN, wait the max amount of time no matter what the +* device says. +*/ + timeout = jiffies +256 *HZ ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + +do { + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (md_status &CXLMDEV_MBOX_IF_READY ) +break ; +if (msleep_interruptible (100 )) +break ; + }while (!time_after (jiffies, timeout)); + +/* It's assumed that once the interface is ready, it will remain ready. */ +if (!(md_status &CXLMDEV_MBOX_IF_READY )) +return -EIO ; cxlds->mbox_send = cxl_pci_mbox_send; cxlds->payload_size =
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 77 lines · Jonathan Cameron
Quoted 60 lines · On Mon, 22 Nov 2021 09:17:31 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.Quoted 8 lines
Quoted 5 lines
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Reasonable to assume is certainly not the same as "in reality". I can soften this wording.
Quoted 38 lines
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Quoted 5 lines
Quoted 2 lines
Jonathan
Let me pose it a different way, if a device advertises 1s, but for whatever takes 4s to come up, should we penalize it over the device advertising 256s?
Yes, because it is buggy. A compliance test should have failed on this anyway.
Quoted 4 lines
The
way this field is defined in the spec would [IMHO] lead vendors to simply put the max field in there to game the driver, so why not start off with just insisting they don't?
Quoted 5 lines
Given reading this value and getting a big number gives the implication that the device is meant to be really slow to initialize, I'd expect that to push vendors a little in the directly of putting realistic values in).
Maybe we should print the value in the log to make them look silly ;)
A print message on the way to a static default timeout value is about all a device's self reported timeout is useful.
"device not ready after waiting %d seconds, continuing to wait up to %d seconds"
It's also not clear to me that the Linux default timeout should be so generous at 256 seconds. It might be suitable to just complain about devices that are taking more than 60 seconds to initialize with an option to override that timeout for odd outliers. Otherwise encourage hardware implementations to beat the Linux timeout value to get support out of the box.
I notice that not even libata waits more than a minute for a given device to finish post-reset shenanigans, so might as well set 60 seconds as what the driver will tolerate out of the box.
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 97 lines · Dan Williams
Quoted 77 lines · Jonathan Cameron
Quoted 60 lines · On Mon, 22 Nov 2021 09:17:31 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.Quoted 8 lines
Quoted 5 lines
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Reasonable to assume is certainly not the same as "in reality". I can soften this wording.
Quoted 38 lines
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Quoted 5 lines
Quoted 2 lines
Jonathan
Let me pose it a different way, if a device advertises 1s, but for whatever takes 4s to come up, should we penalize it over the device advertising 256s?
Yes, because it is buggy. A compliance test should have failed on this anyway.
Quoted 4 lines
The
way this field is defined in the spec would [IMHO] lead vendors to simply put the max field in there to game the driver, so why not start off with just insisting they don't?Quoted 5 lines
Given reading this value and getting a big number gives the implication that the device is meant to be really slow to initialize, I'd expect that to push vendors a little in the directly of putting realistic values in).
Maybe we should print the value in the log to make them look silly ;)
A print message on the way to a static default timeout value is about all a device's self reported timeout is useful.
"device not ready after waiting %d seconds, continuing to wait up to %d seconds"
It's also not clear to me that the Linux default timeout should be so generous at 256 seconds. It might be suitable to just complain about devices that are taking more than 60 seconds to initialize with an option to override that timeout for odd outliers. Otherwise encourage hardware implementations to beat the Linux timeout value to get support out of the box.
Quoted 3 lines
I notice that not even libata waits more than a minute for a given device to finish post-reset shenanigans, so might as well set 60 seconds as what the driver will tolerate out of the box.
60s is infinity, so 4x * infinity doesn't really make much difference does it :P?
In my opinion if we're going to pick a limit, might as well tie it to a spec definition rather than 60s.. Perhaps 60s has some relevance I'm unaware of, but it seems equally arbitrary to me.
Re: [PATCH 04/23] cxl/pci: Implement Interface Ready Timeout
Quoted 99 lines · Ben Widawsky
Quoted 97 lines · Dan Williams
Quoted 77 lines · Jonathan Cameron
Quoted 60 lines · On Mon, 22 Nov 2021 09:17:31 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:31 -0800 Ben Widawsky <[email protected]> wrote:
The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggy back off of, since the latter doesn't have a defined timeout. This functionality, introduced in
8adaf747c9f0("cxl/mem: Find device capabilities"), can now be improved since a timeout has been defined with an ECN to the 2.0 spec.Quoted 8 lines
Quoted 5 lines
While devices implemented prior to the ECN could have an arbitrarily long wait and still be within spec, the max ECN value (256s) is chosen as the default for all devices. All vendors in the consortium agreed to this amount and so it is reasonable to assume no devices made will exceed this amount.
Optimistic :)
Reasonable to assume is certainly not the same as "in reality". I can soften this wording.
Quoted 38 lines
Quoted 22 lines
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)drivers/cxl/pci.c +1
-0@@ -2,6 +2,7 @@/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/module.h> +#include <linux/delay.h> #include <linux/sizes.h> #include <linux/mutex.h> #include <linux/list.h> @@ -298,6 +299,34 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cstatic int cxl_pci_setup_mailbox (struct cxl_dev_state *cxlds)drivers/cxl/pci.c static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c · line 309
Quoted 10 lines
drivers/cxl/pci.c +8
-0{const int cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +unsigned long timeout; +u64 md_status; +int rc; + +/* +* CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to +* dictate how long to wait for the mailbox to become ready. For +* simplicity, and to handle devices that might have been implemented I'm not keen on the 'for simplicity' argument here. If the device is advertising a lower value, then that is what we should use. It's fine to wait the max time if nothing is specified. It'll cost us a few lines of code at most unless I am missing something...
Quoted 5 lines
Quoted 2 lines
Jonathan
Let me pose it a different way, if a device advertises 1s, but for whatever takes 4s to come up, should we penalize it over the device advertising 256s?
Yes, because it is buggy. A compliance test should have failed on this anyway.
Quoted 4 lines
The
way this field is defined in the spec would [IMHO] lead vendors to simply put the max field in there to game the driver, so why not start off with just insisting they don't?Quoted 5 lines
Given reading this value and getting a big number gives the implication that the device is meant to be really slow to initialize, I'd expect that to push vendors a little in the directly of putting realistic values in).
Maybe we should print the value in the log to make them look silly ;)
A print message on the way to a static default timeout value is about all a device's self reported timeout is useful.
"device not ready after waiting %d seconds, continuing to wait up to %d seconds"
It's also not clear to me that the Linux default timeout should be so generous at 256 seconds. It might be suitable to just complain about devices that are taking more than 60 seconds to initialize with an option to override that timeout for odd outliers. Otherwise encourage hardware implementations to beat the Linux timeout value to get support out of the box.
Quoted 6 lines
Quoted 3 lines
I notice that not even libata waits more than a minute for a given device to finish post-reset shenanigans, so might as well set 60 seconds as what the driver will tolerate out of the box.
60s is infinity, so 4x * infinity doesn't really make much difference does it :P?
1 minute is half the hung task timeout in case something accidentally did an uninterruptible sleep on probe completion event. 4 minutes is on the order of what it takes a large server to boot. A single device needs as much time as a server to boot?
Quoted 3 lines
In my opinion if we're going to pick a limit, might as well tie it to a spec definition rather than 60s.. Perhaps 60s has some relevance I'm unaware of, but it seems equally arbitrary to me.
4 minutes just seems an unreasonable amount of time to wait to make a decision that something is likely broken. If the industry actually builds devices that nominally take multiple minutes to boot it's already going to be in the realm of something custom in terms of application expectations for when the server is ready. I'll buy you a beverage of your choice if someone actually builds such a thing.
PATCH 5/23 cxl/pci: Don't poll doorbell for mailbox access
In reply to Ben WidawskyThe expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in 8adaf747c9f0 ("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part of probe() it's no longer necessary to use the piggyback.
With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
---
This patch did not exist in RFCv2
--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)
drivers/cxl/pci.c +6 -19
@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define +* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. */ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
Signature · 1 line
2.34.0
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 2 lines · On Fri, 19 Nov 2021 16:02:32 -0800 Ben Widawsky <[email protected]> wrote:
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface.
Reword this? Perhaps
"The expectation is that the mailbox interface ready bit will be set
at the start of any access through the mailbox interface."
Quoted 10 lines
Therefore, waiting for the
doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.
Quoted 5 lines
With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>
Trivial comment inline - with that fixed either by calling it out, or by pulling it out of this patch.
Quoted 11 lines
---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 224
Quoted 5 lines
drivers/cxl/pci.c +1
-1@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the
Whilst it's trivial, I'd prefer white space cleanup in separate patches. I guess this one is obvious enough to just call out in the patch description though.
Quoted 28 lines
drivers/cxl/pci.c +5
-18* bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define +* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. */ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 42 lines · Jonathan Cameron
Quoted 2 lines · On Fri, 19 Nov 2021 16:02:32 -0800 Ben Widawsky <[email protected]> wrote:
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface.
Reword this? Perhaps
"The expectation is that the mailbox interface ready bit will be set
at the start of any access through the mailbox interface."Quoted 10 lines
Therefore, waiting for the
doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.Quoted 5 lines
With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>Trivial comment inline - with that fixed either by calling it out, or by pulling it out of this patch.
Reviewed-by Jonathan Cameron <[email protected]>Quoted 12 lines
---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +0
-0@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)
Quoted 9 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 224
Quoted 4 lines
drivers/cxl/pci.c +1
-1/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the Whilst it's trivial, I'd prefer white space cleanup in separate patches. I guess this one is obvious enough to just call out in the patch description though.
Okay. I'll keep this in mind for the future, and just fixup the commit messages with your suggestion and this, now.
Thanks.
Ben
Quoted 29 lines
Quoted 28 lines
drivers/cxl/pci.c +5
-18* bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define +* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. */ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
In reply to Ben WidawskyQuoted 41 lines · Ben Widawsky
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in
8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +1
-8@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 230
Quoted 10 lines
drivers/cxl/pci.c +5
-5-* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define +* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN.
Can we just drop comment block altogether? Outside of cxl_pci_setup_mailbox() the only time the mailbox status should be checked is after a doorbell timeout after submitting a command.
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 234
Quoted 10 lines
drivers/cxl/pci.c +0
-6*/ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 48 lines · Dan Williams
Quoted 47 lines · Ben Widawsky
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in
8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +2
-13@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define
Quoted 9 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 230
Quoted 4 lines
drivers/cxl/pci.c +4
-0+* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. Can we just drop comment block altogether? Outside of
cxl_pci_setup_mailbox()the only time the mailbox status should be checked is after a doorbell timeout after submitting a command.
Yes, I think it's fine to drop it.
Quoted 11 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 234
Quoted 10 lines
drivers/cxl/pci.c +0
-6*/ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
Quoted 3 lines
This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
The message is wrong, but I think the logic is still valuable. How about:
"mbox: reported interface ready, but mbox not ready"
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 73 lines · Ben Widawsky
Quoted 48 lines · Dan Williams
Quoted 47 lines · Ben Widawsky
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in
8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +2
-13@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define Quoted 9 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 230
Quoted 4 lines
drivers/cxl/pci.c +4
-0+* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. Can we just drop comment block altogether? Outside of
cxl_pci_setup_mailbox()the only time the mailbox status should be checked is after a doorbell timeout after submitting a command.Yes, I think it's fine to drop it.
Quoted 11 lines
Quoted 10 lines
drivers/cxl/pci.c +0
-6*/ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );
Quoted 6 lines
Quoted 3 lines
This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
The message is wrong, but I think the logic is still valuable. How about:
"mbox: reported interface ready, but mbox not ready"
You mean check this every time even though the spec says the driver only needs to check it once per-reset?
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 78 lines · Dan Williams
Quoted 77 lines · Ben Widawsky
Quoted 48 lines · Dan Williams
Quoted 47 lines · Ben Widawsky
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in
8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +2
-13@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define Quoted 9 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 230
Quoted 4 lines
drivers/cxl/pci.c +4
-0+* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. Can we just drop comment block altogether? Outside of
cxl_pci_setup_mailbox()the only time the mailbox status should be checked is after a doorbell timeout after submitting a command.Yes, I think it's fine to drop it.
Quoted 14 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 234
Quoted 10 lines
drivers/cxl/pci.c +0
-6*/ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
Quoted 5 lines
Quoted 2 lines
The message is wrong, but I think the logic is still valuable. How about:
"mbox: reported interface ready, but mbox not ready"You mean check this every time even though the spec says the driver only needs to check it once per-reset?
Unfortunately it does not say that. "... it shall remain set until the next reset or the device encounters an error that prevents any mailbox communication."
Once we have real error checking in place, this could go away, though I see no harm in leaving it.
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 86 lines · Ben Widawsky
Quoted 78 lines · Dan Williams
Quoted 77 lines · Ben Widawsky
Quoted 48 lines · Dan Williams
Quoted 47 lines · Ben Widawsky
The expectation is that the mailbox interface ready bit is the first step in access through the mailbox interface. Therefore, waiting for the doorbell busy bit to be clear would imply that the mailbox interface is ready. The original driver implementation used the doorbell timeout for the Mailbox Interface Ready bit to piggyback off of, since the latter doesn't have a defined timeout (introduced in
8adaf747c9f0("cxl/mem:
Find device capabilities"), a timeout has since been defined with an ECN to the 2.0 spec). With the current driver waiting for mailbox interface ready as a part ofprobe()it's no longer necessary to use the piggyback.With the piggybacking no longer necessary it doesn't make sense to check doorbell status when acquiring the mailbox. It will be checked during the normal mailbox exchange protocol.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-)drivers/cxl/pci.c +2
-13@@ -221,27 +221,14 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)/* * XXX: There is some amount of ambiguity in the 2.0 version of the spec -* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the +* around the mailbox interface ready (8.2.8.5.1.1). The purpose of the * bit is to allow firmware running on the device to notify the driver -* that it's ready to receive commands. It is unclear if the bit needs -* to be read for each transaction mailbox, ie. the firmware can switch -* it on and off as needed. Second, there is no defined timeout for -* mailbox ready, like there is for the doorbell interface. -* -* Assumptions: -* 1. The firmware might toggle the Mailbox Interface Ready bit, check -* it for every command. -* -* 2. If the doorbell is clear, the firmware should have first set the -* Mailbox Interface Ready bit. Therefore, waiting for the doorbell -* to be ready is sufficient. +* that it's ready to receive commands. The spec does not clearly define Quoted 9 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 230
Quoted 4 lines
drivers/cxl/pci.c +4
-0+* under what conditions the bit may get set or cleared. As of the 2.0 +* base specification there was no defined timeout for mailbox ready, +* like there is for the doorbell interface. This was fixed with an ECN, +* but it's possible early devices implemented this before the ECN. Can we just drop comment block altogether? Outside of
cxl_pci_setup_mailbox()the only time the mailbox status should be checked is after a doorbell timeout after submitting a command.Yes, I think it's fine to drop it.
Quoted 14 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 234
Quoted 10 lines
drivers/cxl/pci.c +0
-6*/ - rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
Quoted 5 lines
Quoted 2 lines
The message is wrong, but I think the logic is still valuable. How about:
"mbox: reported interface ready, but mbox not ready"You mean check this every time even though the spec says the driver only needs to check it once per-reset?
Quoted 6 lines
Unfortunately it does not say that. "... it shall remain set until the next reset or the device encounters an error that prevents any mailbox communication."
Once we have real error checking in place, this could go away, though I see no harm in leaving it.
Right, there's no harm in the check, it just seems overly paranoid to me if it was already checked once. Until a doorbell timeout happens it's an extra MMIO cycle that can saved for a "what happened?" check after a timeout.
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
On 21-11-29 11:18:36, Dan Williams wrote:
[snip]
Quoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 234
Quoted 9 lines
drivers/cxl/pci.c +0
-6- rc =cxl_pci_mbox_wait_for_doorbell (cxlds); -if (rc) { -dev_warn (dev,"Mailbox interface not ready\n" ); - goto out; - } - md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET );if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" );This error message is obsolete since nothing is pre-checking the mailbox anymore, and per above I see no problem waiting to check the status until after the mailbox has failed to respond after a timeout.
Quoted 5 lines
Quoted 2 lines
The message is wrong, but I think the logic is still valuable. How about:
"mbox: reported interface ready, but mbox not ready"You mean check this every time even though the spec says the driver only needs to check it once per-reset?
Unfortunately it does not say that. "... it shall remain set until the next reset or the device encounters an error that prevents any mailbox communication."
Quoted 7 lines
Quoted 2 lines
Once we have real error checking in place, this could go away, though I see no harm in leaving it.
Right, there's no harm in the check, it just seems overly paranoid to me if it was already checked once. Until a doorbell timeout happens it's an extra MMIO cycle that can saved for a "what happened?" check after a timeout.
Well I suspect we're just rearranging the deck chairs on the Titanic now, but...
I see doorbell timeouts as disconnected from whether or not the mailbox interface is ready. If they were the same, we wouldn't need both bits and we could just wait extra long for the doorbell when probing.
In other words, I expect if the interface goes unready, doorbell timeout will occur, but I don't think we should assume if doorbell timeout occurs, the interface is no longer ready. I don't purport to know why a doorbell timeout might occur while the interface remains available (likely a firmware bug, I presume).
It does seem interesting to check if the interface is no longer ready on timeout though.
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
On Mon, Nov 29, 2021 at 11:32 AM Ben Widawsky <[email protected]> wrote:
[..]
Quoted 7 lines
Quoted 5 lines
Right, there's no harm in the check, it just seems overly paranoid to me if it was already checked once. Until a doorbell timeout happens it's an extra MMIO cycle that can saved for a "what happened?" check after a timeout.
Well I suspect we're just rearranging the deck chairs on the Titanic now, but...
Not so much, just trying to get this driver in line with other error handling designs.
Quoted 4 lines
I see doorbell timeouts as disconnected from whether or not the mailbox interface is ready. If they were the same, we wouldn't need both bits and we could just wait extra long for the doorbell when probing.
Quoted 8 lines
In other words, I expect if the interface goes unready, doorbell timeout will occur, but I don't think we should assume if doorbell timeout occurs, the interface is no longer ready. I don't purport to know why a doorbell timeout might occur while the interface remains available (likely a firmware bug, I presume).
It does seem interesting to check if the interface is no longer ready on timeout though.
So I'm just modeling this off of NVME error handling where there is a Controller Fatal Status bit that could be checked every transaction, but instead the driver waits until a command timeout to collect if the device went fatal / not-ready.
Re: [PATCH 05/23] cxl/pci: Don't poll doorbell for mailbox access
Quoted 8 lines · Dan Williams
On Mon, Nov 29, 2021 at 11:32 AM Ben Widawsky <[email protected]> wrote:
[..]Quoted 6 lines
Quoted 5 lines
Right, there's no harm in the check, it just seems overly paranoid to me if it was already checked once. Until a doorbell timeout happens it's an extra MMIO cycle that can saved for a "what happened?" check after a timeout.
Quoted 4 lines
Quoted 1 line
Well I suspect we're just rearranging the deck chairs on the Titanic now, but...
Not so much, just trying to get this driver in line with other error handling designs.
Okay. I shall remove it then.
Quoted 11 lines
Quoted 11 lines
I see doorbell timeouts as disconnected from whether or not the mailbox interface is ready. If they were the same, we wouldn't need both bits and we could just wait extra long for the doorbell when probing.
In other words, I expect if the interface goes unready, doorbell timeout will occur, but I don't think we should assume if doorbell timeout occurs, the interface is no longer ready. I don't purport to know why a doorbell timeout might occur while the interface remains available (likely a firmware bug, I presume).
Quoted 7 lines
Quoted 2 lines
It does seem interesting to check if the interface is no longer ready on timeout though.
So I'm just modeling this off of NVME error handling where there is a Controller Fatal Status bit that could be checked every transaction, but instead the driver waits until a command timeout to collect if the device went fatal / not-ready.
No error interrupts?
PATCH 6/23 cxl/pci: Don't check media status for mbox access
In reply to Ben WidawskyMedia status is necessary for using HDM contained in a CXL device but is not needed for mailbox accesses. Therefore remove this check. It will be necessary to have this check (in a different place) when enabling HDM.
---
This patch did not exist in RFCv2
--- drivers/cxl/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/pci.c +1 -1
@@ -230,7 +230,7 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* but it's possible early devices implemented this before the ECN. */ md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); -if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) { +if (!(md_status &CXLMDEV_MBOX_IF_READY )) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" ); rc = -EBUSY ; goto out;
Signature · 1 line
2.34.0
Re: [PATCH 06/23] cxl/pci: Don't check media status for mbox access
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:33 -0800 Ben Widawsky <[email protected]> wrote:
Media status is necessary for using HDM contained in a CXL device but is not needed for mailbox accesses. Therefore remove this check. It will be necessary to have this check (in a different place) when enabling HDM.
Signed-off-by Ben Widawsky <[email protected]>
One could perhaps argue this was a bug, but it was hopefully harmless on real devices. Out of curiosity, did you find a clear definition of what 'User data' includes? Obviously includes any data in the HDM region, but what about region meta data?
Quoted 19 lines
---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/pci.c +1
-1@@ -230,7 +230,7 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* but it's possible early devices implemented this before the ECN. */ md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); -if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) { +if (!(md_status &CXLMDEV_MBOX_IF_READY )) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" ); rc = -EBUSY ; goto out;
Re: [PATCH 06/23] cxl/pci: Don't check media status for mbox access
In reply to Ben WidawskyQuoted 17 lines · Ben Widawsky
Media status is necessary for using HDM contained in a CXL device but is not needed for mailbox accesses. Therefore remove this check. It will be necessary to have this check (in a different place) when enabling HDM.
Signed-off-by Ben Widawsky <[email protected]>---
This patch did not exist in RFCv2--- drivers/cxl/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds) · line 236
Quoted 8 lines
drivers/cxl/pci.c +1
-1@@ -230,7 +230,7 @@ static int cxl_pci_mbox_get(struct cxl_dev_state *cxlds)* but it's possible early devices implemented this before the ECN. */ md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); -if (!(md_status &CXLMDEV_MBOX_IF_READY &&CXLMDEV_READY (md_status))) { +if (!(md_status &CXLMDEV_MBOX_IF_READY )) {dev_err (dev,"mbox: reported doorbell ready, but not mbox ready\n" ); rc = -EBUSY ; goto out;
Per comment on last patch I think this whole 'if' block can go. "Ready" need only be checked once at the beginning of time and then only as a forensic step after a future failure.
PATCH 7/23 cxl/pci: Add new DVSEC definitions
In reply to Ben WidawskyWhile the new definitions are yet necessary at this point, they are introduced at this point to help solidify the newly minted schema for naming registers.
---
This was split from
https://lore.kernel.org/linux-cxl/[email protected]/T/#u
per Dan's request.
--- drivers/cxl/pci.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
drivers/cxl/pci.h +15 -0
@@ -16,6 +16,21 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0 +/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ +#define CXL_DVSEC_FUNCTION_MAP 2 + +/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */ +#define CXL_DVSEC_PORT_EXTENSIONS 3 + +/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */ +#define CXL_DVSEC_PORT_GPF 4 + +/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */ +#define CXL_DVSEC_DEVICE_GPF 5 + +/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */ +#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7 +/* CXL 2.0 8.1.9: Register Locator DVSEC */ #define CXL_DVSEC_REG_LOCATOR 8#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC
Signature · 1 line
2.34.0
Re: [PATCH 07/23] cxl/pci: Add new DVSEC definitions
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:34 -0800 Ben Widawsky <[email protected]> wrote:
While the new definitions are yet necessary at this point, they are introduced at this point to help solidify the newly minted schema for naming registers.
Signed-off-by Ben Widawsky <[email protected]>
Quoted 35 lines
---
This was split from
https://lore.kernel.org/linux-cxl/[email protected]/T/#u
per Dan's request.--- drivers/cxl/pci.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)drivers/cxl/pci.h +15
-0@@ -16,6 +16,21 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0 +/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ +#define CXL_DVSEC_FUNCTION_MAP 2 + +/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */ +#define CXL_DVSEC_PORT_EXTENSIONS 3 + +/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */ +#define CXL_DVSEC_PORT_GPF 4 + +/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */ +#define CXL_DVSEC_DEVICE_GPF 5 + +/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */ +#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7 +/* CXL 2.0 8.1.9: Register Locator DVSEC */ #define CXL_DVSEC_REG_LOCATOR 8#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC
Re: [PATCH 07/23] cxl/pci: Add new DVSEC definitions
Quoted 10 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:34 -0800 Ben Widawsky <[email protected]> wrote:
While the new definitions are yet necessary at this point, they are introduced at this point to help solidify the newly minted schema for naming registers.
Signed-off-by Ben Widawsky <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Thanks. I realized on re-reading this I didn't like the commit message. I reworded to this:
While the new definitions are not yet necessary at this point, they are introduced to help solidify the newly minted schema for naming registers.
Please let me know if you'd like me to drop your reviewed-by tag.
Quoted 37 lines
Quoted 36 lines
---
This was split from
https://lore.kernel.org/linux-cxl/[email protected]/T/#u
per Dan's request.--- drivers/cxl/pci.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)drivers/cxl/pci.h +15
-0@@ -16,6 +16,21 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0 +/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ +#define CXL_DVSEC_FUNCTION_MAP 2 + +/* CXL 2.0 8.1.5: CXL 2.0 Extensions DVSEC for Ports */ +#define CXL_DVSEC_PORT_EXTENSIONS 3 + +/* CXL 2.0 8.1.6: GPF DVSEC for CXL Port */ +#define CXL_DVSEC_PORT_GPF 4 + +/* CXL 2.0 8.1.7: GPF DVSEC for CXL Device */ +#define CXL_DVSEC_DEVICE_GPF 5 + +/* CXL 2.0 8.1.8: PCIe DVSEC for Flex Bus Port */ +#define CXL_DVSEC_PCIE_FLEXBUS_PORT 7 +/* CXL 2.0 8.1.9: Register Locator DVSEC */ #define CXL_DVSEC_REG_LOCATOR 8#define CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET 0xC
Dan Williams
Reviewed-byRe: [PATCH 07/23] cxl/pci: Add new DVSEC definitions
Quoted 13 lines · Ben Widawsky
Quoted 10 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:34 -0800 Ben Widawsky <[email protected]> wrote:
While the new definitions are yet necessary at this point, they are introduced at this point to help solidify the newly minted schema for naming registers.
Signed-off-by Ben Widawsky <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>
Quoted 8 lines
Thanks. I realized on re-reading this I didn't like the commit message. I reworded to this:
While the new definitions are not yet necessary at this point, they are introduced to help solidify the newly minted schema for naming registers.
Please let me know if you'd like me to drop your reviewed-by tag.
The typical changelog template for patches like this is:
"In preparation for adding features X, Y, and Z, add definitions for A, B, and C."
Otherwise, patch looks good.
PATCH 8/23 cxl/acpi: Map component registers for Root Ports
In reply to Ben WidawskyThis implements the TODO in cxl_acpi for mapping component registers. cxl_acpi becomes the second consumer of CXL register block enumeration (cxl_pci being the first). Moving the functionality to cxl_core allows both of these drivers to use the functionality. Equally importantly it allows cxl_core to use the functionality in the future.
CXL 2.0 root ports are similar to CXL 2.0 Downstream Ports with the main distinction being they're a part of the CXL 2.0 host bridge. While mapping their component registers is not immediately useful for the CXL drivers, the movement of register block enumeration into core is a vital step towards HDM decoder programming.
---
Changes since RFCv2:
- Squash commits together (Dan)
- Reword commit message to account for above.
--- drivers/cxl/acpi.c | 10 ++++++-- drivers/cxl/core/regs.c | 54 +++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/pci.c | 52 --------------------------------------- drivers/cxl/pci.h | 4 +++ 5 files changed, 70 insertions(+), 54 deletions(-)
drivers/cxl/acpi.c +8 -2
@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include "cxl.h" +#include "pci.h" /* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */ #define CFMWS_INTERLEAVE_WAYS (x) (1 << (x)->interleave_ways)
@@ -134,11 +135,13 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, __mock int match_add_root_ports(struct pci_dev *pdev, void *data) { +resource_size_t creg =CXL_RESOURCE_NONE ;struct cxl_walk_context *ctx = data;struct pci_bus *root_bus = ctx->root;struct cxl_port *port = ctx->port;int type =pci_pcie_type (pdev);struct device *dev = ctx->dev; +struct cxl_register_map map;u32 lnkcap, port_num;int rc;
@@ -152,9 +155,12 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) &lnkcap) != PCIBIOS_SUCCESSFUL)return 0 ; -/* TODO walk DVSEC to find component register base */ + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (!rc) + creg =cxl_reg_block (pdev, &map); + port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg);if (rc) { ctx->error = rc;return rc;
drivers/cxl/core/regs.c +54 -0
63 lines not on this page · raw message
drivers/cxl/cxl.h +4 -0
@@ -145,6 +145,10 @@ int cxl_map_device_regs(struct pci_dev *pdev,struct cxl_device_regs *regs, struct cxl_register_map *map); +enum cxl_regloc_type ; +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map); +#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20
drivers/cxl/pci.c +0 -52
58 lines not on this page · raw message
drivers/cxl/pci.h +4 -0
@@ -47,4 +47,8 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; +#define cxl_reg_block (pdev, map) \ + ((resource_size_t)(pci_resource_start(pdev, (map)->barno) + \ + (map)->block_offset)) +#endif /* __CXL_PCI_H__ */
Signature · 1 line
2.34.0
Re: [PATCH 08/23] cxl/acpi: Map component registers for Root Ports
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:35 -0800 Ben Widawsky <[email protected]> wrote:
This implements the TODO in
cxl_acpifor mapping component registers.cxl_acpibecomes the second consumer of CXL register block enumeration (cxl_pcibeing the first). Moving the functionality tocxl_coreallows both of these drivers to use the functionality. Equally importantly it allowscxl_coreto use the functionality in the future.
Quoted 7 lines
CXL 2.0 root ports are similar to CXL 2.0 Downstream Ports with the main distinction being they're a part of the CXL 2.0 host bridge. While mapping their component registers is not immediately useful for the CXL drivers, the movement of register block enumeration into core is a vital step towards HDM decoder programming.
Signed-off-by Ben Widawsky <[email protected]>
A few minor comments below.
Jonathan
Quoted 40 lines
---
Changes since RFCv2:
- Squash commits together (Dan)
- Reword commit message to account for above.--- drivers/cxl/acpi.c | 10 ++++++-- drivers/cxl/core/regs.c | 54 +++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/pci.c | 52 --------------------------------------- drivers/cxl/pci.h | 4 +++ 5 files changed, 70 insertions(+), 54 deletions(-)drivers/cxl/acpi.c +3
-0@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include "cxl.h" +#include "pci.h" /* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */ #define CFMWS_INTERLEAVE_WAYS (x) (1 << (x)->interleave_ways)@@ -134,11 +135,13 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, __mock int match_add_root_ports(struct pci_dev *pdev, void *data) { +resource_size_t creg =CXL_RESOURCE_NONE ;struct cxl_walk_context *ctx = data;struct pci_bus *root_bus = ctx->root;struct cxl_port *port = ctx->port;int type =pci_pcie_type (pdev);struct device *dev = ctx->dev; +struct cxl_register_map map;u32 lnkcap, port_num;int rc;
drivers/cxl/acpi.c __mock int match_add_root_ports(struct pci_dev *pdev, void *data) · line 158
Quoted 5 lines
drivers/cxl/acpi.c +1
-1@@ -152,9 +155,12 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) &lnkcap) != PCIBIOS_SUCCESSFUL)return 0 ; -/* TODO walk DVSEC to find component register base */ + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map);
Perhaps a comment to explain why this is optional?
Quoted 34 lines
drivers/cxl/acpi.c +4
-1+if (!rc) + creg =cxl_reg_block (pdev, &map); + port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg);if (rc) { ctx->error = rc;return rc;drivers/cxl/core/regs.c +10
-0@@ -5,6 +5,7 @@#include <linux/slab.h> #include <linux/pci.h> #include <cxlmem.h> +#include <pci.h> /** * DOC: cxl registers @@ -247,3 +248,56 @@ int cxl_map_device_regs(struct pci_dev *pdev,return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_map_device_regs,CXL ); + +static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, +struct cxl_register_map *map) +{ + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); +}
drivers/cxl/core/regs.c int cxl_map_device_regs(struct pci_dev *pdev, · line 269
Quoted 10 lines
drivers/cxl/core/regs.c +10
-0+ +/** +* cxl_find_regblock() - Locate register blocks by type +* @pdev: The CXL PCI device to enumerate. +* @type: Register Block Indicator id +* @map: Enumeration output, clobbered on error +* +* Return: 0 if register block enumerated, negative error code otherwise +* +* A CXL DVSEC may additional point one or more register blocks, search
Why additional? I'm not sure what this means.
point to one or more additional register blocks perhaps?
Quoted 125 lines
drivers/cxl/core/regs.c +34
-0+* for them by @type. +*/ +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +u32 regloc_size, regblocks; +int regloc, i; + + regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_REG_LOCATOR ); +if (!regloc) +return -ENXIO ; + +pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); + regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); + + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; + +for (i =0 ; i < regblocks; i++, regloc +=8 ) { +u32 reg_lo, reg_hi; + +pci_read_config_dword (pdev, regloc, ®_lo); +pci_read_config_dword (pdev, regloc +4 , ®_hi); + +cxl_decode_regblock (reg_lo, reg_hi, map); + +if (map->reg_type == type) +return 0 ; + } + +return -ENODEV ; +} +EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );drivers/cxl/cxl.h +4
-0@@ -145,6 +145,10 @@ int cxl_map_device_regs(struct pci_dev *pdev,struct cxl_device_regs *regs, struct cxl_register_map *map); +enum cxl_regloc_type ; +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map); +#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20drivers/cxl/pci.c +0
-52@@ -433,58 +433,6 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mareturn 0 ; } -static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, -struct cxl_register_map *map) -{ - map->block_offset = ((u64 )reg_hi <<32 ) | - (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); - map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); -} - -/** -* cxl_find_regblock() - Locate register blocks by type -* @pdev: The CXL PCI device to enumerate. -* @type: Register Block Indicator id -* @map: Enumeration output, clobbered on error -* -* Return: 0 if register block enumerated, negative error code otherwise -* -* A CXL DVSEC may point to one or more register blocks, search for them -* by @type. -*/ -static int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, -struct cxl_register_map *map) -{ -u32 regloc_size, regblocks; -int regloc, i; - - regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -CXL_DVSEC_REG_LOCATOR ); -if (!regloc) -return -ENXIO ; - -pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); - regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - - regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; - regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; - -for (i =0 ; i < regblocks; i++, regloc +=8 ) { -u32 reg_lo, reg_hi; - -pci_read_config_dword (pdev, regloc, ®_lo); -pci_read_config_dword (pdev, regloc +4 , ®_hi); - -cxl_decode_regblock (reg_lo, reg_hi, map); - -if (map->reg_type == type) -return 0 ; - } - -return -ENODEV ; -} -static int cxl_setup_regs (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map) {drivers/cxl/pci.h +4
-0@@ -47,4 +47,8 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; +#define cxl_reg_block (pdev, map) \ + ((resource_size_t)(pci_resource_start(pdev, (map)->barno) + \ + (map)->block_offset)) +#endif /* __CXL_PCI_H__ */
Re: [PATCH 08/23] cxl/acpi: Map component registers for Root Ports
Quoted 60 lines · Jonathan Cameron
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:35 -0800 Ben Widawsky <[email protected]> wrote:
This implements the TODO in
cxl_acpifor mapping component registers.cxl_acpibecomes the second consumer of CXL register block enumeration (cxl_pcibeing the first). Moving the functionality tocxl_coreallows both of these drivers to use the functionality. Equally importantly it allowscxl_coreto use the functionality in the future.Quoted 7 lines
CXL 2.0 root ports are similar to CXL 2.0 Downstream Ports with the main distinction being they're a part of the CXL 2.0 host bridge. While mapping their component registers is not immediately useful for the CXL drivers, the movement of register block enumeration into core is a vital step towards HDM decoder programming.
Signed-off-by Ben Widawsky <[email protected]>A few minor comments below.
Jonathan
Quoted 39 lines
---
Changes since RFCv2:
- Squash commits together (Dan)
- Reword commit message to account for above.--- drivers/cxl/acpi.c | 10 ++++++-- drivers/cxl/core/regs.c | 54 +++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/pci.c | 52 --------------------------------------- drivers/cxl/pci.h | 4 +++ 5 files changed, 70 insertions(+), 54 deletions(-)drivers/cxl/acpi.c +3
-0@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include "cxl.h" +#include "pci.h" /* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */ #define CFMWS_INTERLEAVE_WAYS (x) (1 << (x)->interleave_ways)@@ -134,11 +135,13 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, __mock int match_add_root_ports(struct pci_dev *pdev, void *data) { +resource_size_t creg =CXL_RESOURCE_NONE ;struct cxl_walk_context *ctx = data;struct pci_bus *root_bus = ctx->root;struct cxl_port *port = ctx->port;int type =pci_pcie_type (pdev);struct device *dev = ctx->dev; +struct cxl_register_map map;u32 lnkcap, port_num;int rc;
Quoted 9 lines
drivers/cxl/acpi.c __mock int match_add_root_ports(struct pci_dev *pdev, void *data) · line 158
Quoted 6 lines
drivers/cxl/acpi.c +1
-1@@ -152,9 +155,12 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) &lnkcap) != PCIBIOS_SUCCESSFUL)return 0 ; -/* TODO walk DVSEC to find component register base */ + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map);Perhaps a comment to explain why this is optional?
Got it. I've also added a dev_info if the regblock isn't found because at some point in the future it might be confusing should we ever want to use those registers.
Quoted 47 lines
Quoted 34 lines
drivers/cxl/acpi.c +4
-1+if (!rc) + creg =cxl_reg_block (pdev, &map); + port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg);if (rc) { ctx->error = rc;return rc;drivers/cxl/core/regs.c +10
-0@@ -5,6 +5,7 @@#include <linux/slab.h> #include <linux/pci.h> #include <cxlmem.h> +#include <pci.h> /** * DOC: cxl registers @@ -247,3 +248,56 @@ int cxl_map_device_regs(struct pci_dev *pdev,return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_map_device_regs,CXL ); + +static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, +struct cxl_register_map *map) +{ + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); +}drivers/cxl/core/regs.c int cxl_map_device_regs(struct pci_dev *pdev, · line 269
Quoted 10 lines
drivers/cxl/core/regs.c +10
-0+ +/** +* cxl_find_regblock() - Locate register blocks by type +* @pdev: The CXL PCI device to enumerate. +* @type: Register Block Indicator id +* @map: Enumeration output, clobbered on error +* +* Return: 0 if register block enumerated, negative error code otherwise +* +* A CXL DVSEC may additional point one or more register blocks, search Why additional? I'm not sure what this means.
Quoted 2 lines
point to one or more additional register blocks perhaps?
Was a typo from how I rebased. Good catch.
Quoted 126 lines
Quoted 125 lines
drivers/cxl/core/regs.c +34
-0+* for them by @type. +*/ +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +u32 regloc_size, regblocks; +int regloc, i; + + regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_REG_LOCATOR ); +if (!regloc) +return -ENXIO ; + +pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); + regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); + + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; + +for (i =0 ; i < regblocks; i++, regloc +=8 ) { +u32 reg_lo, reg_hi; + +pci_read_config_dword (pdev, regloc, ®_lo); +pci_read_config_dword (pdev, regloc +4 , ®_hi); + +cxl_decode_regblock (reg_lo, reg_hi, map); + +if (map->reg_type == type) +return 0 ; + } + +return -ENODEV ; +} +EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );drivers/cxl/cxl.h +4
-0@@ -145,6 +145,10 @@ int cxl_map_device_regs(struct pci_dev *pdev,struct cxl_device_regs *regs, struct cxl_register_map *map); +enum cxl_regloc_type ; +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map); +#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20drivers/cxl/pci.c +0
-52@@ -433,58 +433,6 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mareturn 0 ; } -static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, -struct cxl_register_map *map) -{ - map->block_offset = ((u64 )reg_hi <<32 ) | - (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); - map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); -} - -/** -* cxl_find_regblock() - Locate register blocks by type -* @pdev: The CXL PCI device to enumerate. -* @type: Register Block Indicator id -* @map: Enumeration output, clobbered on error -* -* Return: 0 if register block enumerated, negative error code otherwise -* -* A CXL DVSEC may point to one or more register blocks, search for them -* by @type. -*/ -static int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, -struct cxl_register_map *map) -{ -u32 regloc_size, regblocks; -int regloc, i; - - regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -CXL_DVSEC_REG_LOCATOR ); -if (!regloc) -return -ENXIO ; - -pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); - regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - - regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; - regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; - -for (i =0 ; i < regblocks; i++, regloc +=8 ) { -u32 reg_lo, reg_hi; - -pci_read_config_dword (pdev, regloc, ®_lo); -pci_read_config_dword (pdev, regloc +4 , ®_hi); - -cxl_decode_regblock (reg_lo, reg_hi, map); - -if (map->reg_type == type) -return 0 ; - } - -return -ENODEV ; -} -static int cxl_setup_regs (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map) {drivers/cxl/pci.h +4
-0@@ -47,4 +47,8 @@ enum cxl_regloc_type {CXL_REGLOC_RBI_TYPES }; +#define cxl_reg_block (pdev, map) \ + ((resource_size_t)(pci_resource_start(pdev, (map)->barno) + \ + (map)->block_offset)) +#endif /* __CXL_PCI_H__ */
Re: [PATCH 08/23] cxl/acpi: Map component registers for Root Ports
In reply to Ben WidawskyQuoted 54 lines · Ben Widawsky
This implements the TODO in
cxl_acpifor mapping component registers.cxl_acpibecomes the second consumer of CXL register block enumeration (cxl_pcibeing the first). Moving the functionality tocxl_coreallows both of these drivers to use the functionality. Equally importantly it allowscxl_coreto use the functionality in the future.CXL 2.0 root ports are similar to CXL 2.0 Downstream Ports with the main distinction being they're a part of the CXL 2.0 host bridge. While mapping their component registers is not immediately useful for the CXL drivers, the movement of register block enumeration into core is a vital step towards HDM decoder programming.
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Squash commits together (Dan)
- Reword commit message to account for above.--- drivers/cxl/acpi.c | 10 ++++++-- drivers/cxl/core/regs.c | 54 +++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/pci.c | 52 --------------------------------------- drivers/cxl/pci.h | 4 +++ 5 files changed, 70 insertions(+), 54 deletions(-)drivers/cxl/acpi.c +3
-0@@ -7,6 +7,7 @@#include <linux/acpi.h> #include <linux/pci.h> #include "cxl.h" +#include "pci.h" /* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */ #define CFMWS_INTERLEAVE_WAYS (x) (1 << (x)->interleave_ways)@@ -134,11 +135,13 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, __mock int match_add_root_ports(struct pci_dev *pdev, void *data) { +resource_size_t creg =CXL_RESOURCE_NONE ;struct cxl_walk_context *ctx = data;struct pci_bus *root_bus = ctx->root;struct cxl_port *port = ctx->port;int type =pci_pcie_type (pdev);struct device *dev = ctx->dev; +struct cxl_register_map map;u32 lnkcap, port_num;int rc;
drivers/cxl/acpi.c __mock int match_add_root_ports(struct pci_dev *pdev, void *data) · line 160
Quoted 7 lines
drivers/cxl/acpi.c +3
-1@@ -152,9 +155,12 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) &lnkcap) != PCIBIOS_SUCCESSFUL)return 0 ; -/* TODO walk DVSEC to find component register base */ + rc =cxl_find_regblock (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (!rc) + creg =cxl_reg_block (pdev, &map);
A couple comments: the difference between cxl_find_regblock() and cxl_reg_block() is not obvious from the names. Setting aside why one is regblock and the other is reg_block I would expect a name like cxl_regmap_to_base() is easier to read.
It occurs to me that if cxl_find_regblock() failures are optional it would be nice if cxl_regmap_to_base() returns CXL_RESOURCE_NONE if the map is not populated. Then this can unconditionally call cxl_regmap_to_base().
Quoted 159 lines
drivers/cxl/acpi.c +2
-1+ port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg);if (rc) { ctx->error = rc;return rc;drivers/cxl/core/regs.c +54
-0@@ -5,6 +5,7 @@#include <linux/slab.h> #include <linux/pci.h> #include <cxlmem.h> +#include <pci.h> /** * DOC: cxl registers @@ -247,3 +248,56 @@ int cxl_map_device_regs(struct pci_dev *pdev,return 0 ; }EXPORT_SYMBOL_NS_GPL (cxl_map_device_regs,CXL ); + +static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, +struct cxl_register_map *map) +{ + map->block_offset = ((u64 )reg_hi <<32 ) | + (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); + map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); + map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); +} + +/** +* cxl_find_regblock() - Locate register blocks by type +* @pdev: The CXL PCI device to enumerate. +* @type: Register Block Indicator id +* @map: Enumeration output, clobbered on error +* +* Return: 0 if register block enumerated, negative error code otherwise +* +* A CXL DVSEC may additional point one or more register blocks, search +* for them by @type. +*/ +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map) +{ +u32 regloc_size, regblocks; +int regloc, i; + + regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_REG_LOCATOR ); +if (!regloc) +return -ENXIO ; + +pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); + regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); + + regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; + regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; + +for (i =0 ; i < regblocks; i++, regloc +=8 ) { +u32 reg_lo, reg_hi; + +pci_read_config_dword (pdev, regloc, ®_lo); +pci_read_config_dword (pdev, regloc +4 , ®_hi); + +cxl_decode_regblock (reg_lo, reg_hi, map); + +if (map->reg_type == type) +return 0 ; + } + +return -ENODEV ; +} +EXPORT_SYMBOL_NS_GPL (cxl_find_regblock,CXL );drivers/cxl/cxl.h +4
-0@@ -145,6 +145,10 @@ int cxl_map_device_regs(struct pci_dev *pdev,struct cxl_device_regs *regs, struct cxl_register_map *map); +enum cxl_regloc_type ; +int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, +struct cxl_register_map *map); +#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20drivers/cxl/pci.c +0
-52@@ -433,58 +433,6 @@ static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *mareturn 0 ; } -static void cxl_decode_regblock (u32 reg_lo,u32 reg_hi, -struct cxl_register_map *map) -{ - map->block_offset = ((u64 )reg_hi <<32 ) | - (reg_lo &CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK ); - map->barno =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BIR_MASK , reg_lo); - map->reg_type =FIELD_GET (CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK , reg_lo); -} - -/** -* cxl_find_regblock() - Locate register blocks by type -* @pdev: The CXL PCI device to enumerate. -* @type: Register Block Indicator id -* @map: Enumeration output, clobbered on error -* -* Return: 0 if register block enumerated, negative error code otherwise -* -* A CXL DVSEC may point to one or more register blocks, search for them -* by @type. -*/ -static int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type, -struct cxl_register_map *map) -{ -u32 regloc_size, regblocks; -int regloc, i; - - regloc =pci_find_dvsec_capability (pdev,PCI_DVSEC_VENDOR_ID_CXL , -CXL_DVSEC_REG_LOCATOR ); -if (!regloc) -return -ENXIO ; - -pci_read_config_dword (pdev, regloc +PCI_DVSEC_HEADER1 , ®loc_size); - regloc_size =FIELD_GET (PCI_DVSEC_HEADER1_LENGTH_MASK , regloc_size); - - regloc +=CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ; - regblocks = (regloc_size -CXL_DVSEC_REG_LOCATOR_BLOCK1_OFFSET ) /8 ; - -for (i =0 ; i < regblocks; i++, regloc +=8 ) { -u32 reg_lo, reg_hi; - -pci_read_config_dword (pdev, regloc, ®_lo); -pci_read_config_dword (pdev, regloc +4 , ®_hi); - -cxl_decode_regblock (reg_lo, reg_hi, map); - -if (map->reg_type == type) -return 0 ; - } - -return -ENODEV ; -} -static int cxl_setup_regs (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map) {drivers/cxl/pci.h +0
-0
drivers/cxl/pci.h enum cxl_regloc_type { · line 53
Quoted 7 lines
drivers/cxl/pci.h +4
-0@@ -47,4 +47,8 @@ enum cxl_regloc_type { CXL_REGLOC_RBI_TYPES }; +#define cxl_reg_block (pdev, map) \ + ((resource_size_t)(pci_resource_start(pdev, (map)->barno) + \ + (map)->block_offset)) +
I see no reason for this to be macro. It's also a bug timebomb if someone in the future does something like:
cxl_reg_block(pdev, map++);
...because the macro references its arguments more than once.
PATCH 9/23 cxl: Introduce module_cxl_driver
In reply to Ben WidawskyMany CXL drivers simply want to register and unregister themselves. module_driver already supported this. A simple wrapper around that reduces a decent amount of boilerplate in upcoming patches.
--- drivers/cxl/cxl.h | 3 +++ 1 file changed, 3 insertions(+)
drivers/cxl/cxl.h +3 -0
@@ -308,6 +308,9 @@ int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,#define cxl_driver_register (x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)void cxl_driver_unregister (struct cxl_driver *cxl_drv); +#define module_cxl_driver (__cxl_driver) \ + module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister) +#define CXL_DEVICE_NVDIMM_BRIDGE 1#define CXL_DEVICE_NVDIMM 2
Signature · 1 line
2.34.0
Re: [PATCH 09/23] cxl: Introduce module_cxl_driver
Quoted 6 lines · On Fri, 19 Nov 2021 16:02:36 -0800 Ben Widawsky <[email protected]> wrote:
Many CXL drivers simply want to register and unregister themselves.
module_driveralready supported this. A simple wrapper around that reduces a decent amount of boilerplate in upcoming patches.Suggested-by Dan Williams <[email protected]>Signed-off-by Ben Widawsky <[email protected]>
Makes sense although, when compared with using the module_driver macro directly it's a fairly minor reduction in boilerplate.
Quoted 18 lines
--- drivers/cxl/cxl.h | 3 +++ 1 file changed, 3 insertions(+)drivers/cxl/cxl.h +3
-0@@ -308,6 +308,9 @@ int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,#define cxl_driver_register (x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)void cxl_driver_unregister (struct cxl_driver *cxl_drv); +#define module_cxl_driver (__cxl_driver) \ + module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister) +#define CXL_DEVICE_NVDIMM_BRIDGE 1#define CXL_DEVICE_NVDIMM 2
Re: [PATCH 09/23] cxl: Introduce module_cxl_driver
In reply to Ben WidawskyQuoted 2 lines · Ben Widawsky
Many CXL drivers simply want to register and unregister themselves.
s/Many/Some/?
Quoted 2 lines
module_driveralready supported this. A simple wrapper around that reduces a decent amount of boilerplate in upcoming patches.
Looks good.
PATCH 10/23 cxl/core: Convert decoder range to resource
In reply to Ben WidawskyCXL decoders manage address ranges in a hierarchical fashion whereby a leaf is a unique subregion of its parent decoder (midlevel or root). It therefore makes sense to use the resource API for handling this.
---
Changes since RFCv2
- Switch to DEFINE_RES_MEM from NAMED variant (Dan)
- Differentiate CFMWS resources and other decoder resources (Ben)
- Make decoder resources be range, nor resource (Dan)
- Set decoder name in cxl_decoder_add() (Dan)
--- drivers/cxl/acpi.c | 16 ++++++---------- drivers/cxl/core/bus.c | 19 +++++++++++++++++-- drivers/cxl/cxl.h | 8 ++++++-- 3 files changed, 29 insertions(+), 14 deletions(-)
drivers/cxl/acpi.c +6 -10
@@ -108,10 +108,8 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, cxld->flags =cfmws_to_decoder_flags (cfmws->restrictions); cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ){ - .start = cfmws->base_hpa, - .end = cfmws->base_hpa + cfmws->window_size -1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (cfmws->base_hpa, + cfmws->window_size); cxld->interleave_ways =CFMWS_INTERLEAVE_WAYS (cfmws); cxld->interleave_granularity =CFMWS_INTERLEAVE_GRANULARITY (cfmws);
@@ -127,8 +125,9 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg,return 0 ; }dev_dbg (dev,"add: %s node: %d range %#llx-%#llx\n" , -dev_name (&cxld->dev),phys_to_target_node (cxld->range.start), - cfmws->base_hpa, cfmws->base_hpa + cfmws->window_size -1 ); +dev_name (&cxld->dev), +phys_to_target_node (cxld->platform_res.start), cfmws->base_hpa, + cfmws->base_hpa + cfmws->window_size -1 );return 0 ; }
@@ -267,10 +266,7 @@ static int add_host_bridge_uport(struct device *match, void *arg) cxld->interleave_ways =1 ; cxld->interleave_granularity =PAGE_SIZE ; cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ) { - .start =0 , - .end =-1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 );device_lock (&port->dev); dport =list_first_entry (&port->dports,typeof (*dport), list);
drivers/cxl/core/bus.c +17 -2
@@ -46,8 +46,14 @@ static ssize_t start_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 start =0 ; -return sysfs_emit (buf,"%#llx\n" , cxld->range.start); +if (is_root_decoder (dev)) + start = cxld->platform_res.start; +else + start = cxld->decoder_range.start; + +return sysfs_emit (buf,"%#llx\n" , start); }static DEVICE_ATTR_RO (start);
@@ -55,8 +61,14 @@ static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 size =0 ; -return sysfs_emit (buf,"%#llx\n" ,range_len (&cxld->range)); +if (is_root_decoder (dev)) + size =resource_size (&cxld->platform_res); +else + size =range_len (&cxld->decoder_range); + +return sysfs_emit (buf,"%#llx\n" , size); }static DEVICE_ATTR_RO (size);
@@ -548,6 +560,9 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (rc)return rc; +if (is_root_decoder (dev)) + cxld->platform_res.name =dev_name (dev); +return device_add (dev); }EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );
drivers/cxl/cxl.h +6 -2
@@ -179,7 +179,8 @@ enum cxl_decoder_type {* struct cxl_decoder - CXL address range decode configuration * @dev: this decoder's device * @id: kernel device name id -* @range: address range considered by this decoder +* @platform_res: address space resources considered by root decoder +* @decoder_range: address space resources considered by midlevel decoder * @interleave_ways: number of cxl_dports in this decode * @interleave_granularity: data stride per dport * @target_type: accelerator vs expander (type2 vs type3) selector
@@ -190,7 +191,10 @@ enum cxl_decoder_type {struct cxl_decoder {struct device dev;int id; -struct range range; +union { +struct resource platform_res; +struct range decoder_range; + };int interleave_ways;int interleave_granularity;enum cxl_decoder_type target_type;
Signature · 1 line
2.34.0
Re: [PATCH 10/23] cxl/core: Convert decoder range to resource
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:37 -0800 Ben Widawsky <[email protected]> wrote:
CXL decoders manage address ranges in a hierarchical fashion whereby a leaf is a unique subregion of its parent decoder (midlevel or root). It therefore makes sense to use the resource API for handling this.
Signed-off-by Ben Widawsky <[email protected]>
LGTM
Quoted 126 lines
---
Changes since RFCv2
- Switch toDEFINE_RES_MEMfrom NAMED variant (Dan)
- Differentiate CFMWS resources and other decoder resources (Ben)
- Make decoder resources be range, nor resource (Dan)
- Set decoder name incxl_decoder_add()(Dan)--- drivers/cxl/acpi.c | 16 ++++++---------- drivers/cxl/core/bus.c | 19 +++++++++++++++++-- drivers/cxl/cxl.h | 8 ++++++-- 3 files changed, 29 insertions(+), 14 deletions(-)drivers/cxl/acpi.c +6
-10@@ -108,10 +108,8 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, cxld->flags =cfmws_to_decoder_flags (cfmws->restrictions); cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ){ - .start = cfmws->base_hpa, - .end = cfmws->base_hpa + cfmws->window_size -1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (cfmws->base_hpa, + cfmws->window_size); cxld->interleave_ways =CFMWS_INTERLEAVE_WAYS (cfmws); cxld->interleave_granularity =CFMWS_INTERLEAVE_GRANULARITY (cfmws);@@ -127,8 +125,9 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg,return 0 ; }dev_dbg (dev,"add: %s node: %d range %#llx-%#llx\n" , -dev_name (&cxld->dev),phys_to_target_node (cxld->range.start), - cfmws->base_hpa, cfmws->base_hpa + cfmws->window_size -1 ); +dev_name (&cxld->dev), +phys_to_target_node (cxld->platform_res.start), cfmws->base_hpa, + cfmws->base_hpa + cfmws->window_size -1 );return 0 ; }@@ -267,10 +266,7 @@ static int add_host_bridge_uport(struct device *match, void *arg) cxld->interleave_ways =1 ; cxld->interleave_granularity =PAGE_SIZE ; cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ) { - .start =0 , - .end =-1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 );device_lock (&port->dev); dport =list_first_entry (&port->dports,typeof (*dport), list);drivers/cxl/core/bus.c +17
-2@@ -46,8 +46,14 @@ static ssize_t start_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 start =0 ; -return sysfs_emit (buf,"%#llx\n" , cxld->range.start); +if (is_root_decoder (dev)) + start = cxld->platform_res.start; +else + start = cxld->decoder_range.start; + +return sysfs_emit (buf,"%#llx\n" , start); }static DEVICE_ATTR_RO (start);@@ -55,8 +61,14 @@ static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 size =0 ; -return sysfs_emit (buf,"%#llx\n" ,range_len (&cxld->range)); +if (is_root_decoder (dev)) + size =resource_size (&cxld->platform_res); +else + size =range_len (&cxld->decoder_range); + +return sysfs_emit (buf,"%#llx\n" , size); }static DEVICE_ATTR_RO (size);@@ -548,6 +560,9 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (rc)return rc; +if (is_root_decoder (dev)) + cxld->platform_res.name =dev_name (dev); +return device_add (dev); }EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );drivers/cxl/cxl.h +6
-2@@ -179,7 +179,8 @@ enum cxl_decoder_type {* struct cxl_decoder - CXL address range decode configuration * @dev: this decoder's device * @id: kernel device name id -* @range: address range considered by this decoder +* @platform_res: address space resources considered by root decoder +* @decoder_range: address space resources considered by midlevel decoder * @interleave_ways: number of cxl_dports in this decode * @interleave_granularity: data stride per dport * @target_type: accelerator vs expander (type2 vs type3) selector @@ -190,7 +191,10 @@ enum cxl_decoder_type {struct cxl_decoder {struct device dev;int id; -struct range range; +union { +struct resource platform_res; +struct range decoder_range; + };int interleave_ways;int interleave_granularity;enum cxl_decoder_type target_type;
Re: [PATCH 10/23] cxl/core: Convert decoder range to resource
In reply to Ben WidawskyQuoted 37 lines · Ben Widawsky
CXL decoders manage address ranges in a hierarchical fashion whereby a leaf is a unique subregion of its parent decoder (midlevel or root). It therefore makes sense to use the resource API for handling this.
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2
- Switch toDEFINE_RES_MEMfrom NAMED variant (Dan)
- Differentiate CFMWS resources and other decoder resources (Ben)
- Make decoder resources be range, nor resource (Dan)
- Set decoder name incxl_decoder_add()(Dan)--- drivers/cxl/acpi.c | 16 ++++++---------- drivers/cxl/core/bus.c | 19 +++++++++++++++++-- drivers/cxl/cxl.h | 8 ++++++-- 3 files changed, 29 insertions(+), 14 deletions(-)drivers/cxl/acpi.c +2
-4@@ -108,10 +108,8 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, cxld->flags =cfmws_to_decoder_flags (cfmws->restrictions); cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ){ - .start = cfmws->base_hpa, - .end = cfmws->base_hpa + cfmws->window_size -1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (cfmws->base_hpa, + cfmws->window_size); cxld->interleave_ways =CFMWS_INTERLEAVE_WAYS (cfmws); cxld->interleave_granularity =CFMWS_INTERLEAVE_GRANULARITY (cfmws);
drivers/cxl/acpi.c static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, · line 130
Quoted 8 lines
drivers/cxl/acpi.c +3
-2@@ -127,8 +125,9 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg,return 0 ; }dev_dbg (dev,"add: %s node: %d range %#llx-%#llx\n" , -dev_name (&cxld->dev),phys_to_target_node (cxld->range.start), - cfmws->base_hpa, cfmws->base_hpa + cfmws->window_size -1 ); +dev_name (&cxld->dev), +phys_to_target_node (cxld->platform_res.start), cfmws->base_hpa, + cfmws->base_hpa + cfmws->window_size -1 );
Since you converted to resource you can us %pr:
dev_dbg(dev, "add: %s node: %d range %pr\n", dev_name(&cxld->dev),
phys_to_target_node(cxld->platform_res.start),
&cxld->platform_res);Quoted 20 lines
drivers/cxl/acpi.c +1
-4return 0 ; }@@ -267,10 +266,7 @@ static int add_host_bridge_uport(struct device *match, void *arg) cxld->interleave_ways =1 ; cxld->interleave_granularity =PAGE_SIZE ; cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->range = (struct range ) { - .start =0 , - .end =-1 , - }; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 );device_lock (&port->dev); dport =list_first_entry (&port->dports,typeof (*dport), list);drivers/cxl/core/bus.c +0
-0
drivers/cxl/core/bus.c static ssize_t start_show(struct device *dev, struct device_attribute *attr, · line 49
Quoted 4 lines
drivers/cxl/core/bus.c +1
-0@@ -46,8 +46,14 @@ static ssize_t start_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 start =0 ;
No need to init to zero.
Quoted 12 lines
drivers/cxl/core/bus.c +6
-1-return sysfs_emit (buf,"%#llx\n" , cxld->range.start); +if (is_root_decoder (dev)) + start = cxld->platform_res.start; +else + start = cxld->decoder_range.start; + +return sysfs_emit (buf,"%#llx\n" , start); }static DEVICE_ATTR_RO (start);
drivers/cxl/core/bus.c static ssize_t size_show(struct device *dev, struct device_attribute *attr, · line 64
Quoted 4 lines
drivers/cxl/core/bus.c +1
-0@@ -55,8 +61,14 @@ static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf) {struct cxl_decoder *cxld =to_cxl_decoder (dev); +u64 size =0 ;
Same "no init" comment.
Quoted 12 lines
drivers/cxl/core/bus.c +6
-1-return sysfs_emit (buf,"%#llx\n" ,range_len (&cxld->range)); +if (is_root_decoder (dev)) + size =resource_size (&cxld->platform_res); +else + size =range_len (&cxld->decoder_range); + +return sysfs_emit (buf,"%#llx\n" , size); }static DEVICE_ATTR_RO (size);
drivers/cxl/core/bus.c int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) · line 564
Quoted 5 lines
drivers/cxl/core/bus.c +2
-0@@ -548,6 +560,9 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (rc)return rc; +if (is_root_decoder (dev)) + cxld->platform_res.name =dev_name (dev);
Maybe a comment about why the resource wants the name of the decoder? Just to help explain the motivation to separate this initialization step from the rest of the resource init.
Other than that you can add:
PATCH 11/23 cxl/core: Document and tighten up decoder APIs
In reply to Ben WidawskySince the code to add decoders for switches and endpoints is on the horizon it helps to have properly documented APIs. In addition, the decoder APIs will never need to support a negative count for downstream targets as the spec explicitly starts numbering them at 1, ie. even 0 is an "invalid" value which can be used as a sentinel.
---
This is respun from a previous incantation here:
https://lore.kernel.org/linux-cxl/[email protected]/
--- drivers/cxl/core/bus.c | 33 +++++++++++++++++++++++++++++++-- drivers/cxl/cxl.h | 3 ++- 2 files changed, 33 insertions(+), 3 deletions(-)
drivers/cxl/core/bus.c +31 -2
@@ -495,7 +495,20 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,return rc; } -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets) +/** +* cxl_decoder_alloc - Allocate a new CXL decoder +* @port: owning port of this decoder +* @nr_targets: downstream targets accessible by this decoder. All upstream +* ports and root ports must have at least 1 target. +* +* A port should contain one or more decoders. Each of those decoders enable +* some address space for CXL.mem utilization. A decoder is expected to be +* configured by the caller before registering. +* +* Return: A new cxl decoder to be registered by cxl_decoder_add() +*/ +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets) {struct cxl_decoder *cxld, cxld_const_init = { .nr_targets = nr_targets,
@@ -503,7 +516,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets)struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets <1 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );
@@ -535,6 +548,22 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets) }EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL ); +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* Certain types of decoders may not have any targets. The main example of this +* is an endpoint device. A more awkward example is a hostbridge whose root +* ports get hot added (technically possible, though unlikely). +* +* Context: Process context. Takes and releases the cxld's device lock. +* +* Return: Negative error code if the decoder wasn't properly configured; else +* returns 0. +*/ int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;
drivers/cxl/cxl.h +2 -1
@@ -288,7 +288,8 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id,struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev); -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets); +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Signature · 1 line
2.34.0
Re: [PATCH 11/23] cxl/core: Document and tighten up decoder APIs
Quoted 7 lines · On Fri, 19 Nov 2021 16:02:38 -0800 Ben Widawsky <[email protected]> wrote:
Since the code to add decoders for switches and endpoints is on the horizon it helps to have properly documented APIs. In addition, the decoder APIs will never need to support a negative count for downstream targets as the spec explicitly starts numbering them at 1, ie. even 0 is an "invalid" value which can be used as a sentinel.
Signed-off-by Ben Widawsky <[email protected]>
Quoted 82 lines
---
This is respun from a previous incantation here:
https://lore.kernel.org/linux-cxl/[email protected]/--- drivers/cxl/core/bus.c | 33 +++++++++++++++++++++++++++++++-- drivers/cxl/cxl.h | 3 ++- 2 files changed, 33 insertions(+), 3 deletions(-)drivers/cxl/core/bus.c +31
-2@@ -495,7 +495,20 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,return rc; } -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets) +/** +* cxl_decoder_alloc - Allocate a new CXL decoder +* @port: owning port of this decoder +* @nr_targets: downstream targets accessible by this decoder. All upstream +* ports and root ports must have at least 1 target. +* +* A port should contain one or more decoders. Each of those decoders enable +* some address space for CXL.mem utilization. A decoder is expected to be +* configured by the caller before registering. +* +* Return: A new cxl decoder to be registered by cxl_decoder_add() +*/ +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets) {struct cxl_decoder *cxld, cxld_const_init = { .nr_targets = nr_targets,@@ -503,7 +516,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets)struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets <1 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );@@ -535,6 +548,22 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets) }EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL ); +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* Certain types of decoders may not have any targets. The main example of this +* is an endpoint device. A more awkward example is a hostbridge whose root +* ports get hot added (technically possible, though unlikely). +* +* Context: Process context. Takes and releases the cxld's device lock. +* +* Return: Negative error code if the decoder wasn't properly configured; else +* returns 0. +*/ int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;drivers/cxl/cxl.h +2
-1@@ -288,7 +288,8 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id,struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev); -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets); +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Re: [PATCH 11/23] cxl/core: Document and tighten up decoder APIs
In reply to Ben WidawskyQuoted 6 lines · Ben Widawsky
Since the code to add decoders for switches and endpoints is on the horizon it helps to have properly documented APIs. In addition, the decoder APIs will never need to support a negative count for downstream targets as the spec explicitly starts numbering them at 1, ie. even 0 is an "invalid" value which can be used as a sentinel.
Looks good to me:
Quoted 87 lines
Signed-off-by Ben Widawsky <[email protected]>---
This is respun from a previous incantation here:
https://lore.kernel.org/linux-cxl/[email protected]/--- drivers/cxl/core/bus.c | 33 +++++++++++++++++++++++++++++++-- drivers/cxl/cxl.h | 3 ++- 2 files changed, 33 insertions(+), 3 deletions(-)drivers/cxl/core/bus.c +31
-2@@ -495,7 +495,20 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,return rc; } -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets) +/** +* cxl_decoder_alloc - Allocate a new CXL decoder +* @port: owning port of this decoder +* @nr_targets: downstream targets accessible by this decoder. All upstream +* ports and root ports must have at least 1 target. +* +* A port should contain one or more decoders. Each of those decoders enable +* some address space for CXL.mem utilization. A decoder is expected to be +* configured by the caller before registering. +* +* Return: A new cxl decoder to be registered by cxl_decoder_add() +*/ +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets) {struct cxl_decoder *cxld, cxld_const_init = { .nr_targets = nr_targets,@@ -503,7 +516,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets)struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets <1 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );@@ -535,6 +548,22 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets) }EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL ); +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* Certain types of decoders may not have any targets. The main example of this +* is an endpoint device. A more awkward example is a hostbridge whose root +* ports get hot added (technically possible, though unlikely). +* +* Context: Process context. Takes and releases the cxld's device lock. +* +* Return: Negative error code if the decoder wasn't properly configured; else +* returns 0. +*/ int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;drivers/cxl/cxl.h +2
-2@@ -288,7 +288,8 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport, int port_id,struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev); -struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,int nr_targets); +struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port, +unsigned int nr_targets);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld); --2.34.0
PATCH 12/23 cxl: Introduce endpoint decoders
In reply to Ben WidawskyEndpoints have decoders too. It is useful to share the same infrastructure from cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.
There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
--- drivers/cxl/core/bus.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)
drivers/cxl/core/bus.c +33 -8
@@ -187,6 +187,12 @@ static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = { NULL, }; +static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { + &cxl_decoder_base_attribute_group, + &cxl_base_attribute_group, +NULL , +}; +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);
@@ -196,6 +202,12 @@ static void cxl_decoder_release(struct device *dev)kfree (cxld); } +static const struct device_type cxl_decoder_endpoint_type = { + .name ="cxl_decoder_endpoint" , + .release = cxl_decoder_release, + .groups = cxl_decoder_endpoint_attribute_groups, +}; +static const struct device_type cxl_decoder_switch_type = { .name ="cxl_decoder_switch" , .release = cxl_decoder_release,
@@ -208,6 +220,11 @@ static const struct device_type cxl_decoder_root_type = { .groups = cxl_decoder_root_attribute_groups, }; +static bool is_endpoint_decoder (struct device *dev) +{ +return dev->type == &cxl_decoder_endpoint_type; +} +bool is_root_decoder (struct device *dev) {return dev->type == &cxl_decoder_root_type;
@@ -499,7 +516,9 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,* cxl_decoder_alloc - Allocate a new CXL decoder * @port: owning port of this decoder * @nr_targets: downstream targets accessible by this decoder. All upstream -* ports and root ports must have at least 1 target. +* ports and root ports must have at least 1 target. Endpoint +* devices will have 0 targets. Callers wishing to register an +* endpoint device should specify 0. * * A port should contain one or more decoders. Each of those decoders enable * some address space for CXL.mem utilization. A decoder is expected to be
@@ -516,7 +535,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );
@@ -535,8 +554,11 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, dev->parent = &port->dev; dev->bus = &cxl_bus_type; +/* Endpoints don't have a target list */ +if (nr_targets ==0 ) + dev->type = &cxl_decoder_endpoint_type;/* root ports do not have a cxl_port_type parent */ -if (port->dev.parent->type == &cxl_port_type) +else if (port->dev.parent->type == &cxl_port_type) dev->type = &cxl_decoder_switch_type;else dev->type = &cxl_decoder_root_type;
@@ -579,12 +601,15 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (cxld->interleave_ways <1 )return -EINVAL ; - port =to_cxl_port (cxld->dev.parent); - rc =decoder_populate_targets (cxld, port, target_map); -if (rc) -return rc; - dev = &cxld->dev; + + port =to_cxl_port (cxld->dev.parent); +if (!is_endpoint_decoder (dev)) { + rc =decoder_populate_targets (cxld, port, target_map); +if (rc) +return rc; + } + rc =dev_set_name (dev,"decoder%d.%d" , port->id, cxld->id);if (rc)return rc;
Signature · 1 line
2.34.0
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Signed-off-by Ben Widawsky <[email protected]>
I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base cxl_decode_alloc(..., bool ep) and possibly wrappers for the non ep case and ep one.
Jonathan
Quoted 100 lines
--- drivers/cxl/core/bus.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)drivers/cxl/core/bus.c +33
-8@@ -187,6 +187,12 @@ static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = { NULL, }; +static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { + &cxl_decoder_base_attribute_group, + &cxl_base_attribute_group, +NULL , +}; +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);@@ -196,6 +202,12 @@ static void cxl_decoder_release(struct device *dev)kfree (cxld); } +static const struct device_type cxl_decoder_endpoint_type = { + .name ="cxl_decoder_endpoint" , + .release = cxl_decoder_release, + .groups = cxl_decoder_endpoint_attribute_groups, +}; +static const struct device_type cxl_decoder_switch_type = { .name ="cxl_decoder_switch" , .release = cxl_decoder_release,@@ -208,6 +220,11 @@ static const struct device_type cxl_decoder_root_type = { .groups = cxl_decoder_root_attribute_groups, }; +static bool is_endpoint_decoder (struct device *dev) +{ +return dev->type == &cxl_decoder_endpoint_type; +} +bool is_root_decoder (struct device *dev) {return dev->type == &cxl_decoder_root_type;@@ -499,7 +516,9 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,* cxl_decoder_alloc - Allocate a new CXL decoder * @port: owning port of this decoder * @nr_targets: downstream targets accessible by this decoder. All upstream -* ports and root ports must have at least 1 target. +* ports and root ports must have at least 1 target. Endpoint +* devices will have 0 targets. Callers wishing to register an +* endpoint device should specify 0. * * A port should contain one or more decoders. Each of those decoders enable * some address space for CXL.mem utilization. A decoder is expected to be @@ -516,7 +535,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );@@ -535,8 +554,11 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, dev->parent = &port->dev; dev->bus = &cxl_bus_type; +/* Endpoints don't have a target list */ +if (nr_targets ==0 ) + dev->type = &cxl_decoder_endpoint_type;/* root ports do not have a cxl_port_type parent */ -if (port->dev.parent->type == &cxl_port_type) +else if (port->dev.parent->type == &cxl_port_type) dev->type = &cxl_decoder_switch_type;else dev->type = &cxl_decoder_root_type;@@ -579,12 +601,15 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (cxld->interleave_ways <1 )return -EINVAL ; - port =to_cxl_port (cxld->dev.parent); - rc =decoder_populate_targets (cxld, port, target_map); -if (rc) -return rc; - dev = &cxld->dev; + + port =to_cxl_port (cxld->dev.parent); +if (!is_endpoint_decoder (dev)) { + rc =decoder_populate_targets (cxld, port, target_map); +if (rc) +return rc; + } + rc =dev_set_name (dev,"decoder%d.%d" , port->id, cxld->id);if (rc)return rc;
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Quoted 8 lines
Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base
cxl_decode_alloc(..., bool ep)and possibly wrappers for the non ep case and ep one.Jonathan
My inclination is the opposite. However, I think you and Dan both brought up something to this effect in the previous RFCs.
Dan, do you have a preference here?
Quoted 101 lines
Quoted 100 lines
--- drivers/cxl/core/bus.c | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-)drivers/cxl/core/bus.c +33
-8@@ -187,6 +187,12 @@ static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = { NULL, }; +static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { + &cxl_decoder_base_attribute_group, + &cxl_base_attribute_group, +NULL , +}; +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);@@ -196,6 +202,12 @@ static void cxl_decoder_release(struct device *dev)kfree (cxld); } +static const struct device_type cxl_decoder_endpoint_type = { + .name ="cxl_decoder_endpoint" , + .release = cxl_decoder_release, + .groups = cxl_decoder_endpoint_attribute_groups, +}; +static const struct device_type cxl_decoder_switch_type = { .name ="cxl_decoder_switch" , .release = cxl_decoder_release,@@ -208,6 +220,11 @@ static const struct device_type cxl_decoder_root_type = { .groups = cxl_decoder_root_attribute_groups, }; +static bool is_endpoint_decoder (struct device *dev) +{ +return dev->type == &cxl_decoder_endpoint_type; +} +bool is_root_decoder (struct device *dev) {return dev->type == &cxl_decoder_root_type;@@ -499,7 +516,9 @@ static int decoder_populate_targets(struct cxl_decoder *cxld,* cxl_decoder_alloc - Allocate a new CXL decoder * @port: owning port of this decoder * @nr_targets: downstream targets accessible by this decoder. All upstream -* ports and root ports must have at least 1 target. +* ports and root ports must have at least 1 target. Endpoint +* devices will have 0 targets. Callers wishing to register an +* endpoint device should specify 0. * * A port should contain one or more decoders. Each of those decoders enable * some address space for CXL.mem utilization. A decoder is expected to be @@ -516,7 +535,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,struct device *dev;int rc =0 ; -if (nr_targets >CXL_DECODER_MAX_INTERLEAVE || nr_targets ==0 ) +if (nr_targets >CXL_DECODER_MAX_INTERLEAVE )return ERR_PTR (-EINVAL ); cxld =kzalloc (struct_size (cxld, target, nr_targets),GFP_KERNEL );@@ -535,8 +554,11 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, dev->parent = &port->dev; dev->bus = &cxl_bus_type; +/* Endpoints don't have a target list */ +if (nr_targets ==0 ) + dev->type = &cxl_decoder_endpoint_type;/* root ports do not have a cxl_port_type parent */ -if (port->dev.parent->type == &cxl_port_type) +else if (port->dev.parent->type == &cxl_port_type) dev->type = &cxl_decoder_switch_type;else dev->type = &cxl_decoder_root_type;@@ -579,12 +601,15 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)if (cxld->interleave_ways <1 )return -EINVAL ; - port =to_cxl_port (cxld->dev.parent); - rc =decoder_populate_targets (cxld, port, target_map); -if (rc) -return rc; - dev = &cxld->dev; + + port =to_cxl_port (cxld->dev.parent); +if (!is_endpoint_decoder (dev)) { + rc =decoder_populate_targets (cxld, port, target_map); +if (rc) +return rc; + } + rc =dev_set_name (dev,"decoder%d.%d" , port->id, cxld->id);if (rc)return rc;
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 19 lines · Ben Widawsky
Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Signed-off-by Ben Widawsky <[email protected]>I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base
cxl_decode_alloc(..., bool ep)and possibly wrappers for the non ep case and ep one.
Quoted 7 lines
Quoted 2 lines
Jonathan
My inclination is the opposite. However, I think you and Dan both brought up something to this effect in the previous RFCs.
Dan, do you have a preference here?
I was thinking something along the lines of what Jonathan wants, explicit per-type APIs, but internal / private to the core can use heuristics like nr_targets == 0 == endpoint.
So unexport cxl_decoder_alloc() and have separate:
cxl_root_decoder_alloc()cxl_switch_decoder_alloc()cxl_endpoint_decoder_alloc()
...apis that use a static cxl_decoder_alloc() internally. Probably also wants a cxl_endpoint_decoder_add() that drops the need to pass a NULL @target_map.
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 34 lines · Dan Williams
Quoted 19 lines · Ben Widawsky
Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Signed-off-by Ben Widawsky <[email protected]>I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base
cxl_decode_alloc(..., bool ep)and possibly wrappers for the non ep case and ep one.Quoted 7 lines
Quoted 2 lines
Jonathan
My inclination is the opposite. However, I think you and Dan both brought up something to this effect in the previous RFCs.
Dan, do you have a preference here?
I was thinking something along the lines of what Jonathan wants, explicit per-type APIs, but internal / private to the core can use heuristics like
nr_targets== 0 == endpoint.So unexport
cxl_decoder_alloc()and have separate:
Quoted 7 lines
cxl_root_decoder_alloc()cxl_switch_decoder_alloc()cxl_endpoint_decoder_alloc()...apis that use a static
cxl_decoder_alloc()internally. Probably also wants acxl_endpoint_decoder_add()that drops the need to pass a NULL @target_map.
Would you a like a prep patch to set up the APIs first, or just do it all in one?
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 40 lines · Ben Widawsky
Quoted 38 lines · Dan Williams
Quoted 19 lines · Ben Widawsky
Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Signed-off-by Ben Widawsky <[email protected]>I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base
cxl_decode_alloc(..., bool ep)and possibly wrappers for the non ep case and ep one.Quoted 7 lines
Quoted 2 lines
Jonathan
My inclination is the opposite. However, I think you and Dan both brought up something to this effect in the previous RFCs.
Dan, do you have a preference here?
I was thinking something along the lines of what Jonathan wants, explicit per-type APIs, but internal / private to the core can use heuristics like
nr_targets== 0 == endpoint.So unexport
cxl_decoder_alloc()and have separate:
cxl_root_decoder_alloc()cxl_switch_decoder_alloc()cxl_endpoint_decoder_alloc()
Quoted 6 lines
Quoted 3 lines
...apis that use a static
cxl_decoder_alloc()internally. Probably also wants acxl_endpoint_decoder_add()that drops the need to pass a NULL @target_map.Would you a like a prep patch to set up the APIs first, or just do it all in one?
Prep patch to switch over the current usages to the new style before introducing more helpers sounds good to me.
Re: [PATCH 12/23] cxl: Introduce endpoint decoders
Quoted 45 lines · Dan Williams
Quoted 44 lines · Ben Widawsky
Quoted 41 lines · Dan Williams
Quoted 19 lines · Ben Widawsky
Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:39 -0800 Ben Widawsky <[email protected]> wrote:
Endpoints have decoders too. It is useful to share the same infrastructure from
cxl_core. Endpoints do not have dports (downstream targets), only the underlying physical medium. As a result, some special casing is needed.There is no functional change introduced yet as endpoints don't actually enumerate decoders yet.
Signed-off-by Ben Widawsky <[email protected]>I'm not a fan of special values like using 0 here to indicate endpoint device. I'd rather see a base
cxl_decode_alloc(..., bool ep)and possibly wrappers for the non ep case and ep one.Quoted 7 lines
Quoted 2 lines
Jonathan
My inclination is the opposite. However, I think you and Dan both brought up something to this effect in the previous RFCs.
Dan, do you have a preference here?
I was thinking something along the lines of what Jonathan wants, explicit per-type APIs, but internal / private to the core can use heuristics like
nr_targets== 0 == endpoint.So unexport
cxl_decoder_alloc()and have separate:
cxl_root_decoder_alloc()cxl_switch_decoder_alloc()cxl_endpoint_decoder_alloc()...apis that use a static
cxl_decoder_alloc()internally. Probably also wants acxl_endpoint_decoder_add()that drops the need to pass a NULL @target_map.
Quoted 5 lines
Quoted 2 lines
Would you a like a prep patch to set up the APIs first, or just do it all in one?
Prep patch to switch over the current usages to the new style before introducing more helpers sounds good to me.
Thanks for the suggestions... Looking again, I think it makes sense to squash it into the patch before this which documents and tightens up this exact API.
PATCH 13/23 cxl/core: Move target population locking to caller
In reply to Ben WidawskyIn preparation for a port driver that enumerates a descendant port + decoder hierarchy, arrange for an unlocked version of cxl_decoder_add(). Otherwise a port-driver that adds a child decoder will deadlock on the device_lock() in ->probe().
---
Changes since RFCv2:
- Reword commit message (Dan)
- Move decoder API changes into this patch (Dan)
--- drivers/cxl/core/bus.c | 59 +++++++++++++++++++++++++++++++----------- drivers/cxl/cxl.h | 1 + 2 files changed, 45 insertions(+), 15 deletions(-)
drivers/cxl/core/bus.c +44 -15
@@ -487,28 +487,22 @@ static int decoder_populate_targets(struct cxl_decoder *cxld, {int rc =0 , i; +device_lock_assert (&port->dev); +if (!target_map)return 0 ; -device_lock (&port->dev); -if (list_empty (&port->dports)) { - rc = -EINVAL ; - goto out_unlock; - } +if (list_empty (&port->dports)) +return -EINVAL ;for (i =0 ; i < cxld->nr_targets; i++) {struct cxl_dport *dport =find_dport (port, target_map[i]); -if (!dport) { - rc = -ENXIO ; - goto out_unlock; - } +if (!dport) +return -ENXIO ; cxld->target[i] = dport; } -out_unlock: -device_unlock (&port->dev); -return rc; }
@@ -571,7 +565,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL );/** -* cxl_decoder_add - Add a decoder with targets +* cxl_decoder_add_locked - Add a decoder with targets * @cxld: The cxl decoder allocated by cxl_decoder_alloc() * @target_map: A list of downstream ports that this decoder can direct memory * traffic to. These numbers should correspond with the port number
@@ -581,12 +575,14 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_alloc, CXL);* is an endpoint device. A more awkward example is a hostbridge whose root * ports get hot added (technically possible, though unlikely). * -* Context: Process context. Takes and releases the cxld's device lock. +* This is the locked variant of cxl_decoder_add(). +* +* Context: Process context. Expects the cxld's device lock to be held. * * Return: Negative error code if the decoder wasn't properly configured; else * returns 0. */ -int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;struct device *dev;
@@ -619,6 +615,39 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)return device_add (dev); } +EXPORT_SYMBOL_NS_GPL (cxl_decoder_add_locked,CXL ); + +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* This is the unlocked variant of cxl_decoder_add_locked(). +* See cxl_decoder_add_locked(). +* +* Context: Process context. Takes and releases the cxld's device lock. +*/ +int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +{ +struct cxl_port *port; +int rc; + +if (WARN_ON_ONCE (!cxld)) +return -EINVAL ; + +if (WARN_ON_ONCE (IS_ERR (cxld))) +return PTR_ERR (cxld); + + port =to_cxl_port (cxld->dev.parent); + +device_lock (&port->dev); + rc =cxl_decoder_add_locked (cxld, target_map); +device_unlock (&port->dev); + +return rc; +}EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );static void cxld_unregister (void *dev)
drivers/cxl/cxl.h +1 -0
@@ -290,6 +290,7 @@ struct cxl_decoder *to_cxl_decoder(struct device *dev);bool is_root_decoder (struct device *dev);struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,unsigned int nr_targets); +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Signature · 1 line
2.34.0
Re: [PATCH 13/23] cxl/core: Move target population locking to caller
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:40 -0800 Ben Widawsky <[email protected]> wrote:
In preparation for a port driver that enumerates a descendant port + decoder hierarchy, arrange for an unlocked version of
cxl_decoder_add(). Otherwise a port-driver that adds a child decoder will deadlock on thedevice_lock()in ->probe().
I think this description should call out that the lock was originally taken for a much shorter time in decoder_populate_targets() but is moved up one layer.
One other query inline. Seems like we the WARN_ON stuff is a bit over paranoid given what's visible in this patch. If there is a good reason for that, then add something to the patch description to justify it.
Quoted 92 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Reword commit message (Dan)
- Move decoder API changes into this patch (Dan)--- drivers/cxl/core/bus.c | 59 +++++++++++++++++++++++++++++++----------- drivers/cxl/cxl.h | 1 + 2 files changed, 45 insertions(+), 15 deletions(-)drivers/cxl/core/bus.c +22
-15@@ -487,28 +487,22 @@ static int decoder_populate_targets(struct cxl_decoder *cxld, {int rc =0 , i; +device_lock_assert (&port->dev); +if (!target_map)return 0 ; -device_lock (&port->dev); -if (list_empty (&port->dports)) { - rc = -EINVAL ; - goto out_unlock; - } +if (list_empty (&port->dports)) +return -EINVAL ;for (i =0 ; i < cxld->nr_targets; i++) {struct cxl_dport *dport =find_dport (port, target_map[i]); -if (!dport) { - rc = -ENXIO ; - goto out_unlock; - } +if (!dport) +return -ENXIO ; cxld->target[i] = dport; } -out_unlock: -device_unlock (&port->dev); -return rc; }@@ -571,7 +565,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL );/** -* cxl_decoder_add - Add a decoder with targets +* cxl_decoder_add_locked - Add a decoder with targets * @cxld: The cxl decoder allocated by cxl_decoder_alloc() * @target_map: A list of downstream ports that this decoder can direct memory * traffic to. These numbers should correspond with the port number @@ -581,12 +575,14 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_alloc, CXL);* is an endpoint device. A more awkward example is a hostbridge whose root * ports get hot added (technically possible, though unlikely). * -* Context: Process context. Takes and releases the cxld's device lock. +* This is the locked variant of cxl_decoder_add(). +* +* Context: Process context. Expects the cxld's device lock to be held. * * Return: Negative error code if the decoder wasn't properly configured; else * returns 0. */ -int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;struct device *dev;@@ -619,6 +615,39 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)return device_add (dev); } +EXPORT_SYMBOL_NS_GPL (cxl_decoder_add_locked,CXL ); + +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* This is the unlocked variant of cxl_decoder_add_locked(). +* See cxl_decoder_add_locked().
drivers/cxl/core/bus.c int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) · line 638
Quoted 10 lines
drivers/cxl/core/bus.c +10
-0+* +* Context: Process context. Takes and releases the cxld's device lock. +*/ +int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +{ +struct cxl_port *port; +int rc; + +if (WARN_ON_ONCE (!cxld)) +return -EINVAL ;
Why do we now need these protections but didn't before?
Quoted 27 lines
drivers/cxl/core/bus.c +12
-0+ +if (WARN_ON_ONCE (IS_ERR (cxld))) +return PTR_ERR (cxld); + + port =to_cxl_port (cxld->dev.parent); + +device_lock (&port->dev); + rc =cxl_decoder_add_locked (cxld, target_map); +device_unlock (&port->dev); + +return rc; +}EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );static void cxld_unregister (void *dev)drivers/cxl/cxl.h +1
-0@@ -290,6 +290,7 @@ struct cxl_decoder *to_cxl_decoder(struct device *dev);bool is_root_decoder (struct device *dev);struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,unsigned int nr_targets); +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Re: [PATCH 13/23] cxl/core: Move target population locking to caller
Quoted 9 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:40 -0800 Ben Widawsky <[email protected]> wrote:
In preparation for a port driver that enumerates a descendant port + decoder hierarchy, arrange for an unlocked version of
cxl_decoder_add(). Otherwise a port-driver that adds a child decoder will deadlock on thedevice_lock()in ->probe().
Quoted 3 lines
I think this description should call out that the lock was originally taken for a much shorter time in
decoder_populate_targets()but is moved up one layer.
Sounds good.
Quoted 102 lines
One other query inline. Seems like we the
WARN_ONstuff is a bit over paranoid given what's visible in this patch. If there is a good reason for that, then add something to the patch description to justify it.Quoted 96 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Reword commit message (Dan)
- Move decoder API changes into this patch (Dan)--- drivers/cxl/core/bus.c | 59 +++++++++++++++++++++++++++++++----------- drivers/cxl/cxl.h | 1 + 2 files changed, 45 insertions(+), 15 deletions(-)drivers/cxl/core/bus.c +26
-15@@ -487,28 +487,22 @@ static int decoder_populate_targets(struct cxl_decoder *cxld, {int rc =0 , i; +device_lock_assert (&port->dev); +if (!target_map)return 0 ; -device_lock (&port->dev); -if (list_empty (&port->dports)) { - rc = -EINVAL ; - goto out_unlock; - } +if (list_empty (&port->dports)) +return -EINVAL ;for (i =0 ; i < cxld->nr_targets; i++) {struct cxl_dport *dport =find_dport (port, target_map[i]); -if (!dport) { - rc = -ENXIO ; - goto out_unlock; - } +if (!dport) +return -ENXIO ; cxld->target[i] = dport; } -out_unlock: -device_unlock (&port->dev); -return rc; }@@ -571,7 +565,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL );/** -* cxl_decoder_add - Add a decoder with targets +* cxl_decoder_add_locked - Add a decoder with targets * @cxld: The cxl decoder allocated by cxl_decoder_alloc() * @target_map: A list of downstream ports that this decoder can direct memory * traffic to. These numbers should correspond with the port number @@ -581,12 +575,14 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_alloc, CXL);* is an endpoint device. A more awkward example is a hostbridge whose root * ports get hot added (technically possible, though unlikely). * -* Context: Process context. Takes and releases the cxld's device lock. +* This is the locked variant of cxl_decoder_add(). +* +* Context: Process context. Expects the cxld's device lock to be held. * * Return: Negative error code if the decoder wasn't properly configured; else * returns 0. */ -int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;struct device *dev;@@ -619,6 +615,39 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)return device_add (dev); } +EXPORT_SYMBOL_NS_GPL (cxl_decoder_add_locked,CXL ); + +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* This is the unlocked variant of cxl_decoder_add_locked(). +* See cxl_decoder_add_locked(). +* +* Context: Process context. Takes and releases the cxld's device lock. +*/ +int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map)
Quoted 8 lines
drivers/cxl/core/bus.c int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) · line 638
Quoted 6 lines
drivers/cxl/core/bus.c +6
-0+{ +struct cxl_port *port; +int rc; + +if (WARN_ON_ONCE (!cxld)) +return -EINVAL ;Why do we now need these protections but didn't before?
I don't quite understand what you're trying to point out.
Prior to this patch, cxl_decoder_add() checks:
- !cxld
- IS_ERR(cxld)
- cxld->interleave_ways != 0
After this patch, cxl_decoder_add() checks:
- !cxld
- IS_ERR(cxld)
- (and then calls cxl_decoder_add_locked())
And cxl_decoder_add_locked() checks:
- !cxld
- IS_ERR(cxld)
- cxld->interleave_ways != 0
Ultimately we want to check all 3, and since cxl_decoder_add() calls cxl_decoder_add_locked(), we're good there. The problem is to get from a cxld to a port, you need to make sure you have a valid cxld, and the API previously allowed !cxld and IS_ERR(cxld). So there are duplicative checks if you call cxl_decoder_add(), but other than that I don't see any new protections.
Quoted 30 lines
Quoted 29 lines
drivers/cxl/core/bus.c +12
-0+ +if (WARN_ON_ONCE (IS_ERR (cxld))) +return PTR_ERR (cxld); + + port =to_cxl_port (cxld->dev.parent); + +device_lock (&port->dev); + rc =cxl_decoder_add_locked (cxld, target_map); +device_unlock (&port->dev); + +return rc; +}EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );static void cxld_unregister (void *dev)drivers/cxl/cxl.h +1
-0@@ -290,6 +290,7 @@ struct cxl_decoder *to_cxl_decoder(struct device *dev);bool is_root_decoder (struct device *dev);struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,unsigned int nr_targets); +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Jonathan Cameron
Reviewed-byRe: [PATCH 13/23] cxl/core: Move target population locking to caller
Quoted 10 lines · On Mon, 22 Nov 2021 13:58:01 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 19 Nov 2021 16:02:40 -0800 Ben Widawsky <[email protected]> wrote:
In preparation for a port driver that enumerates a descendant port + decoder hierarchy, arrange for an unlocked version of
cxl_decoder_add(). Otherwise a port-driver that adds a child decoder will deadlock on thedevice_lock()in ->probe().
Quoted 5 lines
Quoted 3 lines
I think this description should call out that the lock was originally taken for a much shorter time in
decoder_populate_targets()but is moved up one layer.Sounds good.
With that added and below discussion resolved.
Quoted 129 lines
Quoted 103 lines
One other query inline. Seems like we the
WARN_ONstuff is a bit over paranoid given what's visible in this patch. If there is a good reason for that, then add something to the patch description to justify it.Quoted 96 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Reword commit message (Dan)
- Move decoder API changes into this patch (Dan)--- drivers/cxl/core/bus.c | 59 +++++++++++++++++++++++++++++++----------- drivers/cxl/cxl.h | 1 + 2 files changed, 45 insertions(+), 15 deletions(-)drivers/cxl/core/bus.c +26
-15@@ -487,28 +487,22 @@ static int decoder_populate_targets(struct cxl_decoder *cxld, {int rc =0 , i; +device_lock_assert (&port->dev); +if (!target_map)return 0 ; -device_lock (&port->dev); -if (list_empty (&port->dports)) { - rc = -EINVAL ; - goto out_unlock; - } +if (list_empty (&port->dports)) +return -EINVAL ;for (i =0 ; i < cxld->nr_targets; i++) {struct cxl_dport *dport =find_dport (port, target_map[i]); -if (!dport) { - rc = -ENXIO ; - goto out_unlock; - } +if (!dport) +return -ENXIO ; cxld->target[i] = dport; } -out_unlock: -device_unlock (&port->dev); -return rc; }@@ -571,7 +565,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL );/** -* cxl_decoder_add - Add a decoder with targets +* cxl_decoder_add_locked - Add a decoder with targets * @cxld: The cxl decoder allocated by cxl_decoder_alloc() * @target_map: A list of downstream ports that this decoder can direct memory * traffic to. These numbers should correspond with the port number @@ -581,12 +575,14 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_alloc, CXL);* is an endpoint device. A more awkward example is a hostbridge whose root * ports get hot added (technically possible, though unlikely). * -* Context: Process context. Takes and releases the cxld's device lock. +* This is the locked variant of cxl_decoder_add(). +* +* Context: Process context. Expects the cxld's device lock to be held. * * Return: Negative error code if the decoder wasn't properly configured; else * returns 0. */ -int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;struct device *dev;@@ -619,6 +615,39 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)return device_add (dev); } +EXPORT_SYMBOL_NS_GPL (cxl_decoder_add_locked,CXL ); + +/** +* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* This is the unlocked variant of cxl_decoder_add_locked(). +* See cxl_decoder_add_locked(). +* +* Context: Process context. Takes and releases the cxld's device lock. +*/ +int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map)Quoted 8 lines
drivers/cxl/core/bus.c int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) · line 638
Quoted 6 lines
drivers/cxl/core/bus.c +6
-0+{ +struct cxl_port *port; +int rc; + +if (WARN_ON_ONCE (!cxld)) +return -EINVAL ;Why do we now need these protections but didn't before?
I don't quite understand what you're trying to point out.
Prior to this patch,
cxl_decoder_add()checks:
- !cxld
-IS_ERR(cxld)
-cxld->interleave_ways!= 0After this patch,
cxl_decoder_add()checks:
- !cxld
-IS_ERR(cxld)
- (and then callscxl_decoder_add_locked())And
cxl_decoder_add_locked()checks:
- !cxld
-IS_ERR(cxld)
-cxld->interleave_ways!= 0
Quoted 5 lines
Ultimately we want to check all 3, and since
cxl_decoder_add()callscxl_decoder_add_locked(), we're good there. The problem is to get from a cxld to a port, you need to make sure you have a valid cxld, and the API previously allowed !cxld andIS_ERR(cxld). So there are duplicative checks if you callcxl_decoder_add(), but other than that I don't see any new protections.
Ah. It was the duplication that I didn't follow.
Fair enough
J
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
drivers/cxl/core/bus.c +12
-0+ +if (WARN_ON_ONCE (IS_ERR (cxld))) +return PTR_ERR (cxld); + + port =to_cxl_port (cxld->dev.parent); + +device_lock (&port->dev); + rc =cxl_decoder_add_locked (cxld, target_map); +device_unlock (&port->dev); + +return rc; +}EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL );static void cxld_unregister (void *dev)drivers/cxl/cxl.h +1
-0@@ -290,6 +290,7 @@ struct cxl_decoder *to_cxl_decoder(struct device *dev);bool is_root_decoder (struct device *dev);struct cxl_decoder *cxl_decoder_alloc (struct cxl_port *port,unsigned int nr_targets); +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map);int cxl_decoder_autoremove (struct device *host,struct cxl_decoder *cxld);
Re: [PATCH 13/23] cxl/core: Move target population locking to caller
In reply to Ben WidawskyQuoted 90 lines · Ben Widawsky
In preparation for a port driver that enumerates a descendant port + decoder hierarchy, arrange for an unlocked version of
cxl_decoder_add(). Otherwise a port-driver that adds a child decoder will deadlock on thedevice_lock()in ->probe().Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
- Reword commit message (Dan)
- Move decoder API changes into this patch (Dan)--- drivers/cxl/core/bus.c | 59 +++++++++++++++++++++++++++++++----------- drivers/cxl/cxl.h | 1 + 2 files changed, 45 insertions(+), 15 deletions(-)drivers/cxl/core/bus.c +14
-15@@ -487,28 +487,22 @@ static int decoder_populate_targets(struct cxl_decoder *cxld, {int rc =0 , i; +device_lock_assert (&port->dev); +if (!target_map)return 0 ; -device_lock (&port->dev); -if (list_empty (&port->dports)) { - rc = -EINVAL ; - goto out_unlock; - } +if (list_empty (&port->dports)) +return -EINVAL ;for (i =0 ; i < cxld->nr_targets; i++) {struct cxl_dport *dport =find_dport (port, target_map[i]); -if (!dport) { - rc = -ENXIO ; - goto out_unlock; - } +if (!dport) +return -ENXIO ; cxld->target[i] = dport; } -out_unlock: -device_unlock (&port->dev); -return rc; }@@ -571,7 +565,7 @@ struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port,EXPORT_SYMBOL_NS_GPL (cxl_decoder_alloc,CXL );/** -* cxl_decoder_add - Add a decoder with targets +* cxl_decoder_add_locked - Add a decoder with targets * @cxld: The cxl decoder allocated by cxl_decoder_alloc() * @target_map: A list of downstream ports that this decoder can direct memory * traffic to. These numbers should correspond with the port number @@ -581,12 +575,14 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_alloc, CXL);* is an endpoint device. A more awkward example is a hostbridge whose root * ports get hot added (technically possible, though unlikely). * -* Context: Process context. Takes and releases the cxld's device lock. +* This is the locked variant of cxl_decoder_add(). +* +* Context: Process context. Expects the cxld's device lock to be held. * * Return: Negative error code if the decoder wasn't properly configured; else * returns 0. */ -int cxl_decoder_add (struct cxl_decoder *cxld,int *target_map) +int cxl_decoder_add_locked (struct cxl_decoder *cxld,int *target_map) {struct cxl_port *port;struct device *dev;@@ -619,6 +615,39 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)return device_add (dev); } +EXPORT_SYMBOL_NS_GPL (cxl_decoder_add_locked,CXL ); + +/**
drivers/cxl/core/bus.c int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) · line 630
Quoted 10 lines
drivers/cxl/core/bus.c +10
-0+* cxl_decoder_add - Add a decoder with targets +* @cxld: The cxl decoder allocated by cxl_decoder_alloc() +* @target_map: A list of downstream ports that this decoder can direct memory +* traffic to. These numbers should correspond with the port number +* in the PCIe Link Capabilities structure. +* +* This is the unlocked variant of cxl_decoder_add_locked(). +* See cxl_decoder_add_locked(). +* +* Context: Process context. Takes and releases the cxld's device lock.
No, it takes the port's lock to walk its dport list.
Otherwise, looks good to me:
PATCH 14/23 cxl: Introduce topology host registration
In reply to Ben WidawskyThe description of the CXL topology will be conveyed by a platform specific entity that is expected to be a singleton. For ACPI based systems, this is ACPI0017. When the topology host goes away, which as of now can only be triggered by module unload, it is desirable to have the entire topology cleaned up. Regular devm unwinding handles most situations already, but what's missing is the ability to teardown the root port. Since the root port is platform specific, the core needs a set of APIs to allow platform specific drivers to register their root ports. With that, all the automatic teardown can occur.
cxl_test makes for an interesting case. cxl_test creates an alternate universe where there are possibly two root topology hosts (a real ACPI0016, and a fake ACPI0016). For this to work in the future, cxl_acpi (or some future platform host driver) will need to be unloaded first.
---
The topology lock can be used for more things. I'd like to save that as an add-on patch since it's extra risk for no reward, at this point.
--- drivers/cxl/acpi.c | 18 ++++++++++--- drivers/cxl/core/bus.c | 57 +++++++++++++++++++++++++++++++++++++++--- drivers/cxl/cxl.h | 5 +++- 3 files changed, 73 insertions(+), 7 deletions(-)
drivers/cxl/acpi.c +15 -3
@@ -224,8 +224,7 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } - port =devm_cxl_add_port (host, match, dport->component_reg_phys, - root_port); + port =devm_cxl_add_port (match, dport->component_reg_phys, root_port);if (IS_ERR (port))return PTR_ERR (port);dev_dbg (host,"%s: add: %s\n" ,dev_name (match),dev_name (&port->dev));
@@ -376,6 +375,11 @@ static int add_root_nvdimm_bridge(struct device *match, void *data)return 1 ; } +static void clear_topology_host (void *data) +{ +cxl_unregister_topology_host (data); +} +static int cxl_acpi_probe (struct platform_device *pdev) {int rc;
@@ -384,7 +388,15 @@ static int cxl_acpi_probe(struct platform_device *pdev)struct acpi_device *adev =ACPI_COMPANION (host);struct cxl_cfmws_context ctx; - root_port =devm_cxl_add_port (host, host,CXL_RESOURCE_NONE ,NULL ); + rc =cxl_register_topology_host (host); +if (rc) +return rc; + + rc =devm_add_action_or_reset (host, clear_topology_host, host); +if (rc) +return rc; + + root_port =devm_cxl_add_port (host,CXL_RESOURCE_NONE , root_port);if (IS_ERR (root_port))return PTR_ERR (root_port);dev_dbg (host,"add: %s\n" ,dev_name (&root_port->dev));
drivers/cxl/core/bus.c +54 -3
84 lines not on this page · raw message
drivers/cxl/cxl.h +4 -1
@@ -152,6 +152,9 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20 +int cxl_register_topology_host (struct device *host); +void cxl_unregister_topology_host (struct device *host); +/* * cxl_decoder flags that define the type of memory / devices this * decoder supports as well as configuration lock status See "CXL 2.0
@@ -279,7 +282,7 @@ struct cxl_dport { };struct cxl_port *to_cxl_port (struct device *dev); -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *uport,resource_size_t component_reg_phys,struct cxl_port *parent_port);
Signature · 1 line
2.34.0
Re: [PATCH 14/23] cxl: Introduce topology host registration
Quoted 10 lines · On Fri, 19 Nov 2021 16:02:41 -0800 Ben Widawsky <[email protected]> wrote:
The description of the CXL topology will be conveyed by a platform specific entity that is expected to be a singleton. For ACPI based systems, this is ACPI0017. When the topology host goes away, which as of now can only be triggered by module unload, it is desirable to have the entire topology cleaned up. Regular devm unwinding handles most situations already, but what's missing is the ability to teardown the root port. Since the root port is platform specific, the core needs a set of APIs to allow platform specific drivers to register their root ports. With that, all the automatic teardown can occur.
Quoted 6 lines
cxl_testmakes for an interesting case.cxl_testcreates an alternate universe where there are possibly two root topology hosts (a real ACPI0016, and a fake ACPI0016). For this to work in the future,cxl_acpi(or some future platform host driver) will need to be unloaded first.Signed-off-by Ben Widawsky <[email protected]>
This is a little unusual looking but having followed through how it is used it seems like a sensible approach to me.
Quoted 166 lines
---
The topology lock can be used for more things. I'd like to save that as an add-on patch since it's extra risk for no reward, at this point.--- drivers/cxl/acpi.c | 18 ++++++++++--- drivers/cxl/core/bus.c | 57 +++++++++++++++++++++++++++++++++++++++--- drivers/cxl/cxl.h | 5 +++- 3 files changed, 73 insertions(+), 7 deletions(-)drivers/cxl/acpi.c +15
-3@@ -224,8 +224,7 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } - port =devm_cxl_add_port (host, match, dport->component_reg_phys, - root_port); + port =devm_cxl_add_port (match, dport->component_reg_phys, root_port);if (IS_ERR (port))return PTR_ERR (port);dev_dbg (host,"%s: add: %s\n" ,dev_name (match),dev_name (&port->dev));@@ -376,6 +375,11 @@ static int add_root_nvdimm_bridge(struct device *match, void *data)return 1 ; } +static void clear_topology_host (void *data) +{ +cxl_unregister_topology_host (data); +} +static int cxl_acpi_probe (struct platform_device *pdev) {int rc;@@ -384,7 +388,15 @@ static int cxl_acpi_probe(struct platform_device *pdev)struct acpi_device *adev =ACPI_COMPANION (host);struct cxl_cfmws_context ctx; - root_port =devm_cxl_add_port (host, host,CXL_RESOURCE_NONE ,NULL ); + rc =cxl_register_topology_host (host); +if (rc) +return rc; + + rc =devm_add_action_or_reset (host, clear_topology_host, host); +if (rc) +return rc; + + root_port =devm_cxl_add_port (host,CXL_RESOURCE_NONE , root_port);if (IS_ERR (root_port))return PTR_ERR (root_port);dev_dbg (host,"add: %s\n" ,dev_name (&root_port->dev));drivers/cxl/core/bus.c +54
-3@@ -25,6 +25,53 @@*/ static DEFINE_IDA (cxl_port_ida); +static DECLARE_RWSEM (topology_host_sem); + +static struct device *cxl_topology_host; + +int cxl_register_topology_host (struct device *host) +{ +down_write (&topology_host_sem); +if (cxl_topology_host) { +up_write (&topology_host_sem); +pr_warn ("%s host currently in use. Please try unloading %s" , +dev_name (cxl_topology_host), host->driver->name); +return -EBUSY ; + } + + cxl_topology_host = host; +up_write (&topology_host_sem); + +return 0 ; +} +EXPORT_SYMBOL_NS_GPL (cxl_register_topology_host,CXL ); + +void cxl_unregister_topology_host (struct device *host) +{ +down_write (&topology_host_sem); +if (cxl_topology_host == host) + cxl_topology_host =NULL ; +else +pr_warn ("topology host in use by %s\n" , + cxl_topology_host->driver->name); +up_write (&topology_host_sem); +} +EXPORT_SYMBOL_NS_GPL (cxl_unregister_topology_host,CXL ); + +static struct device *get_cxl_topology_host (void ) +{ +down_read (&topology_host_sem); +if (cxl_topology_host) +return cxl_topology_host; +up_read (&topology_host_sem); +return NULL ; +} + +static void put_cxl_topology_host (struct device *dev) +{ +WARN_ON (dev != cxl_topology_host); +up_read (&topology_host_sem); +}static ssize_t devtype_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -362,17 +409,16 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,/** * devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy -* @host: host device for devm operations * @uport: "physical" device implementing this upstream port * @component_reg_phys: (optional) for configurable cxl_port instances * @parent_port: next hop up in the CXL memory decode hierarchy */ -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *uport,resource_size_t component_reg_phys,struct cxl_port *parent_port) { +struct device *dev, *host;struct cxl_port *port; -struct device *dev;int rc; port =cxl_port_alloc (uport, component_reg_phys, parent_port);@@ -391,7 +437,12 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,if (rc) goto err; + host =get_cxl_topology_host (); +if (!host) +return ERR_PTR (-ENODEV ); + rc =devm_add_action_or_reset (host, unregister_port, port); +put_cxl_topology_host (host);if (rc)return ERR_PTR (rc);drivers/cxl/cxl.h +4
-1@@ -152,6 +152,9 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20 +int cxl_register_topology_host (struct device *host); +void cxl_unregister_topology_host (struct device *host); +/* * cxl_decoder flags that define the type of memory / devices this * decoder supports as well as configuration lock status See "CXL 2.0 @@ -279,7 +282,7 @@ struct cxl_dport { };struct cxl_port *to_cxl_port (struct device *dev); -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *uport,resource_size_t component_reg_phys,struct cxl_port *parent_port);
Re: [PATCH 14/23] cxl: Introduce topology host registration
Quoted 18 lines · Jonathan Cameron
Quoted 15 lines · On Fri, 19 Nov 2021 16:02:41 -0800 Ben Widawsky <[email protected]> wrote:
The description of the CXL topology will be conveyed by a platform specific entity that is expected to be a singleton. For ACPI based systems, this is ACPI0017. When the topology host goes away, which as of now can only be triggered by module unload, it is desirable to have the entire topology cleaned up. Regular devm unwinding handles most situations already, but what's missing is the ability to teardown the root port. Since the root port is platform specific, the core needs a set of APIs to allow platform specific drivers to register their root ports. With that, all the automatic teardown can occur.
cxl_testmakes for an interesting case.cxl_testcreates an alternate universe where there are possibly two root topology hosts (a real ACPI0016, and a fake ACPI0016). For this to work in the future,cxl_acpi(or some future platform host driver) will need to be unloaded first.
Quoted 6 lines
Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>This is a little unusual looking but having followed through how it is used it seems like a sensible approach to me.
Reviewed-by Jonathan Cameron <[email protected]>
Thanks. I noticed another commit message bug, s/ACPI0016/ACPI0017\/CEDT above.
Quoted 167 lines
Quoted 166 lines
---
The topology lock can be used for more things. I'd like to save that as an add-on patch since it's extra risk for no reward, at this point.--- drivers/cxl/acpi.c | 18 ++++++++++--- drivers/cxl/core/bus.c | 57 +++++++++++++++++++++++++++++++++++++++--- drivers/cxl/cxl.h | 5 +++- 3 files changed, 73 insertions(+), 7 deletions(-)drivers/cxl/acpi.c +15
-3@@ -224,8 +224,7 @@ static int add_host_bridge_uport(struct device *match, void *arg)return 0 ; } - port =devm_cxl_add_port (host, match, dport->component_reg_phys, - root_port); + port =devm_cxl_add_port (match, dport->component_reg_phys, root_port);if (IS_ERR (port))return PTR_ERR (port);dev_dbg (host,"%s: add: %s\n" ,dev_name (match),dev_name (&port->dev));@@ -376,6 +375,11 @@ static int add_root_nvdimm_bridge(struct device *match, void *data)return 1 ; } +static void clear_topology_host (void *data) +{ +cxl_unregister_topology_host (data); +} +static int cxl_acpi_probe (struct platform_device *pdev) {int rc;@@ -384,7 +388,15 @@ static int cxl_acpi_probe(struct platform_device *pdev)struct acpi_device *adev =ACPI_COMPANION (host);struct cxl_cfmws_context ctx; - root_port =devm_cxl_add_port (host, host,CXL_RESOURCE_NONE ,NULL ); + rc =cxl_register_topology_host (host); +if (rc) +return rc; + + rc =devm_add_action_or_reset (host, clear_topology_host, host); +if (rc) +return rc; + + root_port =devm_cxl_add_port (host,CXL_RESOURCE_NONE , root_port);if (IS_ERR (root_port))return PTR_ERR (root_port);dev_dbg (host,"add: %s\n" ,dev_name (&root_port->dev));drivers/cxl/core/bus.c +54
-3@@ -25,6 +25,53 @@*/ static DEFINE_IDA (cxl_port_ida); +static DECLARE_RWSEM (topology_host_sem); + +static struct device *cxl_topology_host; + +int cxl_register_topology_host (struct device *host) +{ +down_write (&topology_host_sem); +if (cxl_topology_host) { +up_write (&topology_host_sem); +pr_warn ("%s host currently in use. Please try unloading %s" , +dev_name (cxl_topology_host), host->driver->name); +return -EBUSY ; + } + + cxl_topology_host = host; +up_write (&topology_host_sem); + +return 0 ; +} +EXPORT_SYMBOL_NS_GPL (cxl_register_topology_host,CXL ); + +void cxl_unregister_topology_host (struct device *host) +{ +down_write (&topology_host_sem); +if (cxl_topology_host == host) + cxl_topology_host =NULL ; +else +pr_warn ("topology host in use by %s\n" , + cxl_topology_host->driver->name); +up_write (&topology_host_sem); +} +EXPORT_SYMBOL_NS_GPL (cxl_unregister_topology_host,CXL ); + +static struct device *get_cxl_topology_host (void ) +{ +down_read (&topology_host_sem); +if (cxl_topology_host) +return cxl_topology_host; +up_read (&topology_host_sem); +return NULL ; +} + +static void put_cxl_topology_host (struct device *dev) +{ +WARN_ON (dev != cxl_topology_host); +up_read (&topology_host_sem); +}static ssize_t devtype_show (struct device *dev,struct device_attribute *attr,char *buf)@@ -362,17 +409,16 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,/** * devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy -* @host: host device for devm operations * @uport: "physical" device implementing this upstream port * @component_reg_phys: (optional) for configurable cxl_port instances * @parent_port: next hop up in the CXL memory decode hierarchy */ -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *uport,resource_size_t component_reg_phys,struct cxl_port *parent_port) { +struct device *dev, *host;struct cxl_port *port; -struct device *dev;int rc; port =cxl_port_alloc (uport, component_reg_phys, parent_port);@@ -391,7 +437,12 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,if (rc) goto err; + host =get_cxl_topology_host (); +if (!host) +return ERR_PTR (-ENODEV ); + rc =devm_add_action_or_reset (host, unregister_port, port); +put_cxl_topology_host (host);if (rc)return ERR_PTR (rc);drivers/cxl/cxl.h +4
-1@@ -152,6 +152,9 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20 +int cxl_register_topology_host (struct device *host); +void cxl_unregister_topology_host (struct device *host); +/* * cxl_decoder flags that define the type of memory / devices this * decoder supports as well as configuration lock status See "CXL 2.0 @@ -279,7 +282,7 @@ struct cxl_dport { };struct cxl_port *to_cxl_port (struct device *dev); -struct cxl_port *devm_cxl_add_port (struct device *host,struct device *uport, +struct cxl_port *devm_cxl_add_port (struct device *uport,resource_size_t component_reg_phys,struct cxl_port *parent_port);
Re: [PATCH 14/23] cxl: Introduce topology host registration
In reply to Ben WidawskyQuoted 10 lines · Ben Widawsky
The description of the CXL topology will be conveyed by a platform specific entity that is expected to be a singleton. For ACPI based systems, this is ACPI0017. When the topology host goes away, which as of now can only be triggered by module unload, it is desirable to have the entire topology cleaned up. Regular devm unwinding handles most situations already, but what's missing is the ability to teardown the root port. Since the root port is platform specific, the core needs a set of APIs to allow platform specific drivers to register their root ports. With that, all the automatic teardown can occur.
Wait, no, that was one of the original motivations, but then we discussed here [1] that devm teardown of a topology can happen naturally / hierarchically.
[1]: https://lore.kernel.org/r/CAPcyv4ikVFFqyfH2zLhBVJ28N1_gufGHd2gVbP2h+Rv2cZEpeA@mail.gmail.com
No, the reason for the cxl_topology_host is as a constraint for when CXL.mem connectivity can be verified from root to endpoint. Given that endpoints can attach at any point in time relative to when the root arrives CXL.mem connectivity needs to be revalidated at every topology host arrival / depart event.
Re: [PATCH 14/23] cxl: Introduce topology host registration
Quoted 16 lines · Dan Williams
Quoted 10 lines · Ben Widawsky
The description of the CXL topology will be conveyed by a platform specific entity that is expected to be a singleton. For ACPI based systems, this is ACPI0017. When the topology host goes away, which as of now can only be triggered by module unload, it is desirable to have the entire topology cleaned up. Regular devm unwinding handles most situations already, but what's missing is the ability to teardown the root port. Since the root port is platform specific, the core needs a set of APIs to allow platform specific drivers to register their root ports. With that, all the automatic teardown can occur.
Wait, no, that was one of the original motivations, but then we discussed here [1] that devm teardown of a topology can happen naturally / hierarchically.
Quoted 7 lines
[1]: https://lore.kernel.org/r/CAPcyv4ikVFFqyfH2zLhBVJ28N1_gufGHd2gVbP2h+Rv2cZEpeA@mail.gmail.com
No, the reason for the
cxl_topology_hostis as a constraint for when CXL.mem connectivity can be verified from root to endpoint. Given that endpoints can attach at any point in time relative to when the root arrives CXL.mem connectivity needs to be revalidated at every topology host arrival / depart event.
Oops. I forgot to update the commit message, I will take what you wrote with slight modification.
Dan Carpenter
build botRe: [PATCH 14/23] cxl: Introduce topology host registration
In reply to Ben WidawskyCC: kbuild-all(a)lists.01.org In-Reply-To: <[email protected]> References: <[email protected]> TO: Ben Widawsky <[email protected]>…
CC: kbuild-all(a)lists.01.org
In-Reply-To: <[email protected]>
References: <[email protected]>
TO: Ben Widawsky <[email protected]>
TO: linux-cxl(a)vger.kernel.org
TO: linux-pci(a)vger.kernel.org
CC: Ben Widawsky <[email protected]>
CC: Alison Schofield <[email protected]>
CC: Dan Williams <[email protected]>
CC: Ira Weiny <[email protected]>
CC: Jonathan Cameron <[email protected]>
CC: Vishal Verma <[email protected]>
Hi Ben,
I love your patch! Perhaps something to improve:
[auto build test WARNING on 53989fad1286e652ea3655ae3367ba698da8d2ff]
url: https://github.com/0day-ci/linux/commits/Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 base: 53989fad1286e652ea3655ae3367ba698da8d2ff
:::::: branch date: 6 days ago
:::::: commit date: 6 days ago
config: x86_64-randconfig-m001-20211118 (https://download.01.org/0day-ci/archive/20211126/202111260523.BAvGTRJR-lkp(a)intel.com/config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
If you fix the issue, kindly add following tag as appropriate
smatch warnings:drivers/cxl/acpi.c:399 cxl_acpi_probe() error: uninitialized symbol 'root_port'.
vim +/root_port +399 drivers/cxl/acpi.c
6b4661f8037e4f Ben Widawsky 2021-11-19 382
4812be97c015bd Dan Williams 2021-06-09 383 static int cxl_acpi_probe(struct platform_device *pdev)
4812be97c015bd Dan Williams 2021-06-09 384 {
3b94ce7b7bc1b4 Dan Williams 2021-06-09 385 int rc;
4812be97c015bd Dan Williams 2021-06-09 386 struct cxl_port *root_port;
4812be97c015bd Dan Williams 2021-06-09 387 struct device *host = &pdev->dev;
7d4b5ca2e2cb5d Dan Williams 2021-06-09 388 struct acpi_device *adev = ACPI_COMPANION(host);
f4ce1f766f1ebf Dan Williams 2021-10-29 389 struct cxl_cfmws_context ctx;
4812be97c015bd Dan Williams 2021-06-09 390
6b4661f8037e4f Ben Widawsky 2021-11-19 391 rc = cxl_register_topology_host(host);
6b4661f8037e4f Ben Widawsky 2021-11-19 392 if (rc)
6b4661f8037e4f Ben Widawsky 2021-11-19 393 return rc;
6b4661f8037e4f Ben Widawsky 2021-11-19 394
6b4661f8037e4f Ben Widawsky 2021-11-19 395 rc = devm_add_action_or_reset(host, clear_topology_host, host);
6b4661f8037e4f Ben Widawsky 2021-11-19 396 if (rc)
6b4661f8037e4f Ben Widawsky 2021-11-19 397 return rc;
6b4661f8037e4f Ben Widawsky 2021-11-19 398
6b4661f8037e4f Ben Widawsky 2021-11-19 @399 root_port = devm_cxl_add_port(host, CXL_RESOURCE_NONE, root_port);
4812be97c015bd Dan Williams 2021-06-09 400 if (IS_ERR(root_port))
4812be97c015bd Dan Williams 2021-06-09 401 return PTR_ERR(root_port);
4812be97c015bd Dan Williams 2021-06-09 402 dev_dbg(host, "add: %s\n", dev_name(&root_port->dev));
4812be97c015bd Dan Williams 2021-06-09 403
3b94ce7b7bc1b4 Dan Williams 2021-06-09 404 rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
7d4b5ca2e2cb5d Dan Williams 2021-06-09 405 add_host_bridge_dport);
f4ce1f766f1ebf Dan Williams 2021-10-29 406 if (rc < 0)
f4ce1f766f1ebf Dan Williams 2021-10-29 407 return rc;
3b94ce7b7bc1b4 Dan Williams 2021-06-09 408
f4ce1f766f1ebf Dan Williams 2021-10-29 409 ctx = (struct cxl_cfmws_context) {
f4ce1f766f1ebf Dan Williams 2021-10-29 410 .dev = host,
f4ce1f766f1ebf Dan Williams 2021-10-29 411 .root_port = root_port,
f4ce1f766f1ebf Dan Williams 2021-10-29 412 };
f4ce1f766f1ebf Dan Williams 2021-10-29 413 acpi_table_parse_cedt(ACPI_CEDT_TYPE_CFMWS, cxl_parse_cfmws, &ctx);
3e23d17ce1980c Alison Schofield 2021-06-17 414
3b94ce7b7bc1b4 Dan Williams 2021-06-09 415 /*
3b94ce7b7bc1b4 Dan Williams 2021-06-09 416 * Root level scanned with host-bridge as dports, now scan host-bridges
3b94ce7b7bc1b4 Dan Williams 2021-06-09 417 * for their role as CXL uports to their CXL-capable PCIe Root Ports.
3b94ce7b7bc1b4 Dan Williams 2021-06-09 418 */
8fdcb1704f61a8 Dan Williams 2021-06-15 419 rc = bus_for_each_dev(adev->dev.bus, NULL, root_port,
3b94ce7b7bc1b4 Dan Williams 2021-06-09 420 add_host_bridge_uport);
f4ce1f766f1ebf Dan Williams 2021-10-29 421 if (rc < 0)
f4ce1f766f1ebf Dan Williams 2021-10-29 422 return rc;
8fdcb1704f61a8 Dan Williams 2021-06-15 423
8fdcb1704f61a8 Dan Williams 2021-06-15 424 if (IS_ENABLED(CONFIG_CXL_PMEM))
8fdcb1704f61a8 Dan Williams 2021-06-15 425 rc = device_for_each_child(&root_port->dev, root_port,
8fdcb1704f61a8 Dan Williams 2021-06-15 426 add_root_nvdimm_bridge);
8fdcb1704f61a8 Dan Williams 2021-06-15 427 if (rc < 0)
8fdcb1704f61a8 Dan Williams 2021-06-15 428 return rc;
f4ce1f766f1ebf Dan Williams 2021-10-29 429
8fdcb1704f61a8 Dan Williams 2021-06-15 430 return 0;
4812be97c015bd Dan Williams 2021-06-09 431 }
4812be97c015bd Dan Williams 2021-06-09 432 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
PATCH 15/23 cxl/core: Store global list of root ports
In reply to Ben WidawskyCXL root ports (the downstream port to a host bridge) are to be enumerated by a platform specific driver. In the case of ACPI compliant systems, this is like the cxl_acpi driver. Root ports are the first CXL spec defined component that can be "found" by that platform specific driver.
By storing a list of these root ports components in lower levels of the topology (switches and endpoints), have a mechanism to walk up their device hierarchy to find an enumerated root port. This will be necessary for region programming.
---
Dan points out in review this is possible to do without a new global list. While I agree, I was unable to get it working in a reasonable mount of time. Will punt on that for now.
--- drivers/cxl/acpi.c | 4 ++-- drivers/cxl/core/bus.c | 34 +++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 5 ++++- tools/testing/cxl/mock_acpi.c | 4 ++-- 4 files changed, 41 insertions(+), 6 deletions(-)
drivers/cxl/acpi.c +2 -2
@@ -159,7 +159,7 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) creg =cxl_reg_block (pdev, &map); port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num, creg); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg, true);if (rc) { ctx->error = rc;return rc;
@@ -341,7 +341,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)return 0 ; } - rc =cxl_add_dport (root_port, match, uid, ctx.chbcr); + rc =cxl_add_dport (root_port, match, uid, ctx.chbcr, false);if (rc) {dev_err (host,"failed to add downstream port: %s\n" ,dev_name (match));
drivers/cxl/core/bus.c +33 -1
@@ -26,6 +26,8 @@static DEFINE_IDA (cxl_port_ida);static DECLARE_RWSEM (topology_host_sem); +static LIST_HEAD (cxl_root_ports); +static DECLARE_RWSEM (root_port_sem);static struct device *cxl_topology_host;
@@ -326,12 +328,31 @@ struct cxl_port *to_cxl_port(struct device *dev) return container_of(dev, struct cxl_port, dev); } +struct cxl_dport *cxl_get_root_dport (struct device *dev) +{ +struct cxl_dport *ret =NULL ; +struct cxl_dport *dport; + +down_read (&root_port_sem); +list_for_each_entry (dport, &cxl_root_ports, root_port_link) { +if (dport->dport == dev) { + ret = dport; +break ; + } + } + +up_read (&root_port_sem); +return ret; +} +EXPORT_SYMBOL_NS_GPL (cxl_get_root_dport,CXL ); +static void unregister_port (void *_port) {struct cxl_port *port = _port;struct cxl_dport *dport;device_lock (&port->dev); +down_read (&root_port_sem);list_for_each_entry (dport, &port->dports, list) {char link_name[CXL_TARGET_STRLEN ];
@@ -339,7 +360,10 @@ static void unregister_port(void *_port) dport->port_id) >= CXL_TARGET_STRLEN)continue ;sysfs_remove_link (&port->dev.kobj, link_name); + +list_del_init (&dport->root_port_link); } +up_read (&root_port_sem);device_unlock (&port->dev);device_unregister (&port->dev); }
@@ -493,12 +517,13 @@ static int add_dport(struct cxl_port *port, struct cxl_dport *new)* @dport_dev: firmware or PCI device representing the dport * @port_id: identifier for this dport in a decoder's target list * @component_reg_phys: optional location of CXL component registers +* @root_port: is this a root port (hostbridge downstream) * * Note that all allocations and links are undone by cxl_port deletion * and release. */ int cxl_add_dport (struct cxl_port *port,struct device *dport_dev,int port_id, -resource_size_t component_reg_phys) +resource_size_t component_reg_phys,bool root_port) {char link_name[CXL_TARGET_STRLEN ];struct cxl_dport *dport;
@@ -513,6 +538,7 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,return -ENOMEM ;INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->root_port_link); dport->dport =get_device (dport_dev); dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;
@@ -526,6 +552,12 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,if (rc) goto err; +if (root_port) { +down_write (&root_port_sem); +list_add_tail (&dport->root_port_link, &cxl_root_ports); +up_write (&root_port_sem); + } +return 0 ; err:cxl_dport_release (dport);
drivers/cxl/cxl.h +4 -1
@@ -272,6 +272,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @root_port_link: node for global list of root ports */ struct cxl_dport {struct device *dport;
@@ -279,6 +280,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head root_port_link; };struct cxl_port *to_cxl_port (struct device *dev);
@@ -287,7 +289,8 @@ struct cxl_port *devm_cxl_add_port(struct device *uport, struct cxl_port *parent_port);int cxl_add_dport (struct cxl_port *port,struct device *dport,int port_id, -resource_size_t component_reg_phys); +resource_size_t component_reg_phys,bool root_port); +struct cxl_dport *cxl_get_root_dport (struct device *dev);struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);
tools/testing/cxl/mock_acpi.c +2 -2
@@ -57,7 +57,7 @@ static int match_add_root_port(struct pci_dev *pdev, void *data)/* TODO walk DVSEC to find component register base */ port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);
@@ -78,7 +78,7 @@ static int mock_add_root_port(struct platform_device *pdev, void *data)struct device *dev = ctx->dev;int rc; - rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);
Signature · 1 line
2.34.0
Re: [PATCH 15/23] cxl/core: Store global list of root ports
Quoted 11 lines · On Fri, 19 Nov 2021 16:02:42 -0800 Ben Widawsky <[email protected]> wrote:
CXL root ports (the downstream port to a host bridge) are to be enumerated by a platform specific driver. In the case of ACPI compliant systems, this is like the
cxl_acpidriver. Root ports are the first CXL spec defined component that can be "found" by that platform specific driver.By storing a list of these root ports components in lower levels of the topology (switches and endpoints), have a mechanism to walk up their device hierarchy to find an enumerated root port. This will be necessary for region programming.
Quoted 6 lines
Signed-off-by Ben Widawsky <[email protected]>---
Dan points out in review this is possible to do without a new global list. While I agree, I was unable to get it working in a reasonable mount of time. Will punt on that for now.
This has made me curious. Is this a punt it for v1, or a punt it for longer term and maybe revisit later?
What you have looks like it should work fine to me.
Quoted 173 lines
--- drivers/cxl/acpi.c | 4 ++-- drivers/cxl/core/bus.c | 34 +++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 5 ++++- tools/testing/cxl/mock_acpi.c | 4 ++-- 4 files changed, 41 insertions(+), 6 deletions(-)drivers/cxl/acpi.c +2
-2@@ -159,7 +159,7 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) creg =cxl_reg_block (pdev, &map); port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num, creg); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg, true);if (rc) { ctx->error = rc;return rc;@@ -341,7 +341,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)return 0 ; } - rc =cxl_add_dport (root_port, match, uid, ctx.chbcr); + rc =cxl_add_dport (root_port, match, uid, ctx.chbcr, false);if (rc) {dev_err (host,"failed to add downstream port: %s\n" ,dev_name (match));drivers/cxl/core/bus.c +33
-1@@ -26,6 +26,8 @@static DEFINE_IDA (cxl_port_ida);static DECLARE_RWSEM (topology_host_sem); +static LIST_HEAD (cxl_root_ports); +static DECLARE_RWSEM (root_port_sem);static struct device *cxl_topology_host;@@ -326,12 +328,31 @@ struct cxl_port *to_cxl_port(struct device *dev) return container_of(dev, struct cxl_port, dev); } +struct cxl_dport *cxl_get_root_dport (struct device *dev) +{ +struct cxl_dport *ret =NULL ; +struct cxl_dport *dport; + +down_read (&root_port_sem); +list_for_each_entry (dport, &cxl_root_ports, root_port_link) { +if (dport->dport == dev) { + ret = dport; +break ; + } + } + +up_read (&root_port_sem); +return ret; +} +EXPORT_SYMBOL_NS_GPL (cxl_get_root_dport,CXL ); +static void unregister_port (void *_port) {struct cxl_port *port = _port;struct cxl_dport *dport;device_lock (&port->dev); +down_read (&root_port_sem);list_for_each_entry (dport, &port->dports, list) {char link_name[CXL_TARGET_STRLEN ];@@ -339,7 +360,10 @@ static void unregister_port(void *_port) dport->port_id) >= CXL_TARGET_STRLEN)continue ;sysfs_remove_link (&port->dev.kobj, link_name); + +list_del_init (&dport->root_port_link); } +up_read (&root_port_sem);device_unlock (&port->dev);device_unregister (&port->dev); }@@ -493,12 +517,13 @@ static int add_dport(struct cxl_port *port, struct cxl_dport *new)* @dport_dev: firmware or PCI device representing the dport * @port_id: identifier for this dport in a decoder's target list * @component_reg_phys: optional location of CXL component registers +* @root_port: is this a root port (hostbridge downstream) * * Note that all allocations and links are undone by cxl_port deletion * and release. */ int cxl_add_dport (struct cxl_port *port,struct device *dport_dev,int port_id, -resource_size_t component_reg_phys) +resource_size_t component_reg_phys,bool root_port) {char link_name[CXL_TARGET_STRLEN ];struct cxl_dport *dport;@@ -513,6 +538,7 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,return -ENOMEM ;INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->root_port_link); dport->dport =get_device (dport_dev); dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;@@ -526,6 +552,12 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,if (rc) goto err; +if (root_port) { +down_write (&root_port_sem); +list_add_tail (&dport->root_port_link, &cxl_root_ports); +up_write (&root_port_sem); + } +return 0 ; err:cxl_dport_release (dport);drivers/cxl/cxl.h +4
-1@@ -272,6 +272,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @root_port_link: node for global list of root ports */ struct cxl_dport {struct device *dport;@@ -279,6 +280,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head root_port_link; };struct cxl_port *to_cxl_port (struct device *dev);@@ -287,7 +289,8 @@ struct cxl_port *devm_cxl_add_port(struct device *uport, struct cxl_port *parent_port);int cxl_add_dport (struct cxl_port *port,struct device *dport,int port_id, -resource_size_t component_reg_phys); +resource_size_t component_reg_phys,bool root_port); +struct cxl_dport *cxl_get_root_dport (struct device *dev);struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);tools/testing/cxl/mock_acpi.c +2
-2@@ -57,7 +57,7 @@ static int match_add_root_port(struct pci_dev *pdev, void *data)/* TODO walk DVSEC to find component register base */ port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);@@ -78,7 +78,7 @@ static int mock_add_root_port(struct platform_device *pdev, void *data)struct device *dev = ctx->dev;int rc; - rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);
Re: [PATCH 15/23] cxl/core: Store global list of root ports
Quoted 21 lines · Jonathan Cameron
Quoted 11 lines · On Fri, 19 Nov 2021 16:02:42 -0800 Ben Widawsky <[email protected]> wrote:
CXL root ports (the downstream port to a host bridge) are to be enumerated by a platform specific driver. In the case of ACPI compliant systems, this is like the
cxl_acpidriver. Root ports are the first CXL spec defined component that can be "found" by that platform specific driver.By storing a list of these root ports components in lower levels of the topology (switches and endpoints), have a mechanism to walk up their device hierarchy to find an enumerated root port. This will be necessary for region programming.
Quoted 6 lines
Signed-off-by Ben Widawsky <[email protected]>---
Dan points out in review this is possible to do without a new global list. While I agree, I was unable to get it working in a reasonable mount of time. Will punt on that for now.
Quoted 5 lines
This has made me curious. Is this a punt it for v1, or a punt it for longer term and maybe revisit later?
What you have looks like it should work fine to me.
Dan said he was going to take a crack at it. I'll leave it up to him whether he wants to make it happen before merge. Either way is fine by me.
Quoted 175 lines
Quoted 174 lines
--- drivers/cxl/acpi.c | 4 ++-- drivers/cxl/core/bus.c | 34 +++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 5 ++++- tools/testing/cxl/mock_acpi.c | 4 ++-- 4 files changed, 41 insertions(+), 6 deletions(-)drivers/cxl/acpi.c +2
-2@@ -159,7 +159,7 @@ __mock int match_add_root_ports(struct pci_dev *pdev, void *data) creg =cxl_reg_block (pdev, &map); port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num, creg); + rc =cxl_add_dport (port, &pdev->dev, port_num, creg, true);if (rc) { ctx->error = rc;return rc;@@ -341,7 +341,7 @@ static int add_host_bridge_dport(struct device *match, void *arg)return 0 ; } - rc =cxl_add_dport (root_port, match, uid, ctx.chbcr); + rc =cxl_add_dport (root_port, match, uid, ctx.chbcr, false);if (rc) {dev_err (host,"failed to add downstream port: %s\n" ,dev_name (match));drivers/cxl/core/bus.c +33
-1@@ -26,6 +26,8 @@static DEFINE_IDA (cxl_port_ida);static DECLARE_RWSEM (topology_host_sem); +static LIST_HEAD (cxl_root_ports); +static DECLARE_RWSEM (root_port_sem);static struct device *cxl_topology_host;@@ -326,12 +328,31 @@ struct cxl_port *to_cxl_port(struct device *dev) return container_of(dev, struct cxl_port, dev); } +struct cxl_dport *cxl_get_root_dport (struct device *dev) +{ +struct cxl_dport *ret =NULL ; +struct cxl_dport *dport; + +down_read (&root_port_sem); +list_for_each_entry (dport, &cxl_root_ports, root_port_link) { +if (dport->dport == dev) { + ret = dport; +break ; + } + } + +up_read (&root_port_sem); +return ret; +} +EXPORT_SYMBOL_NS_GPL (cxl_get_root_dport,CXL ); +static void unregister_port (void *_port) {struct cxl_port *port = _port;struct cxl_dport *dport;device_lock (&port->dev); +down_read (&root_port_sem);list_for_each_entry (dport, &port->dports, list) {char link_name[CXL_TARGET_STRLEN ];@@ -339,7 +360,10 @@ static void unregister_port(void *_port) dport->port_id) >= CXL_TARGET_STRLEN)continue ;sysfs_remove_link (&port->dev.kobj, link_name); + +list_del_init (&dport->root_port_link); } +up_read (&root_port_sem);device_unlock (&port->dev);device_unregister (&port->dev); }@@ -493,12 +517,13 @@ static int add_dport(struct cxl_port *port, struct cxl_dport *new)* @dport_dev: firmware or PCI device representing the dport * @port_id: identifier for this dport in a decoder's target list * @component_reg_phys: optional location of CXL component registers +* @root_port: is this a root port (hostbridge downstream) * * Note that all allocations and links are undone by cxl_port deletion * and release. */ int cxl_add_dport (struct cxl_port *port,struct device *dport_dev,int port_id, -resource_size_t component_reg_phys) +resource_size_t component_reg_phys,bool root_port) {char link_name[CXL_TARGET_STRLEN ];struct cxl_dport *dport;@@ -513,6 +538,7 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,return -ENOMEM ;INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->root_port_link); dport->dport =get_device (dport_dev); dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;@@ -526,6 +552,12 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,if (rc) goto err; +if (root_port) { +down_write (&root_port_sem); +list_add_tail (&dport->root_port_link, &cxl_root_ports); +up_write (&root_port_sem); + } +return 0 ; err:cxl_dport_release (dport);drivers/cxl/cxl.h +4
-1@@ -272,6 +272,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @root_port_link: node for global list of root ports */ struct cxl_dport {struct device *dport;@@ -279,6 +280,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head root_port_link; };struct cxl_port *to_cxl_port (struct device *dev);@@ -287,7 +289,8 @@ struct cxl_port *devm_cxl_add_port(struct device *uport, struct cxl_port *parent_port);int cxl_add_dport (struct cxl_port *port,struct device *dport,int port_id, -resource_size_t component_reg_phys); +resource_size_t component_reg_phys,bool root_port); +struct cxl_dport *cxl_get_root_dport (struct device *dev);struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);tools/testing/cxl/mock_acpi.c +2
-2@@ -57,7 +57,7 @@ static int match_add_root_port(struct pci_dev *pdev, void *data)/* TODO walk DVSEC to find component register base */ port_num =FIELD_GET (PCI_EXP_LNKCAP_PN , lnkcap); - rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, port_num,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);@@ -78,7 +78,7 @@ static int mock_add_root_port(struct platform_device *pdev, void *data)struct device *dev = ctx->dev;int rc; - rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE ); + rc =cxl_add_dport (port, &pdev->dev, pdev->id,CXL_RESOURCE_NONE , true);if (rc) {dev_err (dev,"failed to add dport: %s (%d)\n" ,dev_name (&pdev->dev), rc);
PATCH 16/23 cxl/pci: Cache device DVSEC offset
In reply to Ben WidawskyThe PCIe device DVSEC, defined in the CXL 2.0 spec, 8.1.3 is required to be implemented by CXL 2.0 endpoint devices. Since the information contained within this DVSEC will be critically important for region configuration, it makes sense to find the value early.
Since this DVSEC is not strictly required for mailbox functionality, failure to find this information does not result in the driver failing to bind.
--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 7 +++++++ 2 files changed, 9 insertions(+)
drivers/cxl/cxlmem.h +2 -0
@@ -98,6 +98,7 @@ struct cxl_mbox_cmd {* * @dev: The device associated with this CXL state * @regs: Parsed register blocks +* @device_dvsec: Offset to the PCIe device DVSEC * @payload_size: Size of space for payload * (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register) * @lsa_size: Size of Label Storage Area
@@ -125,6 +126,7 @@ struct cxl_dev_state {struct device *dev;struct cxl_regs regs; +int device_dvsec;size_t payload_size;size_t lsa_size;
drivers/cxl/pci.c +7 -0
@@ -474,6 +474,13 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (IS_ERR (cxlds))return PTR_ERR (cxlds); + cxlds->device_dvsec =pci_find_dvsec_capability (pdev, +PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!cxlds->device_dvsec) +dev_warn (&pdev->dev, +"Device DVSEC not present. Expect limited functionality.\n" ); + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)return rc;
Signature · 1 line
2.34.0
Re: [PATCH 16/23] cxl/pci: Cache device DVSEC offset
Quoted 8 lines · On Fri, 19 Nov 2021 16:02:43 -0800 Ben Widawsky <[email protected]> wrote:
The PCIe device DVSEC, defined in the CXL 2.0 spec, 8.1.3 is required to be implemented by CXL 2.0 endpoint devices. Since the information contained within this DVSEC will be critically important for region configuration, it makes sense to find the value early.
Since this DVSEC is not strictly required for mailbox functionality, failure to find this information does not result in the driver failing to bind.
That feels like a path we are going to forget to test sometime in the future. Given it's a specification requirement, I'd treat it as an error and make our lives easier going forwards!
Otherwise looks good to me.
Quoted 45 lines
Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 7 +++++++ 2 files changed, 9 insertions(+)drivers/cxl/cxlmem.h +2
-0@@ -98,6 +98,7 @@ struct cxl_mbox_cmd {* * @dev: The device associated with this CXL state * @regs: Parsed register blocks +* @device_dvsec: Offset to the PCIe device DVSEC * @payload_size: Size of space for payload * (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register) * @lsa_size: Size of Label Storage Area @@ -125,6 +126,7 @@ struct cxl_dev_state {struct device *dev;struct cxl_regs regs; +int device_dvsec;size_t payload_size;size_t lsa_size;drivers/cxl/pci.c +7
-0@@ -474,6 +474,13 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (IS_ERR (cxlds))return PTR_ERR (cxlds); + cxlds->device_dvsec =pci_find_dvsec_capability (pdev, +PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!cxlds->device_dvsec) +dev_warn (&pdev->dev, +"Device DVSEC not present. Expect limited functionality.\n" ); + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)return rc;
Re: [PATCH 16/23] cxl/pci: Cache device DVSEC offset
Quoted 12 lines · Jonathan Cameron
Quoted 8 lines · On Fri, 19 Nov 2021 16:02:43 -0800 Ben Widawsky <[email protected]> wrote:
The PCIe device DVSEC, defined in the CXL 2.0 spec, 8.1.3 is required to be implemented by CXL 2.0 endpoint devices. Since the information contained within this DVSEC will be critically important for region configuration, it makes sense to find the value early.
Since this DVSEC is not strictly required for mailbox functionality, failure to find this information does not result in the driver failing to bind.
Quoted 6 lines
That feels like a path we are going to forget to test sometime in the future. Given it's a specification requirement, I'd treat it as an error and make our lives easier going forwards!
Otherwise looks good to me.
Agreed. This is silly. Basically nothing will work if the device dvsec can't be found. I don't remember what I was thinking...
Quoted 46 lines
Quoted 45 lines
Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 7 +++++++ 2 files changed, 9 insertions(+)drivers/cxl/cxlmem.h +2
-0@@ -98,6 +98,7 @@ struct cxl_mbox_cmd {* * @dev: The device associated with this CXL state * @regs: Parsed register blocks +* @device_dvsec: Offset to the PCIe device DVSEC * @payload_size: Size of space for payload * (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register) * @lsa_size: Size of Label Storage Area @@ -125,6 +126,7 @@ struct cxl_dev_state {struct device *dev;struct cxl_regs regs; +int device_dvsec;size_t payload_size;size_t lsa_size;drivers/cxl/pci.c +7
-0@@ -474,6 +474,13 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (IS_ERR (cxlds))return PTR_ERR (cxlds); + cxlds->device_dvsec =pci_find_dvsec_capability (pdev, +PCI_DVSEC_VENDOR_ID_CXL , +CXL_DVSEC_PCIE_DEVICE ); +if (!cxlds->device_dvsec) +dev_warn (&pdev->dev, +"Device DVSEC not present. Expect limited functionality.\n" ); + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)return rc;
PATCH 17/23 cxl: Cache and pass DVSEC ranges
In reply to Ben WidawskyCXL 1.1 specification provided a mechanism for mapping an address space of a CXL device. That functionality is known as a "range" and can be programmed through PCIe DVSEC. In addition to this, the specification defines an active bit which a device will expose through the same DVSEC to notify system software that memory is initialized and ready.
While CXL 2.0 introduces a more powerful mechanism called HDM decoders that are controlled by MMIO behind a PCIe BAR, the spec does allow the 1.1 style mapping to still be present. In such a case, when the CXL driver takes over, if it were to enable HDM decoding and there was an actively used range, things would likely blow up, in particular if it wasn't an identical mapping.
This patch caches the relevant information which the cxl_mem driver will need to make the proper decision and passes it along.
--- drivers/cxl/cxlmem.h | 19 +++++++ drivers/cxl/pci.c | 126 +++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/pci.h | 13 +++++ 3 files changed, 158 insertions(+)
drivers/cxl/cxlmem.h +19 -0
@@ -89,6 +89,22 @@ struct cxl_mbox_cmd {*/ #define CXL_CAPACITY_MULTIPLIER SZ_256M +/** +* struct cxl_endpoint_dvsec_info - Cached DVSEC info +* @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE +* @ranges: Number of HDM ranges this device contains. +* @range.base: cached value of the range base in the DVSEC, PCIE_DEVICE +* @range.size: cached value of the range size in the DVSEC, PCIE_DEVICE +*/ +struct cxl_endpoint_dvsec_info { +bool mem_enabled; +int ranges; +struct { +u64 base; +u64 size; + } range[2 ]; +}; +/** * struct cxl_dev_state - The driver device state *
@@ -117,6 +133,7 @@ struct cxl_mbox_cmd {* @active_persistent_bytes: sum of hard + soft persistent * @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset +* @info: Cached DVSEC information about the device. * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
@@ -147,6 +164,8 @@ struct cxl_dev_state {u64 next_volatile_bytes;u64 next_persistent_bytes; +struct cxl_endpoint_dvsec_info *info; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };
drivers/cxl/pci.c +126 -0
140 lines not on this page · raw message
drivers/cxl/pci.h +13 -0
@@ -15,6 +15,19 @@/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ #define CXL_DVSEC_PCIE_DEVICE 0 +#define CXL_DVSEC_PCIE_DEVICE_CAP_OFFSET 0xA +#define CXL_DVSEC_PCIE_DEVICE_MEM_CAPABLE BIT(2) +#define CXL_DVSEC_PCIE_DEVICE_HDM_COUNT_MASK GENMASK(5, 4) +#define CXL_DVSEC_PCIE_DEVICE_CTRL_OFFSET 0xC +#define CXL_DVSEC_PCIE_DEVICE_MEM_ENABLE BIT(2) +#define CXL_DVSEC_PCIE_DEVICE_RANGE_SIZE_HIGH_OFFSET (i) 0x18 + (i * 0x10) +#define CXL_DVSEC_PCIE_DEVICE_RANGE_SIZE_LOW_OFFSET (i) 0x1C + (i * 0x10) +#define CXL_DVSEC_PCIE_DEVICE_MEM_INFO_VALID BIT(0) +#define CXL_DVSEC_PCIE_DEVICE_MEM_ACTIVE BIT(1) +#define CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK GENMASK(31, 28) +#define CXL_DVSEC_PCIE_DEVICE_RANGE_BASE_HIGH_OFFSET (i) 0x20 + (i * 0x10) +#define CXL_DVSEC_PCIE_DEVICE_RANGE_BASE_LOW_OFFSET (i) 0x24 + (i * 0x10) +#define CXL_DVSEC_PCIE_DEVICE_MEM_BASE_LOW_MASK GENMASK(31, 28)/* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ #define CXL_DVSEC_FUNCTION_MAP 2
Signature · 1 line
2.34.0
kernel test robot
build botRe: [PATCH 17/23] cxl: Cache and pass DVSEC ranges
I love your patch! Perhaps something to improve:
Hi Ben,
I love your patch! Perhaps something to improve:
[auto build test WARNING on 53989fad1286e652ea3655ae3367ba698da8d2ff]
url: https://github.com/0day-ci/linux/commits/Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 base: 53989fad1286e652ea3655ae3367ba698da8d2ff config: riscv-buildonly-randconfig-r001-20211119 (attached as .config)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/0day-ci/linux/commit/cfdf51e15fc8229a494ee59d05bc7459ab5eecd8
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 git checkout cfdf51e15fc8229a494ee59d05bc7459ab5eecd8
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=riscv
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> drivers/cxl/pci.c:469:7: warning: variable 'size' set but not used [-Wunused-but-set-variable]
u64 size;
^1 warning generated.
vim +/size +469 drivers/cxl/pci.c
454
455 #define CDPD(cxlds, which) \
456 cxlds->device_dvsec + CXL_DVSEC_PCIE_DEVICE_##which##_OFFSET
457
458 #define CDPDR(cxlds, which, sorb, lohi) \
459 cxlds->device_dvsec + \
460 CXL_DVSEC_PCIE_DEVICE_RANGE_##sorb##_##lohi##_OFFSET(which)
461
462 static int wait_for_valid(struct cxl_dev_state *cxlds)
463 {
464 struct pci_dev *pdev = to_pci_dev(cxlds->dev);
465 const unsigned long timeout = jiffies + HZ;
466 bool valid;
467
468 do {
> 469 u64 size;
470 u32 temp;
471 int rc;
472
473 rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH),
474 &temp);
475 if (rc)
476 return -ENXIO;
477 size = (u64)temp << 32;
478
479 rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, LOW),
480 &temp);
481 if (rc)
482 return -ENXIO;
483 size |= temp & CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK;
484
485 /*
486 * Memory_Info_Valid: When set, indicates that the CXL Range 1
487 * Size high and Size Low registers are valid. Must be set
488 * within 1 second of deassertion of reset to CXL device.
489 */
490 valid = FIELD_GET(CXL_DVSEC_PCIE_DEVICE_MEM_INFO_VALID, temp);
491 if (valid)
492 break;
493 cpu_relax();
494 } while (!time_after(jiffies, timeout));
495
496 return valid ? 0 : -ETIMEDOUT;
497 }
498
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
Re: [PATCH 17/23] cxl: Cache and pass DVSEC ranges
In reply to Ben WidawskyQuoted 13 lines · On Fri, 19 Nov 2021 16:02:44 -0800 Ben Widawsky <[email protected]> wrote:
CXL 1.1 specification provided a mechanism for mapping an address space of a CXL device. That functionality is known as a "range" and can be programmed through PCIe DVSEC. In addition to this, the specification defines an active bit which a device will expose through the same DVSEC to notify system software that memory is initialized and ready.
While CXL 2.0 introduces a more powerful mechanism called HDM decoders that are controlled by MMIO behind a PCIe BAR, the spec does allow the 1.1 style mapping to still be present. In such a case, when the CXL driver takes over, if it were to enable HDM decoding and there was an actively used range, things would likely blow up, in particular if it wasn't an identical mapping.
Quoted 4 lines
This patch caches the relevant information which the
cxl_memdriver will need to make the proper decision and passes it along.Signed-off-by Ben Widawsky <[email protected]>
0-day spotted issues in same code as me. See below.
This is another case where I'd treat failure as fatal. Anything that fails is either dead, or non spec compliant so don't bother loading the driver if that happens. Fewer paths to test etc...
Quoted 55 lines
--- drivers/cxl/cxlmem.h | 19 +++++++ drivers/cxl/pci.c | 126 +++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/pci.h | 13 +++++ 3 files changed, 158 insertions(+)drivers/cxl/cxlmem.h +19
-0@@ -89,6 +89,22 @@ struct cxl_mbox_cmd {*/ #define CXL_CAPACITY_MULTIPLIER SZ_256M +/** +* struct cxl_endpoint_dvsec_info - Cached DVSEC info +* @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE +* @ranges: Number of HDM ranges this device contains. +* @range.base: cached value of the range base in the DVSEC, PCIE_DEVICE +* @range.size: cached value of the range size in the DVSEC, PCIE_DEVICE +*/ +struct cxl_endpoint_dvsec_info { +bool mem_enabled; +int ranges; +struct { +u64 base; +u64 size; + } range[2 ]; +}; +/** * struct cxl_dev_state - The driver device state * @@ -117,6 +133,7 @@ struct cxl_mbox_cmd {* @active_persistent_bytes: sum of hard + soft persistent * @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset +* @info: Cached DVSEC information about the device. * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -147,6 +164,8 @@ struct cxl_dev_state {u64 next_volatile_bytes;u64 next_persistent_bytes; +struct cxl_endpoint_dvsec_info *info; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0
drivers/cxl/pci.c static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, · line 456
Quoted 5 lines
drivers/cxl/pci.c +2
-0@@ -452,8 +452,126 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,return rc; } +#define CDPD (cxlds, which) \ + cxlds->device_dvsec + CXL_DVSEC_PCIE_DEVICE_##which##_OFFSET
My usual grumble :) I personally find macros like this a bit of a pain when reviewing. I'd really like to see things spelled out in the code so I can immediately see what register we are dealing with even if it does seem rather repetitive in the code.
Quoted 26 lines
drivers/cxl/pci.c +26
-0+ +#define CDPDR (cxlds, which, sorb, lohi) \ + cxlds->device_dvsec + \ + CXL_DVSEC_PCIE_DEVICE_RANGE_##sorb##_##lohi##_OFFSET(which) + +static int wait_for_valid (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +const unsigned long timeout = jiffies +HZ ; +bool valid; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc) + return -ENXIO; + size = (u64)temp << 32; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, LOW), + &temp); + if (rc) + return -ENXIO;
drivers/cxl/pci.c static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, · line 492
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + +/* +* Memory_Info_Valid: When set, indicates that the CXL Range 1 +* Size high and Size Low registers are valid. Must be set +* within 1 second of deassertion of reset to CXL device. +*/ + valid =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_INFO_VALID , temp); +if (valid) +break ;
I think there is a race here. What if you read the high part, get garbage and then read the low part which is now valid...
Swap this around so you read this one first and it will be fine.
However given as 0-day pointed out, size isn't used, this is fine anyway (subject to removing the pointless code).
Quoted 100 lines
drivers/cxl/pci.c +88
-0+cpu_relax (); + }while (!time_after (jiffies, timeout)); + +return valid ?0 : -ETIMEDOUT ; +} + +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +struct cxl_endpoint_dvsec_info *info; +int hdm_count, rc, i; +u16 cap, ctrl; + + rc =pci_read_config_word (pdev,CDPD (cxlds,CAP ), &cap); +if (rc) +return ERR_PTR (-ENXIO ); + rc =pci_read_config_word (pdev,CDPD (cxlds,CTRL ), &ctrl); +if (rc) +return ERR_PTR (-ENXIO ); + +if (!(cap &CXL_DVSEC_PCIE_DEVICE_MEM_CAPABLE )) +return ERR_PTR (-ENODEV ); + +/* +* It is not allowed by spec for MEM.capable to be set and have 0 HDM +* decoders. Therefore, the device is not considered CXL.mem capable. +*/ + hdm_count =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_HDM_COUNT_MASK , cap); +if (!hdm_count || hdm_count >2 ) +return ERR_PTR (-EINVAL ); + + rc =wait_for_valid (cxlds); +if (rc) +return ERR_PTR (rc); + + info =devm_kzalloc (cxlds->dev,sizeof (*info),GFP_KERNEL ); +if (!info) +return ERR_PTR (-ENOMEM ); + + info->mem_enabled =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ENABLE , ctrl); + +for (i =0 ; i < hdm_count; i++) { +u64 base, size; +u32 temp; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,SIZE ,HIGH ), + &temp); +if (rc) +continue ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,SIZE ,LOW ), + &temp); +if (rc) +continue ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,BASE ,HIGH ), + &temp); +if (rc) +continue ; + base = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,BASE ,LOW ), + &temp); +if (rc) +continue ; + base |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_BASE_LOW_MASK ; + + info->range[i].base = base; + info->range[i].size = size; + info->ranges++; + } + +return info; +} + +#undef CDPDR +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) { +struct cxl_endpoint_dvsec_info *info;struct cxl_register_map map;struct cxl_memdev *cxlmd;struct cxl_dev_state *cxlds;@@ -505,6 +623,14 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; + info =dvsec_ranges (cxlds); +if (IS_ERR (info)) +dev_err (&pdev->dev, +"Failed to get DVSEC range information (%ld)\n" , +PTR_ERR (info)); +else + cxlds->info = info; + cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Re: [PATCH 17/23] cxl: Cache and pass DVSEC ranges
Quoted 19 lines · Jonathan Cameron
Quoted 16 lines · On Fri, 19 Nov 2021 16:02:44 -0800 Ben Widawsky <[email protected]> wrote:
CXL 1.1 specification provided a mechanism for mapping an address space of a CXL device. That functionality is known as a "range" and can be programmed through PCIe DVSEC. In addition to this, the specification defines an active bit which a device will expose through the same DVSEC to notify system software that memory is initialized and ready.
While CXL 2.0 introduces a more powerful mechanism called HDM decoders that are controlled by MMIO behind a PCIe BAR, the spec does allow the 1.1 style mapping to still be present. In such a case, when the CXL driver takes over, if it were to enable HDM decoding and there was an actively used range, things would likely blow up, in particular if it wasn't an identical mapping.
This patch caches the relevant information which the
cxl_memdriver will need to make the proper decision and passes it along.
Quoted 7 lines
Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>0-day spotted issues in same code as me. See below.
This is another case where I'd treat failure as fatal. Anything that fails is either dead, or non spec compliant so don't bother loading the driver if that happens. Fewer paths to test etc...
I disagree about treating failure as fatal. The failure here only forbids using the memory on the device. I can envision firmware bugs or the like where these things might fail, but the mailbox interfaces still work perfectly fine, or at least fine enough to upgrade the firmware and try again.
Dan and I had talked about a modparam (or perhaps sysfs on some higher level, like CXL bus) to control the length of the timeout, so that one doesn't have to wait forever to deal with this usage. Figured we'd cross that bridge when we came to it.
Quoted 59 lines
Quoted 59 lines
--- drivers/cxl/cxlmem.h | 19 +++++++ drivers/cxl/pci.c | 126 +++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/pci.h | 13 +++++ 3 files changed, 158 insertions(+)drivers/cxl/cxlmem.h +19
-0@@ -89,6 +89,22 @@ struct cxl_mbox_cmd {*/ #define CXL_CAPACITY_MULTIPLIER SZ_256M +/** +* struct cxl_endpoint_dvsec_info - Cached DVSEC info +* @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE +* @ranges: Number of HDM ranges this device contains. +* @range.base: cached value of the range base in the DVSEC, PCIE_DEVICE +* @range.size: cached value of the range size in the DVSEC, PCIE_DEVICE +*/ +struct cxl_endpoint_dvsec_info { +bool mem_enabled; +int ranges; +struct { +u64 base; +u64 size; + } range[2 ]; +}; +/** * struct cxl_dev_state - The driver device state * @@ -117,6 +133,7 @@ struct cxl_mbox_cmd {* @active_persistent_bytes: sum of hard + soft persistent * @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset +* @info: Cached DVSEC information about the device. * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -147,6 +164,8 @@ struct cxl_dev_state {u64 next_volatile_bytes;u64 next_persistent_bytes; +struct cxl_endpoint_dvsec_info *info; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -452,8 +452,126 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,return rc; }
Quoted 7 lines
drivers/cxl/pci.c static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, · line 456
Quoted 2 lines
drivers/cxl/pci.c +2
-0+#define CDPD (cxlds, which) \ + cxlds->device_dvsec + CXL_DVSEC_PCIE_DEVICE_##which##_OFFSETMy usual grumble :) I personally find macros like this a bit of a pain when reviewing. I'd really like to see things spelled out in the code so I can immediately see what register we are dealing with even if it does seem rather repetitive in the code.
I understand. It's this or have every line look strange because of character limits. I agree that whomever writes this stuff can reason it out better, and that makes it harder on reviewers. I don't mind changing it back, I'd like hear any other opinions before I do though.
Quoted 41 lines
Quoted 27 lines
drivers/cxl/pci.c +26
-0+ +#define CDPDR (cxlds, which, sorb, lohi) \ + cxlds->device_dvsec + \ + CXL_DVSEC_PCIE_DEVICE_RANGE_##sorb##_##lohi##_OFFSET(which) + +static int wait_for_valid (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +const unsigned long timeout = jiffies +HZ ; +bool valid; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc) + return -ENXIO; + size = (u64)temp << 32; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, LOW), + &temp); + if (rc) + return -ENXIO;drivers/cxl/pci.c static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, · line 492
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + +/* +* Memory_Info_Valid: When set, indicates that the CXL Range 1 +* Size high and Size Low registers are valid. Must be set +* within 1 second of deassertion of reset to CXL device. +*/ + valid =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_INFO_VALID , temp); +if (valid) +break ;I think there is a race here. What if you read the high part, get garbage and then read the low part which is now valid...
Quoted 5 lines
Swap this around so you read this one first and it will be fine.
However given as 0-day pointed out, size isn't used, this is fine anyway (subject to removing the pointless code).
Yes. I've fixed that. And you're right, size is potentially invalid in this case even if it found the valid bit.
Quoted 102 lines
Quoted 100 lines
drivers/cxl/pci.c +88
-0+cpu_relax (); + }while (!time_after (jiffies, timeout)); + +return valid ?0 : -ETIMEDOUT ; +} + +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) +{ +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +struct cxl_endpoint_dvsec_info *info; +int hdm_count, rc, i; +u16 cap, ctrl; + + rc =pci_read_config_word (pdev,CDPD (cxlds,CAP ), &cap); +if (rc) +return ERR_PTR (-ENXIO ); + rc =pci_read_config_word (pdev,CDPD (cxlds,CTRL ), &ctrl); +if (rc) +return ERR_PTR (-ENXIO ); + +if (!(cap &CXL_DVSEC_PCIE_DEVICE_MEM_CAPABLE )) +return ERR_PTR (-ENODEV ); + +/* +* It is not allowed by spec for MEM.capable to be set and have 0 HDM +* decoders. Therefore, the device is not considered CXL.mem capable. +*/ + hdm_count =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_HDM_COUNT_MASK , cap); +if (!hdm_count || hdm_count >2 ) +return ERR_PTR (-EINVAL ); + + rc =wait_for_valid (cxlds); +if (rc) +return ERR_PTR (rc); + + info =devm_kzalloc (cxlds->dev,sizeof (*info),GFP_KERNEL ); +if (!info) +return ERR_PTR (-ENOMEM ); + + info->mem_enabled =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ENABLE , ctrl); + +for (i =0 ; i < hdm_count; i++) { +u64 base, size; +u32 temp; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,SIZE ,HIGH ), + &temp); +if (rc) +continue ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,SIZE ,LOW ), + &temp); +if (rc) +continue ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,BASE ,HIGH ), + &temp); +if (rc) +continue ; + base = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds, i,BASE ,LOW ), + &temp); +if (rc) +continue ; + base |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_BASE_LOW_MASK ; + + info->range[i].base = base; + info->range[i].size = size; + info->ranges++; + } + +return info; +} + +#undef CDPDR +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) { +struct cxl_endpoint_dvsec_info *info;struct cxl_register_map map;struct cxl_memdev *cxlmd;struct cxl_dev_state *cxlds;@@ -505,6 +623,14 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; + info =dvsec_ranges (cxlds); +if (IS_ERR (info)) +dev_err (&pdev->dev, +"Failed to get DVSEC range information (%ld)\n" , +PTR_ERR (info)); +else + cxlds->info = info; + cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Re: [PATCH 17/23] cxl: Cache and pass DVSEC ranges
In reply to Ben WidawskyQuoted 35 lines · On Fri, 19 Nov 2021 16:02:44 -0800 Ben Widawsky <[email protected]> wrote:
CXL 1.1 specification provided a mechanism for mapping an address space of a CXL device. That functionality is known as a "range" and can be programmed through PCIe DVSEC. In addition to this, the specification defines an active bit which a device will expose through the same DVSEC to notify system software that memory is initialized and ready.
While CXL 2.0 introduces a more powerful mechanism called HDM decoders that are controlled by MMIO behind a PCIe BAR, the spec does allow the 1.1 style mapping to still be present. In such a case, when the CXL driver takes over, if it were to enable HDM decoding and there was an actively used range, things would likely blow up, in particular if it wasn't an identical mapping.
This patch caches the relevant information which the
cxl_memdriver will need to make the proper decision and passes it along.Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/cxlmem.h | 19 +++++++ drivers/cxl/pci.c | 126 +++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/pci.h | 13 +++++ 3 files changed, 158 insertions(+)drivers/cxl/cxlmem.h +4
-0@@ -89,6 +89,22 @@ struct cxl_mbox_cmd {*/ #define CXL_CAPACITY_MULTIPLIER SZ_256M +/** +* struct cxl_endpoint_dvsec_info - Cached DVSEC info +* @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE +* @ranges: Number of HDM ranges this device contains.
drivers/cxl/cxlmem.h struct cxl_mbox_cmd { · line 105
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+* @range.base: cached value of the range base in the DVSEC, PCIE_DEVICE +* @range.size: cached value of the range size in the DVSEC, PCIE_DEVICE +*/ +struct cxl_endpoint_dvsec_info { +bool mem_enabled; +int ranges; +struct { +u64 base; +u64 size; + } range[2 ];
kernel-doc wants documentation for range as well.
Quoted 1 line
drivers/cxl/cxlmem.h +1
-0+};
PATCH 18/23 cxl/pci: Implement wait for media active
In reply to Ben WidawskyCXL 2.0 8.1.3.8.2 defines "Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1. Memory_Active_Timeout of deassertion of reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Unfortunately, Memory_Active can take quite a long time depending on media size (up to 256s per 2.0 spec). Since the cxl_pci driver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.
---
This patch did not exist in RFCv2
--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)
drivers/cxl/cxlmem.h +1 -0
@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds); };enum cxl_opcode {
drivers/cxl/pci.c +56 -0
68 lines not on this page · raw message
Signature · 1 line
2.34.0
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 3 lines · On Fri, 19 Nov 2021 16:02:45 -0800 Ben Widawsky <[email protected]> wrote:
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of
Range 1?
Quoted 10 lines
reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Unfortunately,
Memory_Activecan take quite a long time depending on media size (up to 256s per 2.0 spec). Since thecxl_pcidriver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.Signed-off-by Ben Widawsky <[email protected]>
Same thing about size not being used...
Quoted 51 lines
---
This patch did not exist in RFCv2--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)drivers/cxl/cxlmem.h +1
-0@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds); };enum cxl_opcode {drivers/cxl/pci.c +24
-0@@ -496,6 +496,60 @@ static int wait_for_valid(struct cxl_dev_state *cxlds)return valid ?0 : -ETIMEDOUT ; } +/* +* Implements Figure 43 of the CXL Type 3 Memory Device Software Guide. Waits a +* full 256s no matter what the device reports. +*/ +static int wait_for_media_ready (struct cxl_dev_state *cxlds) +{ +const unsigned long timeout = jiffies + (256 *HZ ); +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +u64 md_status; +bool active; +int rc; + + rc =wait_for_valid (cxlds); +if (rc) +return rc; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc)
drivers/cxl/pci.c static int wait_for_valid(struct cxl_dev_state *cxlds) · line 532
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return -ENXIO ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds,0 ,SIZE ,LOW ), + &temp); +if (rc) +return -ENXIO ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + active =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ACTIVE , temp);
Only need to read the register to get active for this particular functionality.
Quoted 32 lines
drivers/cxl/pci.c +22
-0+if (active) +break ; +cpu_relax (); +mdelay (100 ); + }while (!time_after (jiffies, timeout)); + +if (!active) +return -ETIMEDOUT ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (!CXLMDEV_READY (md_status)) +return -EIO ; + +return 0 ; +} +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) {struct pci_dev *pdev =to_pci_dev (cxlds->dev);@@ -598,6 +652,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (!cxlds->device_dvsec)dev_warn (&pdev->dev,"Device DVSEC not present. Expect limited functionality.\n" ); +else + cxlds->wait_media_ready = wait_for_media_ready; rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 9 lines · Jonathan Cameron
Quoted 3 lines · On Fri, 19 Nov 2021 16:02:45 -0800 Ben Widawsky <[email protected]> wrote:
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion ofRange 1?
Not my numbering... It's the first DVSEC range.
Quoted 4 lines
Quoted 4 lines
reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Quoted 9 lines
Quoted 6 lines
Unfortunately,
Memory_Activecan take quite a long time depending on media size (up to 256s per 2.0 spec). Since thecxl_pcidriver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.Signed-off-by Ben Widawsky <[email protected]>Same thing about size not being used...
Yep, got it.
Quoted 97 lines
Quoted 51 lines
---
This patch did not exist in RFCv2--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)drivers/cxl/cxlmem.h +1
-0@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds); };enum cxl_opcode {drivers/cxl/pci.c +24
-0@@ -496,6 +496,60 @@ static int wait_for_valid(struct cxl_dev_state *cxlds)return valid ?0 : -ETIMEDOUT ; } +/* +* Implements Figure 43 of the CXL Type 3 Memory Device Software Guide. Waits a +* full 256s no matter what the device reports. +*/ +static int wait_for_media_ready (struct cxl_dev_state *cxlds) +{ +const unsigned long timeout = jiffies + (256 *HZ ); +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +u64 md_status; +bool active; +int rc; + + rc =wait_for_valid (cxlds); +if (rc) +return rc; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc)drivers/cxl/pci.c static int wait_for_valid(struct cxl_dev_state *cxlds) · line 532
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return -ENXIO ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds,0 ,SIZE ,LOW ), + &temp); +if (rc) +return -ENXIO ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + active =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ACTIVE , temp);Only need to read the register to get active for this particular functionality.
Quoted 32 lines
drivers/cxl/pci.c +22
-0+if (active) +break ; +cpu_relax (); +mdelay (100 ); + }while (!time_after (jiffies, timeout)); + +if (!active) +return -ETIMEDOUT ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (!CXLMDEV_READY (md_status)) +return -EIO ; + +return 0 ; +} +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) {struct pci_dev *pdev =to_pci_dev (cxlds->dev);@@ -598,6 +652,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (!cxlds->device_dvsec)dev_warn (&pdev->dev,"Device DVSEC not present. Expect limited functionality.\n" ); +else + cxlds->wait_media_ready = wait_for_media_ready; rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 4 lines · On Mon, 22 Nov 2021 14:57:51 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron
On Fri, 19 Nov 2021 16:02:45 -0800
Ben Widawsky <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion ofRange 1?
Not my numbering... It's the first DVSEC range.
Ah, got it. Maybe Range 1: Memory Active timeout ?
Quoted 114 lines
Quoted 5 lines
Quoted 5 lines
reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Quoted 9 lines
Quoted 6 lines
Unfortunately,
Memory_Activecan take quite a long time depending on media size (up to 256s per 2.0 spec). Since thecxl_pcidriver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.Signed-off-by Ben Widawsky <[email protected]>Same thing about size not being used...
Yep, got it.
Quoted 97 lines
Quoted 51 lines
---
This patch did not exist in RFCv2--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)drivers/cxl/cxlmem.h +1
-0@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds); };enum cxl_opcode {drivers/cxl/pci.c +24
-0@@ -496,6 +496,60 @@ static int wait_for_valid(struct cxl_dev_state *cxlds)return valid ?0 : -ETIMEDOUT ; } +/* +* Implements Figure 43 of the CXL Type 3 Memory Device Software Guide. Waits a +* full 256s no matter what the device reports. +*/ +static int wait_for_media_ready (struct cxl_dev_state *cxlds) +{ +const unsigned long timeout = jiffies + (256 *HZ ); +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +u64 md_status; +bool active; +int rc; + + rc =wait_for_valid (cxlds); +if (rc) +return rc; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc)drivers/cxl/pci.c static int wait_for_valid(struct cxl_dev_state *cxlds) · line 532
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return -ENXIO ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds,0 ,SIZE ,LOW ), + &temp); +if (rc) +return -ENXIO ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + active =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ACTIVE , temp);Only need to read the register to get active for this particular functionality.
Quoted 32 lines
drivers/cxl/pci.c +22
-0+if (active) +break ; +cpu_relax (); +mdelay (100 ); + }while (!time_after (jiffies, timeout)); + +if (!active) +return -ETIMEDOUT ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (!CXLMDEV_READY (md_status)) +return -EIO ; + +return 0 ; +} +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) {struct pci_dev *pdev =to_pci_dev (cxlds->dev);@@ -598,6 +652,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (!cxlds->device_dvsec)dev_warn (&pdev->dev,"Device DVSEC not present. Expect limited functionality.\n" ); +else + cxlds->wait_media_ready = wait_for_media_ready; rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Mon, 22 Nov 2021 14:57:51 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 7 lines · Jonathan Cameron
Quoted 3 lines · On Fri, 19 Nov 2021 16:02:45 -0800 Ben Widawsky <[email protected]> wrote:
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Range 1?
Not my numbering... It's the first DVSEC range.
Ah, got it. Maybe Range 1: Memory Active timeout ?
I can, but this is just quoted from the spec. Would this be better:
The CXL Type 3 Memory Device Software Guide (Revision 1.0) describes the need to check media active before using HDM. CXL 2.0 8.1.3.8.2 states:
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1. Memory_Active_Timeout of deassertion of reset to CXL device if CXL.mem HwInit Mode=1
Unfortunately, Memory_Active can take quite a long time depending on media size (up to 256s per 2.0 spec). Since the cxl_pci driver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.
Quoted 116 lines
Quoted 115 lines
Quoted 6 lines
Quoted 6 lines
reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Quoted 9 lines
Quoted 6 lines
Unfortunately,
Memory_Activecan take quite a long time depending on media size (up to 256s per 2.0 spec). Since thecxl_pcidriver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.Signed-off-by Ben Widawsky <[email protected]>Same thing about size not being used...
Yep, got it.
Quoted 97 lines
Quoted 51 lines
---
This patch did not exist in RFCv2--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)drivers/cxl/cxlmem.h +1
-0@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds); };enum cxl_opcode {drivers/cxl/pci.c +24
-0@@ -496,6 +496,60 @@ static int wait_for_valid(struct cxl_dev_state *cxlds)return valid ?0 : -ETIMEDOUT ; } +/* +* Implements Figure 43 of the CXL Type 3 Memory Device Software Guide. Waits a +* full 256s no matter what the device reports. +*/ +static int wait_for_media_ready (struct cxl_dev_state *cxlds) +{ +const unsigned long timeout = jiffies + (256 *HZ ); +struct pci_dev *pdev =to_pci_dev (cxlds->dev); +u64 md_status; +bool active; +int rc; + + rc =wait_for_valid (cxlds); +if (rc) +return rc; + + do { + u64 size; + u32 temp; + int rc; + + rc = pci_read_config_dword(pdev, CDPDR(cxlds, 0, SIZE, HIGH), + &temp); + if (rc)drivers/cxl/pci.c static int wait_for_valid(struct cxl_dev_state *cxlds) · line 532
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return -ENXIO ; + size = (u64 )temp <<32 ; + + rc =pci_read_config_dword (pdev,CDPDR (cxlds,0 ,SIZE ,LOW ), + &temp); +if (rc) +return -ENXIO ; + size |= temp &CXL_DVSEC_PCIE_DEVICE_MEM_SIZE_LOW_MASK ; + + active =FIELD_GET (CXL_DVSEC_PCIE_DEVICE_MEM_ACTIVE , temp);Only need to read the register to get active for this particular functionality.
Quoted 32 lines
drivers/cxl/pci.c +22
-0+if (active) +break ; +cpu_relax (); +mdelay (100 ); + }while (!time_after (jiffies, timeout)); + +if (!active) +return -ETIMEDOUT ; + + rc =check_device_status (cxlds); +if (rc) +return rc; + + md_status =readq (cxlds->regs.memdev +CXLMDEV_STATUS_OFFSET ); +if (!CXLMDEV_READY (md_status)) +return -EIO ; + +return 0 ; +} +static struct cxl_endpoint_dvsec_info *dvsec_ranges (struct cxl_dev_state *cxlds) {struct pci_dev *pdev =to_pci_dev (cxlds->dev);@@ -598,6 +652,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (!cxlds->device_dvsec)dev_warn (&pdev->dev,"Device DVSEC not present. Expect limited functionality.\n" ); +else + cxlds->wait_media_ready = wait_for_media_ready; rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_MEMDEV , &map);if (rc)
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 17 lines · Ben Widawsky
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Mon, 22 Nov 2021 14:57:51 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron
On Fri, 19 Nov 2021 16:02:45 -0800
Ben Widawsky <[email protected]> wrote:Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of...
Ah, got it. Maybe Range 1: Memory Active timeout ?I can, but this is just quoted from the spec. Would this be better:
Quoted 7 lines
The CXL Type 3 Memory Device Software Guide (Revision 1.0) describes the need to check media active before using HDM. CXL 2.0 8.1.3.8.2 states:
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of reset to CXL device if CXL.mem HwInit Mode=1
That is some weird wording. I stumbled over that, too. I like the quote format better, but I still don't know what it means.
That last piece ("Memory_Active_Timeout of deassertion ...") purports to be a sentence, but is not.
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
Quoted 26 lines · Bjorn Helgaas
Quoted 17 lines · Ben Widawsky
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Mon, 22 Nov 2021 14:57:51 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron
On Fri, 19 Nov 2021 16:02:45 -0800
Ben Widawsky <[email protected]> wrote:Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of...
Ah, got it. Maybe Range 1: Memory Active timeout ?I can, but this is just quoted from the spec. Would this be better:
Quoted 7 lines
The CXL Type 3 Memory Device Software Guide (Revision 1.0) describes the need to check media active before using HDM. CXL 2.0 8.1.3.8.2 states:
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion of reset to CXL device if CXL.mem HwInit Mode=1
Quoted 5 lines
That is some weird wording. I stumbled over that, too. I like the quote format better, but I still don't know what it means.
That last piece ("
Memory_Active_Timeoutof deassertion ...") purports to be a sentence, but is not.
I've reported this to the person locally who drives spec changes. Since it's now confused multiple people, I will rewrite it with my interpretation if people think that is more optimal.
"Memory Active bit must be set within Memory_Active_Timeout amount of time after reset."
Re: [PATCH 18/23] cxl/pci: Implement wait for media active
In reply to Jonathan CameronQuoted 36 lines · On Mon, 22 Nov 2021 17:03:35 +0000 Jonathan Cameron…
Quoted 3 lines · On Fri, 19 Nov 2021 16:02:45 -0800 Ben Widawsky <[email protected]> wrote:
CXL 2.0 8.1.3.8.2 defines "
Memory_Active: When set, indicates that the CXL Range 1 memory is fully initialized and available for software use. Must be set within Range 1.Memory_Active_Timeoutof deassertion ofRange 1?
Quoted 10 lines
reset to CXL device if CXL.mem HwInit Mode=1" The CXL* Type 3 Memory Device Software Guide (Revision 1.0) further describes the need to check this bit before using HDM.
Unfortunately,
Memory_Activecan take quite a long time depending on media size (up to 256s per 2.0 spec). Since thecxl_pcidriver doesn't care about this, a callback is exported as part of driver state for use by drivers that do care.Signed-off-by Ben Widawsky <[email protected]>Same thing about size not being used...
Quoted 14 lines
---
This patch did not exist in RFCv2--- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)drivers/cxl/cxlmem.h +0
-0@@ -167,6 +167,7 @@ struct cxl_dev_state {struct cxl_endpoint_dvsec_info *info;
Quoted 2 lines
drivers/cxl/cxlmem.h struct cxl_dev_state { · line 170
Quoted 2 lines
drivers/cxl/cxlmem.h +1
-0int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int (*wait_media_ready)(struct cxl_dev_state *cxlds);
Missing docs for this.
Jonathan
PATCH 19/23 cxl/pci: Store component register base in cxlds
In reply to Ben WidawskyThe component register base address is enumerated and stored in the cxl_device_state structure for use by the endpoint driver.
Component register base addresses are obtained through PCI mechanisms. As such it makes most sense for the cxl_pci driver to obtain that address. In order to reuse the port driver for enumerating decoder resources for an endpoint, it is desirable to be able to add the endpoint as a port. The endpoint does know of the cxlds and can pull the component register base from there and pass it along to port creation.
---
Changes since RFCv2:
This patch was originally named, "cxl/core: Store component register base for memdevs". It plumbed the component registers through memdev creation. After more work, it became apparent we needed to plumb other stuff from the pci driver, so going forward, cxlds will just be referenced by the cxl_mem driver. This also allows us to ignore the change needed to cxl_test
- Rework patch to store the base in cxlds
- Remove defunct comment (Dan)
--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 11 +++++++++++ 2 files changed, 13 insertions(+)
drivers/cxl/cxlmem.h +2 -0
@@ -134,6 +134,7 @@ struct cxl_endpoint_dvsec_info {* @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset * @info: Cached DVSEC information about the device. +* @component_reg_phys: register base of component registers * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
@@ -165,6 +166,7 @@ struct cxl_dev_state {u64 next_persistent_bytes;struct cxl_endpoint_dvsec_info *info; +resource_size_t component_reg_phys;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd);int (*wait_media_ready)(struct cxl_dev_state *cxlds);
drivers/cxl/pci.c +11 -0
@@ -663,6 +663,17 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +/* +* If the component registers can't be found, the cxl_pci driver may +* still be useful for management functions so don't return an error. +*/ + cxlds->component_reg_phys =CXL_RESOURCE_NONE ; + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) +dev_warn (&cxlmd->dev,"No component registers (%d)\n" , rc); +else + cxlds->component_reg_phys =cxl_reg_block (pdev, &map); + rc =cxl_pci_setup_mailbox (cxlds);if (rc)return rc;
Signature · 1 line
2.34.0
kernel test robot
build botRe: [PATCH 19/23] cxl/pci: Store component register base in cxlds
I love your patch! Perhaps something to improve:
Hi Ben,
I love your patch! Perhaps something to improve:
[auto build test WARNING on 53989fad1286e652ea3655ae3367ba698da8d2ff]
url: https://github.com/0day-ci/linux/commits/Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 base: 53989fad1286e652ea3655ae3367ba698da8d2ff config: riscv-buildonly-randconfig-r001-20211119 (attached as .config)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/0day-ci/linux/commit/3f74c99d751a24a4c12ba76c23b68c2832f805e1
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 git checkout 3f74c99d751a24a4c12ba76c23b68c2832f805e1
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=riscv
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
drivers/cxl/pci.c:469:7: warning: variable 'size' set but not used [-Wunused-but-set-variable]
u64 size;
^
drivers/cxl/pci.c:516:7: warning: variable 'size' set but not used [-Wunused-but-set-variable]
u64 size;
^
>> drivers/cxl/pci.c:673:13: warning: variable 'cxlmd' is uninitialized when used here [-Wuninitialized]
dev_warn(&cxlmd->dev, "No component registers (%d)\n", rc);
^~~~~
include/linux/dev_printk.h:146:49: note: expanded from macro 'dev_warn'
dev_printk_index_wrap(_dev_warn, KERN_WARNING, dev, dev_fmt(fmt), ##__VA_ARGS__)
^~~
include/linux/dev_printk.h:110:11: note: expanded from macro 'dev_printk_index_wrap'
_p_func(dev, fmt, ##__VA_ARGS__); \
^~~
drivers/cxl/pci.c:630:26: note: initialize the variable 'cxlmd' to silence this warning
struct cxl_memdev *cxlmd;
^
= NULL3 warnings generated.
vim +/cxlmd +673 drivers/cxl/pci.c
625
626 static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
627 {
628 struct cxl_endpoint_dvsec_info *info;
629 struct cxl_register_map map;
630 struct cxl_memdev *cxlmd;
631 struct cxl_dev_state *cxlds;
632 int rc;
633
634 /*
635 * Double check the anonymous union trickery in struct cxl_regs
636 * FIXME switch to struct_group()
637 */
638 BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
639 offsetof(struct cxl_regs, device_regs.memdev));
640
641 rc = pcim_enable_device(pdev);
642 if (rc)
643 return rc;
644
645 cxlds = cxl_dev_state_create(&pdev->dev);
646 if (IS_ERR(cxlds))
647 return PTR_ERR(cxlds);
648
649 cxlds->device_dvsec = pci_find_dvsec_capability(pdev,
650 PCI_DVSEC_VENDOR_ID_CXL,
651 CXL_DVSEC_PCIE_DEVICE);
652 if (!cxlds->device_dvsec)
653 dev_warn(&pdev->dev,
654 "Device DVSEC not present. Expect limited functionality.\n");
655 else
656 cxlds->wait_media_ready = wait_for_media_ready;
657
658 rc = cxl_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map);
659 if (rc)
660 return rc;
661
662 rc = cxl_map_regs(cxlds, &map);
663 if (rc)
664 return rc;
665
666 /*
667 * If the component registers can't be found, the cxl_pci driver may
668 * still be useful for management functions so don't return an error.
669 */
670 cxlds->component_reg_phys = CXL_RESOURCE_NONE;
671 rc = cxl_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT, &map);
672 if (rc)
> 673 dev_warn(&cxlmd->dev, "No component registers (%d)\n", rc);
674 else
675 cxlds->component_reg_phys = cxl_reg_block(pdev, &map);
676
677 rc = cxl_pci_setup_mailbox(cxlds);
678 if (rc)
679 return rc;
680
681 rc = cxl_enumerate_cmds(cxlds);
682 if (rc)
683 return rc;
684
685 rc = cxl_dev_state_identify(cxlds);
686 if (rc)
687 return rc;
688
689 rc = cxl_mem_create_range_info(cxlds);
690 if (rc)
691 return rc;
692
693 info = dvsec_ranges(cxlds);
694 if (IS_ERR(info))
695 dev_err(&pdev->dev,
696 "Failed to get DVSEC range information (%ld)\n",
697 PTR_ERR(info));
698 else
699 cxlds->info = info;
700
701 cxlmd = devm_cxl_add_memdev(cxlds);
702 if (IS_ERR(cxlmd))
703 return PTR_ERR(cxlmd);
704
705 if (range_len(&cxlds->pmem_range) && IS_ENABLED(CONFIG_CXL_PMEM))
706 rc = devm_cxl_add_nvdimm(&pdev->dev, cxlmd);
707
708 return rc;
709 }
710
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [PATCH 19/23] cxl/pci: Store component register base in cxlds
In reply to Ben WidawskyQuoted 9 lines · On Fri, 19 Nov 2021 16:02:46 -0800 Ben Widawsky <[email protected]> wrote:
The component register base address is enumerated and stored in the
cxl_device_statestructure for use by the endpoint driver.Component register base addresses are obtained through PCI mechanisms. As such it makes most sense for the
cxl_pcidriver to obtain that address. In order to reuse the port driver for enumerating decoder resources for an endpoint, it is desirable to be able to add the endpoint as a port. The endpoint does know of the cxlds and can pull the component register base from there and pass it along to port creation.
This feels like a lot of explanation in for trivial caching of an address. I'm not sure you need to be that detailed, though I guess it does no real harm.
Another one where I'm unsure why we are muddling on after an error...
Quoted 60 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
This patch was originally named, "cxl/core: Store component register base for memdevs". It plumbed the component registers through memdev creation. After more work, it became apparent we needed to plumb other stuff from the pci driver, so going forward, cxlds will just be referenced by thecxl_memdriver. This also allows us to ignore the change needed tocxl_test- Rework patch to store the base in cxlds
- Remove defunct comment (Dan)--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 11 +++++++++++ 2 files changed, 13 insertions(+)drivers/cxl/cxlmem.h +2
-0@@ -134,6 +134,7 @@ struct cxl_endpoint_dvsec_info {* @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset * @info: Cached DVSEC information about the device. +* @component_reg_phys: register base of component registers * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -165,6 +166,7 @@ struct cxl_dev_state {u64 next_persistent_bytes;struct cxl_endpoint_dvsec_info *info; +resource_size_t component_reg_phys;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd);int (*wait_media_ready)(struct cxl_dev_state *cxlds);drivers/cxl/pci.c +11
-0@@ -663,6 +663,17 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +/* +* If the component registers can't be found, the cxl_pci driver may +* still be useful for management functions so don't return an error. +*/ + cxlds->component_reg_phys =CXL_RESOURCE_NONE ; + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) +dev_warn (&cxlmd->dev,"No component registers (%d)\n" , rc); +else + cxlds->component_reg_phys =cxl_reg_block (pdev, &map); + rc =cxl_pci_setup_mailbox (cxlds);if (rc)return rc;
Re: [PATCH 19/23] cxl/pci: Store component register base in cxlds
Quoted 13 lines · Jonathan Cameron
Quoted 9 lines · On Fri, 19 Nov 2021 16:02:46 -0800 Ben Widawsky <[email protected]> wrote:
The component register base address is enumerated and stored in the
cxl_device_statestructure for use by the endpoint driver.Component register base addresses are obtained through PCI mechanisms. As such it makes most sense for the
cxl_pcidriver to obtain that address. In order to reuse the port driver for enumerating decoder resources for an endpoint, it is desirable to be able to add the endpoint as a port. The endpoint does know of the cxlds and can pull the component register base from there and pass it along to port creation.
Quoted 3 lines
This feels like a lot of explanation in for trivial caching of an address. I'm not sure you need to be that detailed, though I guess it does no real harm.
It is mostly to articulate that cxl_pci is responsible for PCI stuff, and cxl_mem is responsible for CXL.mem stuff, and that's why we didn't just do this work in cxl_mem (which indeed was what I originally did).
Quoted 2 lines
Another one where I'm unsure why we are muddling on after an error...
Same motivation as the other - mailbox can still be used even if the media isn't available.
Quoted 63 lines
Quoted 62 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since RFCv2:
This patch was originally named, "cxl/core: Store component register base for memdevs". It plumbed the component registers through memdev creation. After more work, it became apparent we needed to plumb other stuff from the pci driver, so going forward, cxlds will just be referenced by thecxl_memdriver. This also allows us to ignore the change needed tocxl_test- Rework patch to store the base in cxlds
- Remove defunct comment (Dan)--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 11 +++++++++++ 2 files changed, 13 insertions(+)drivers/cxl/cxlmem.h +2
-0@@ -134,6 +134,7 @@ struct cxl_endpoint_dvsec_info {* @next_volatile_bytes: volatile capacity change pending device reset * @next_persistent_bytes: persistent capacity change pending device reset * @info: Cached DVSEC information about the device. +* @component_reg_phys: register base of component registers * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -165,6 +166,7 @@ struct cxl_dev_state {u64 next_persistent_bytes;struct cxl_endpoint_dvsec_info *info; +resource_size_t component_reg_phys;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd);int (*wait_media_ready)(struct cxl_dev_state *cxlds);drivers/cxl/pci.c +11
-0@@ -663,6 +663,17 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +/* +* If the component registers can't be found, the cxl_pci driver may +* still be useful for management functions so don't return an error. +*/ + cxlds->component_reg_phys =CXL_RESOURCE_NONE ; + rc =cxl_setup_regs (pdev,CXL_REGLOC_RBI_COMPONENT , &map); +if (rc) +dev_warn (&cxlmd->dev,"No component registers (%d)\n" , rc); +else + cxlds->component_reg_phys =cxl_reg_block (pdev, &map); + rc =cxl_pci_setup_mailbox (cxlds);if (rc)return rc;
PATCH 20/23 cxl/port: Introduce a port driver
In reply to Ben WidawskyThe CXL port driver is responsible for managing the decoder resources contained within the port. It will also provide APIs that other drivers will consume for managing these resources.
There are 4 types of ports in a system:
1. Platform port. This is a non-programmable entity. Such a port is named rootX. It is enumerated by cxl_acpi in an ACPI based system.
2. Hostbridge port. This ports register access is defined in a platform specific way (CHBS for ACPI platforms). It has n downstream ports, each of which are known as CXL 2.0 root ports. Once the platform specific mechanism to get the offset to the registers is obtained it operates just like other CXL components. The enumeration of this component is started by cxl_acpi and completed by cxl_port.
3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained within cxl_port.
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started by cxl_pci, and completed by cxl_port.
---
Changes since RFCv2:
- Reword commit message tense (Dan)
- Reword commit message
- Drop SOFTDEP since it's not needed yet (Dan)
- Add CONFIG_CXL_PORT (Dan)
- s/CXL_DECODER_F_EN/CXL_DECODER_F_ENABLE (Dan)
- rename cxl_hdm_decoder_ functions to "to_" (Dan)
- remove useless inline (Dan)
- Check endpoint decoder based on dport list instead of driver id (Dan)
- Use range instead of resource per dependent patch change
- Use clever union packing for target list (Dan)
- Only check NULL from devm_cxl_iomap_block (Dan)
- Properly parent the created cxl decoders
- Move bus rescanning from cxl_acpi to here (Dan)
- Remove references to "CFMWS" in core (Dan)
- Use macro to help keep within 80 character lines
--- .../driver-api/cxl/memory-devices.rst | 5 + drivers/cxl/Kconfig | 22 ++ drivers/cxl/Makefile | 2 + drivers/cxl/core/bus.c | 67 ++++ drivers/cxl/core/regs.c | 6 +- drivers/cxl/cxl.h | 34 +- drivers/cxl/port.c | 323 ++++++++++++++++++ 7 files changed, 450 insertions(+), 9 deletions(-) create mode 100644 drivers/cxl/port.c
Documentation/driver-api/cxl/memory-devices.rst +5 -0
@@ -28,6 +28,11 @@ CXL Memory Device .. kernel-doc:: drivers/cxl/pci.c :internal: +CXL Port +-------- +.. kernel-doc:: drivers/cxl/port.c + :doc: cxl port + CXL Core -------- .. kernel-doc:: drivers/cxl/cxl.h
drivers/cxl/Kconfig +22 -0
@@ -77,4 +77,26 @@ config CXL_PMEM provisioning the persistent memory capacity of CXL memory expanders. If unsure say 'm'. + +config CXL_MEM + tristate "CXL.mem: Memory Devices" + select CXL_PORT + depends on CXL_PCI + default CXL_BUS + help + The CXL.mem protocol allows a device to act as a provider of "System + RAM" and/or "Persistent Memory" that is fully coherent as if the + memory was attached to the typical CPU memory controller. This is + known as HDM "Host-managed Device Memory". + + Say 'y/m' to enable a driver that will attach to CXL.mem devices for + memory expansion and control of HDM. See Chapter 9.13 in the CXL 2.0 + specification for a detailed description of HDM. + + If unsure say 'm'. + + +config CXL_PORT + tristate + endif
drivers/cxl/Makefile +2 -0
@@ -3,7 +3,9 @@ obj-$(CONFIG_CXL_BUS) += core/ obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o +obj-$(CONFIG_CXL_PORT) += cxl_port.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o +cxl_port-y := port.o
drivers/cxl/core/bus.c +67 -0
103 lines not on this page · raw message
drivers/cxl/core/regs.c +3 -3
@@ -159,9 +159,8 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, }EXPORT_SYMBOL_NS_GPL (cxl_probe_device_regs,CXL ); -static void __iomem *devm_cxl_iomap_block (struct device *dev, -resource_size_t addr, -resource_size_t length) +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length) {void __iomem *ret_val;struct resource *res;
@@ -180,6 +179,7 @@ static void __iomem *devm_cxl_iomap_block(struct device *dev,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct pci_dev *pdev, struct cxl_component_regs *regs,
drivers/cxl/cxl.h +28 -6
@@ -17,6 +17,9 @@* (port-driver, region-driver, nvdimm object-drivers... etc). */ +/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ +#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K +/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/ #define CXL_CM_OFFSET 0x1000#define CXL_CM_CAP_HDR_OFFSET 0x0
@@ -36,11 +39,22 @@#define CXL_HDM_DECODER_CAP_OFFSET 0x0#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER0_BASE_LOW_OFFSET 0x10 -#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET 0x14 -#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET 0x18 -#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET 0x1c -#define CXL_HDM_DECODER0_CTRL_OFFSET 0x20 +#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) +#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) +#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE BIT(1) +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET (i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET (i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET (i) (0x20 * (i) + 0x18) +#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (i) (0x20 * (i) + 0x1c) +#define CXL_HDM_DECODER0_CTRL_OFFSET (i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) +#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) +#define CXL_HDM_DECODER0_CTRL_TYPE BIT(12) +#define CXL_HDM_DECODER0_TL_LOW (i) (0x20 * (i) + 0x24) +#define CXL_HDM_DECODER0_TL_HIGH (i) (0x20 * (i) + 0x28)static inline int cxl_hdm_decoder_count (u32 cap_hdr) {
@@ -148,6 +162,8 @@ int cxl_map_device_regs(struct pci_dev *pdev,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20
@@ -165,7 +181,8 @@ void cxl_unregister_topology_host(struct device *host);#define CXL_DECODER_F_TYPE2 BIT(2)#define CXL_DECODER_F_TYPE3 BIT(3)#define CXL_DECODER_F_LOCK BIT(4) -#define CXL_DECODER_F_MASK GENMASK(4, 0) +#define CXL_DECODER_F_ENABLE BIT(5) +#define CXL_DECODER_F_MASK GENMASK(5, 0)enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,
@@ -255,6 +272,8 @@ struct cxl_walk_context {* @dports: cxl_dport instances referenced by decoders * @decoder_ida: allocator for decoder ids * @component_reg_phys: component register capability base address (optional) +* @rescan_work: worker object for bus rescans after port additions +* @data: opaque data with driver specific usage */ struct cxl_port {struct device dev;
@@ -263,6 +282,8 @@ struct cxl_port {struct list_head dports;struct ida decoder_ida;resource_size_t component_reg_phys; +struct work_struct rescan_work; +void *data; };/**
@@ -325,6 +346,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_NVDIMM_BRIDGE 1#define CXL_DEVICE_NVDIMM 2 +#define CXL_DEVICE_PORT 3#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"
drivers/cxl/port.c new file +323 -0
323 lines not on this page · raw message
Signature · 1 line
2.34.0
kernel test robot
build botRe: [PATCH 20/23] cxl/port: Introduce a port driver
I love your patch! Yet something to improve:
Hi Ben,
I love your patch! Yet something to improve:
[auto build test ERROR on 53989fad1286e652ea3655ae3367ba698da8d2ff]
url: https://github.com/0day-ci/linux/commits/Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 base: 53989fad1286e652ea3655ae3367ba698da8d2ff config: alpha-randconfig-r026-20211118 (attached as .config) compiler: alpha-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/8ff43502e84dd4fa1296a131cb0cc82146389db4
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 git checkout 8ff43502e84dd4fa1296a131cb0cc82146389db4
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross ARCH=alpha
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
drivers/cxl/port.c: In function 'get_decoder_size':
>> drivers/cxl/port.c:105:16: error: implicit declaration of function 'ioread64_hi_lo' [-Werror=implicit-function-declaration]
105 | return ioread64_hi_lo(hdm_decoder +
| ^~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/ioread64_hi_lo +105 drivers/cxl/port.c
97
98 static u64 get_decoder_size(void __iomem *hdm_decoder, int n)
99 {
100 u32 ctrl = readl(hdm_decoder + CXL_HDM_DECODER0_CTRL_OFFSET(n));
101
102 if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)
103 return 0;
104
> 105 return ioread64_hi_lo(hdm_decoder + 106 CXL_HDM_DECODER0_SIZE_LOW_OFFSET(n));
107 }
108
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Ben WidawskyI love your patch! Yet something to improve:
Hi Ben,
I love your patch! Yet something to improve:
[auto build test ERROR on 53989fad1286e652ea3655ae3367ba698da8d2ff]
url: https://github.com/0day-ci/linux/commits/Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 base: 53989fad1286e652ea3655ae3367ba698da8d2ff config: arm-randconfig-r034-20211119 (attached as .config)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/0day-ci/linux/commit/8ff43502e84dd4fa1296a131cb0cc82146389db4
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ben-Widawsky/Add-drivers-for-CXL-ports-and-mem-devices/20211120-080513 git checkout 8ff43502e84dd4fa1296a131cb0cc82146389db4
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=arm
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
>> drivers/cxl/port.c:105:9: error: implicit declaration of function 'ioread64_hi_lo' [-Werror,-Wimplicit-function-declaration]
return ioread64_hi_lo(hdm_decoder +
^
drivers/cxl/port.c:191:11: error: implicit declaration of function 'ioread64_hi_lo' [-Werror,-Wimplicit-function-declaration]
base = ioread64_hi_lo(decoderN(BASE_LOW_OFFSET, i));
^2 errors generated.
vim +/ioread64_hi_lo +105 drivers/cxl/port.c
97
98 static u64 get_decoder_size(void __iomem *hdm_decoder, int n)
99 {
100 u32 ctrl = readl(hdm_decoder + CXL_HDM_DECODER0_CTRL_OFFSET(n));
101
102 if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)
103 return 0;
104
> 105 return ioread64_hi_lo(hdm_decoder + 106 CXL_HDM_DECODER0_SIZE_LOW_OFFSET(n));
107 }
108
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Ben WidawskyQuoted 8 lines · On Fri, 19 Nov 2021 16:02:47 -0800 Ben Widawsky <[email protected]> wrote:
The CXL port driver is responsible for managing the decoder resources contained within the port. It will also provide APIs that other drivers will consume for managing these resources.
There are 4 types of ports in a system:
1. Platform port. This is a non-programmable entity. Such a port is named rootX. It is enumerated bycxl_acpiin an ACPI based system.
2. Hostbridge port. This ports register access is defined in a platform
port's
Quoted 15 lines
specific way (CHBS for ACPI platforms). It has n downstream ports, each of which are known as CXL 2.0 root ports. Once the platform specific mechanism to get the offset to the registers is obtained it operates just like other CXL components. The enumeration of this component is started by
cxl_acpiand completed bycxl_port.
3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained withincxl_port.
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started bycxl_pci, and completed bycxl_port.
Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>
A few comments inline including what looks to me memory on the stack which has gone out of scope when it's accessed.
Jonathan
Quoted 489 lines
---
Changes since RFCv2:
- Reword commit message tense (Dan)
- Reword commit message
- Drop SOFTDEP since it's not needed yet (Dan)
- AddCONFIG_CXL_PORT(Dan)
- s/CXL_DECODER_F_EN/CXL_DECODER_F_ENABLE (Dan)
- renamecxl_hdm_decoder_functions to "to_" (Dan)
- remove useless inline (Dan)
- Check endpoint decoder based on dport list instead of driver id (Dan)
- Use range instead of resource per dependent patch change
- Use clever union packing for target list (Dan)
- Only check NULL fromdevm_cxl_iomap_block(Dan)
- Properly parent the created cxl decoders
- Move bus rescanning fromcxl_acpito here (Dan)
- Remove references to "CFMWS" in core (Dan)
- Use macro to help keep within 80 character lines--- .../driver-api/cxl/memory-devices.rst | 5 + drivers/cxl/Kconfig | 22 ++ drivers/cxl/Makefile | 2 + drivers/cxl/core/bus.c | 67 ++++ drivers/cxl/core/regs.c | 6 +- drivers/cxl/cxl.h | 34 +- drivers/cxl/port.c | 323 ++++++++++++++++++ 7 files changed, 450 insertions(+), 9 deletions(-) create mode 100644 drivers/cxl/port.cDocumentation/driver-api/cxl/memory-devices.rst +5
-0@@ -28,6 +28,11 @@ CXL Memory Device .. kernel-doc:: drivers/cxl/pci.c :internal: +CXL Port +-------- +.. kernel-doc:: drivers/cxl/port.c + :doc: cxl port + CXL Core -------- .. kernel-doc:: drivers/cxl/cxl.hdrivers/cxl/Kconfig +22
-0@@ -77,4 +77,26 @@ config CXL_PMEM provisioning the persistent memory capacity of CXL memory expanders. If unsure say 'm'. + +config CXL_MEM + tristate "CXL.mem: Memory Devices" + select CXL_PORT + depends on CXL_PCI + default CXL_BUS + help + The CXL.mem protocol allows a device to act as a provider of "System + RAM" and/or "Persistent Memory" that is fully coherent as if the + memory was attached to the typical CPU memory controller. This is + known as HDM "Host-managed Device Memory". + + Say 'y/m' to enable a driver that will attach to CXL.mem devices for + memory expansion and control of HDM. See Chapter 9.13 in the CXL 2.0 + specification for a detailed description of HDM. + + If unsure say 'm'. + + +config CXL_PORT + tristate + endifdrivers/cxl/Makefile +2
-0@@ -3,7 +3,9 @@ obj-$(CONFIG_CXL_BUS) += core/ obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o +obj-$(CONFIG_CXL_PORT) += cxl_port.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o +cxl_port-y := port.odrivers/cxl/core/bus.c +67
-0@@ -31,6 +31,8 @@ static DECLARE_RWSEM(root_port_sem);static struct device *cxl_topology_host; +static bool is_cxl_decoder (struct device *dev); +int cxl_register_topology_host (struct device *host) {down_write (&topology_host_sem);@@ -75,6 +77,45 @@ static void put_cxl_topology_host(struct device *dev)up_read (&topology_host_sem); } +static int decoder_match (struct device *dev,void *data) +{ +struct resource *theirs = (struct resource *)data; +struct cxl_decoder *cxld; + +if (!is_cxl_decoder (dev)) +return 0 ; + + cxld =to_cxl_decoder (dev); +if (theirs->start <= cxld->decoder_range.start && + theirs->end >= cxld->decoder_range.end) +return 1 ; + +return 0 ; +} + +static struct cxl_decoder *cxl_find_root_decoder (resource_size_t base, +resource_size_t size) +{ +struct cxl_decoder *cxld =NULL ; +struct device *cxldd; +struct device *host; +struct resource res = (struct resource ){ + .start = base, + .end = base + size -1 , + }; + + host =get_cxl_topology_host (); +if (!host) +return NULL ; + + cxldd =device_find_child (host, &res, decoder_match); +if (cxldd) + cxld =to_cxl_decoder (cxldd); + +put_cxl_topology_host (host); +return cxld; +} +static ssize_t devtype_show (struct device *dev,struct device_attribute *attr,char *buf) {@@ -280,6 +321,11 @@ bool is_root_decoder(struct device *dev) }EXPORT_SYMBOL_NS_GPL (is_root_decoder,CXL ); +static bool is_cxl_decoder (struct device *dev) +{ +return dev->type->release == cxl_decoder_release; +} +struct cxl_decoder *to_cxl_decoder (struct device *dev) { if (dev_WARN_ONCE(dev, dev->type->release != cxl_decoder_release,@@ -327,6 +373,7 @@ struct cxl_port *to_cxl_port(struct device *dev)return NULL ; return container_of(dev, struct cxl_port, dev); } +EXPORT_SYMBOL_NS_GPL (to_cxl_port,CXL );struct cxl_dport *cxl_get_root_dport (struct device *dev) {@@ -735,6 +782,24 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_add, CXL);static void cxld_unregister (void *dev) { +struct cxl_decoder *plat_decoder, *cxld =to_cxl_decoder (dev); +resource_size_t base, size; + +if (is_root_decoder (dev)) { +device_unregister (dev); +return ; + } + + base = cxld->decoder_range.start; + size =range_len (&cxld->decoder_range); + +if (size) { + plat_decoder =cxl_find_root_decoder (base, size); +if (plat_decoder) +__release_region (&plat_decoder->platform_res, base, + size); + } +device_unregister (dev); }@@ -789,6 +854,8 @@ static int cxl_device_id(struct device *dev)return CXL_DEVICE_NVDIMM_BRIDGE ;if (dev->type == &cxl_nvdimm_type)return CXL_DEVICE_NVDIMM ; +if (dev->type == &cxl_port_type) +return CXL_DEVICE_PORT ;return 0 ; }drivers/cxl/core/regs.c +3
-3@@ -159,9 +159,8 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, }EXPORT_SYMBOL_NS_GPL (cxl_probe_device_regs,CXL ); -static void __iomem *devm_cxl_iomap_block (struct device *dev, -resource_size_t addr, -resource_size_t length) +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length) {void __iomem *ret_val;struct resource *res;@@ -180,6 +179,7 @@ static void __iomem *devm_cxl_iomap_block(struct device *dev,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct pci_dev *pdev, struct cxl_component_regs *regs,drivers/cxl/cxl.h +28
-6@@ -17,6 +17,9 @@* (port-driver, region-driver, nvdimm object-drivers... etc). */ +/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ +#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K +/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/ #define CXL_CM_OFFSET 0x1000#define CXL_CM_CAP_HDR_OFFSET 0x0@@ -36,11 +39,22 @@#define CXL_HDM_DECODER_CAP_OFFSET 0x0#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER0_BASE_LOW_OFFSET 0x10 -#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET 0x14 -#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET 0x18 -#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET 0x1c -#define CXL_HDM_DECODER0_CTRL_OFFSET 0x20 +#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) +#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) +#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE BIT(1) +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET (i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET (i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET (i) (0x20 * (i) + 0x18) +#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (i) (0x20 * (i) + 0x1c) +#define CXL_HDM_DECODER0_CTRL_OFFSET (i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) +#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) +#define CXL_HDM_DECODER0_CTRL_TYPE BIT(12) +#define CXL_HDM_DECODER0_TL_LOW (i) (0x20 * (i) + 0x24) +#define CXL_HDM_DECODER0_TL_HIGH (i) (0x20 * (i) + 0x28)static inline int cxl_hdm_decoder_count (u32 cap_hdr) {@@ -148,6 +162,8 @@ int cxl_map_device_regs(struct pci_dev *pdev,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20@@ -165,7 +181,8 @@ void cxl_unregister_topology_host(struct device *host);#define CXL_DECODER_F_TYPE2 BIT(2)#define CXL_DECODER_F_TYPE3 BIT(3)#define CXL_DECODER_F_LOCK BIT(4) -#define CXL_DECODER_F_MASK GENMASK(4, 0) +#define CXL_DECODER_F_ENABLE BIT(5) +#define CXL_DECODER_F_MASK GENMASK(5, 0)enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,@@ -255,6 +272,8 @@ struct cxl_walk_context {* @dports: cxl_dport instances referenced by decoders * @decoder_ida: allocator for decoder ids * @component_reg_phys: component register capability base address (optional) +* @rescan_work: worker object for bus rescans after port additions +* @data: opaque data with driver specific usage */ struct cxl_port {struct device dev;@@ -263,6 +282,8 @@ struct cxl_port {struct list_head dports;struct ida decoder_ida;resource_size_t component_reg_phys; +struct work_struct rescan_work; +void *data; };/** @@ -325,6 +346,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_NVDIMM_BRIDGE 1#define CXL_DEVICE_NVDIMM 2 +#define CXL_DEVICE_PORT 3#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"drivers/cxl/port.c new file +169
-0@@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> + +#include "cxlmem.h" + +/** +* DOC: cxl port +* +* The port driver implements the set of functionality needed to allow full +* decoder enumeration and routing. A CXL port is an abstraction of a CXL +* component that implements some amount of CXL decoding of CXL.mem traffic. +* As of the CXL 2.0 spec, this includes: +* +* .. list-table:: CXL Components w/ Ports +* :widths: 25 25 50 +* :header-rows: 1 +* +* * - component +* - upstream +* - downstream +* * - Hostbridge +* - ACPI0016 +* - root port +* * - Switch +* - Switch Upstream Port +* - Switch Downstream Port +* * - Endpoint +* - Endpoint Port +* - N/A +* +* The primary service this driver provides is enumerating HDM decoders and +* presenting APIs to other drivers to utilize the decoders. +*/ + +static struct workqueue_struct *cxl_port_wq; + +struct cxl_port_data { +struct cxl_component_regs regs; + +struct port_caps { +unsigned int count; +unsigned int tc; +unsigned int interleave11_8; +unsigned int interleave14_12; + } caps; +}; + +static inline int to_interleave_granularity (u32 ctrl) +{ +int val =FIELD_GET (CXL_HDM_DECODER0_CTRL_IG_MASK , ctrl); + +return 256 << val; +} + +static inline int to_interleave_ways (u32 ctrl) +{ +int val =FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl); + +return 1 << val; +} + +static void get_caps (struct cxl_port *port,struct cxl_port_data *cpd) +{ +void __iomem *hdm_decoder = cpd->regs.hdm_decoder; +struct port_caps *caps = &cpd->caps; +u32 hdm_cap; + + hdm_cap =readl (hdm_decoder +CXL_HDM_DECODER_CAP_OFFSET ); + + caps->count =cxl_hdm_decoder_count (hdm_cap); + caps->tc =FIELD_GET (CXL_HDM_DECODER_TARGET_COUNT_MASK , hdm_cap); + caps->interleave11_8 = +FIELD_GET (CXL_HDM_DECODER_INTERLEAVE_11_8 , hdm_cap); + caps->interleave14_12 = +FIELD_GET (CXL_HDM_DECODER_INTERLEAVE_14_12 , hdm_cap); +} + +static int map_regs (struct cxl_port *port,void __iomem *crb, +struct cxl_port_data *cpd) +{ +struct cxl_register_map map; +struct cxl_component_reg_map *comp_map = &map.component_map; + +cxl_probe_component_regs (&port->dev, crb, comp_map); +if (!comp_map->hdm_decoder.valid) { +dev_err (&port->dev,"HDM decoder registers invalid\n" ); +return -ENXIO ; + } + + cpd->regs.hdm_decoder = crb + comp_map->hdm_decoder.offset; + +return 0 ; +} + +static u64 get_decoder_size (void __iomem *hdm_decoder,int n) +{ +u32 ctrl =readl (hdm_decoder +CXL_HDM_DECODER0_CTRL_OFFSET (n)); + +if (ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ) +return 0 ; + +return ioread64_hi_lo (hdm_decoder + +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (n)); +} + +static bool is_endpoint_port (struct cxl_port *port) +{ +/* Endpoints can't be ports... yet! */ +return false; +} + +static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" ); +} + +/* Minor layering violation */ +static int dvsec_range_used (struct cxl_port *port) +{ +struct cxl_endpoint_dvsec_info *info; +int i, ret =0 ; + +if (!is_endpoint_port (port)) +return 0 ; + + info = port->data; +for (i =0 ; i < info->ranges; i++) +if (info->range[i].size) + ret |=1 << i; + +return ret; +} + +static int enumerate_hdm_decoders (struct cxl_port *port, +struct cxl_port_data *portdata) +{ +void __iomem *hdm_decoder = portdata->regs.hdm_decoder; +bool global_enable; +u32 global_ctrl; +int i =0 ; + + global_ctrl =readl (hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + global_enable = global_ctrl &CXL_HDM_DECODER_ENABLE ; +if (!global_enable) { + i =dvsec_range_used (port); +if (i) { +dev_err (&port->dev, +"Couldn't add port because device is using DVSEC range registers\n" ); +return -EBUSY ; + } + } + +for (i =0 ; i < portdata->caps.count; i++) { +int rc, target_count = portdata->caps.tc; +struct cxl_decoder *cxld; +int *target_map =NULL ; +u64 size; + +if (is_endpoint_port (port)) + target_count =0 ; + + cxld =cxl_decoder_alloc (port, target_count); +if (IS_ERR (cxld)) { +dev_warn (&port->dev, +"Failed to allocate the decoder\n" );
drivers/cxl/port.c line 179
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+return PTR_ERR (cxld); + } + + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->interleave_ways =1 ; + cxld->interleave_granularity =0 ; + + size =get_decoder_size (hdm_decoder, i); +if (size !=0 ) { +#define decoderN (reg, n) hdm_decoder + CXL_HDM_DECODER0_##reg(n)
Perhaps this block in the if (size != 0) would be more readable if broken out to a utility function?
As normal I'm not keen on the macro magic if we can avoid it.
drivers/cxl/port.c line 187
Quoted 8 lines
drivers/cxl/port.c new file +8
-0+int temp[CXL_DECODER_MAX_INTERLEAVE ]; +u64 base; +u32 ctrl; +int j; +union { +u64 value; +char target_id[8 ]; + } target_list;
I thought we tried to avoid this union usage in kernel because of the whole thing about c compilers being able to do what they like with it...
drivers/cxl/port.c line 189
Quoted 2 lines
drivers/cxl/port.c new file +2
-0+ + target_map = temp;
This is set to a variable that goes out of scope whilst target_map is still in use.
Quoted 19 lines
drivers/cxl/port.c new file +19
-0+ ctrl =readl (decoderN (CTRL_OFFSET , i)); + base =ioread64_hi_lo (decoderN (BASE_LOW_OFFSET , i)); + + cxld->decoder_range = (struct range ){ + .start = base, + .end = base + size -1 + }; + + cxld->flags =CXL_DECODER_F_ENABLE ; + cxld->interleave_ways =to_interleave_ways (ctrl); + cxld->interleave_granularity = +to_interleave_granularity (ctrl); + +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_TYPE , ctrl) ==0 ) + cxld->target_type =CXL_DECODER_ACCELERATOR ; + + target_list.value =ioread64_hi_lo (decoderN (TL_LOW , i)); +for (j =0 ; j < cxld->interleave_ways; j++) + target_map[j] = target_list.target_id[j];
drivers/cxl/port.c line 218
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+#undef decoderN + } + + rc =cxl_decoder_add_locked (cxld, target_map); +if (rc) +put_device (&cxld->dev); +else + rc =cxl_decoder_autoremove (&port->dev, cxld); +if (rc) +dev_err (&port->dev,"Failed to add decoder\n" );
If that fails on the autoremove registration (unlikely) this message will be rather confusing - as the add was fine...
This nest of carrying on when we have an error is getting ever deeper...
Quoted 17 lines
drivers/cxl/port.c new file +17
-0+else +dev_dbg (&cxld->dev,"Added to port %s\n" , +dev_name (&port->dev)); + } + +/* +* Turn on global enable now since DVSEC ranges aren't being used and +* we'll eventually want the decoder enabled. +*/ +if (!global_enable) { +dev_dbg (&port->dev,"Enabling HDM decode\n" ); +writel (global_ctrl |CXL_HDM_DECODER_ENABLE , hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + } + +return 0 ; +} +
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 529 lines · Jonathan Cameron
Quoted 8 lines · On Fri, 19 Nov 2021 16:02:47 -0800 Ben Widawsky <[email protected]> wrote:
The CXL port driver is responsible for managing the decoder resources contained within the port. It will also provide APIs that other drivers will consume for managing these resources.
There are 4 types of ports in a system:
1. Platform port. This is a non-programmable entity. Such a port is named rootX. It is enumerated bycxl_acpiin an ACPI based system.
2. Hostbridge port. This ports register access is defined in a platformport's
Quoted 15 lines
specific way (CHBS for ACPI platforms). It has n downstream ports, each of which are known as CXL 2.0 root ports. Once the platform specific mechanism to get the offset to the registers is obtained it operates just like other CXL components. The enumeration of this component is started by
cxl_acpiand completed bycxl_port.
3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained withincxl_port.
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started bycxl_pci, and completed bycxl_port.Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>A few comments inline including what looks to me memory on the stack which has gone out of scope when it's accessed.
Jonathan
Quoted 494 lines
---
Changes since RFCv2:
- Reword commit message tense (Dan)
- Reword commit message
- Drop SOFTDEP since it's not needed yet (Dan)
- AddCONFIG_CXL_PORT(Dan)
- s/CXL_DECODER_F_EN/CXL_DECODER_F_ENABLE (Dan)
- renamecxl_hdm_decoder_functions to "to_" (Dan)
- remove useless inline (Dan)
- Check endpoint decoder based on dport list instead of driver id (Dan)
- Use range instead of resource per dependent patch change
- Use clever union packing for target list (Dan)
- Only check NULL fromdevm_cxl_iomap_block(Dan)
- Properly parent the created cxl decoders
- Move bus rescanning fromcxl_acpito here (Dan)
- Remove references to "CFMWS" in core (Dan)
- Use macro to help keep within 80 character lines--- .../driver-api/cxl/memory-devices.rst | 5 + drivers/cxl/Kconfig | 22 ++ drivers/cxl/Makefile | 2 + drivers/cxl/core/bus.c | 67 ++++ drivers/cxl/core/regs.c | 6 +- drivers/cxl/cxl.h | 34 +- drivers/cxl/port.c | 323 ++++++++++++++++++ 7 files changed, 450 insertions(+), 9 deletions(-) create mode 100644 drivers/cxl/port.cDocumentation/driver-api/cxl/memory-devices.rst +5
-0@@ -28,6 +28,11 @@ CXL Memory Device .. kernel-doc:: drivers/cxl/pci.c :internal: +CXL Port +-------- +.. kernel-doc:: drivers/cxl/port.c + :doc: cxl port + CXL Core -------- .. kernel-doc:: drivers/cxl/cxl.hdrivers/cxl/Kconfig +22
-0@@ -77,4 +77,26 @@ config CXL_PMEM provisioning the persistent memory capacity of CXL memory expanders. If unsure say 'm'. + +config CXL_MEM + tristate "CXL.mem: Memory Devices" + select CXL_PORT + depends on CXL_PCI + default CXL_BUS + help + The CXL.mem protocol allows a device to act as a provider of "System + RAM" and/or "Persistent Memory" that is fully coherent as if the + memory was attached to the typical CPU memory controller. This is + known as HDM "Host-managed Device Memory". + + Say 'y/m' to enable a driver that will attach to CXL.mem devices for + memory expansion and control of HDM. See Chapter 9.13 in the CXL 2.0 + specification for a detailed description of HDM. + + If unsure say 'm'. + + +config CXL_PORT + tristate + endifdrivers/cxl/Makefile +2
-0@@ -3,7 +3,9 @@ obj-$(CONFIG_CXL_BUS) += core/ obj-$(CONFIG_CXL_PCI) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o +obj-$(CONFIG_CXL_PORT) += cxl_port.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o +cxl_port-y := port.odrivers/cxl/core/bus.c +67
-0@@ -31,6 +31,8 @@ static DECLARE_RWSEM(root_port_sem);static struct device *cxl_topology_host; +static bool is_cxl_decoder (struct device *dev); +int cxl_register_topology_host (struct device *host) {down_write (&topology_host_sem);@@ -75,6 +77,45 @@ static void put_cxl_topology_host(struct device *dev)up_read (&topology_host_sem); } +static int decoder_match (struct device *dev,void *data) +{ +struct resource *theirs = (struct resource *)data; +struct cxl_decoder *cxld; + +if (!is_cxl_decoder (dev)) +return 0 ; + + cxld =to_cxl_decoder (dev); +if (theirs->start <= cxld->decoder_range.start && + theirs->end >= cxld->decoder_range.end) +return 1 ; + +return 0 ; +} + +static struct cxl_decoder *cxl_find_root_decoder (resource_size_t base, +resource_size_t size) +{ +struct cxl_decoder *cxld =NULL ; +struct device *cxldd; +struct device *host; +struct resource res = (struct resource ){ + .start = base, + .end = base + size -1 , + }; + + host =get_cxl_topology_host (); +if (!host) +return NULL ; + + cxldd =device_find_child (host, &res, decoder_match); +if (cxldd) + cxld =to_cxl_decoder (cxldd); + +put_cxl_topology_host (host); +return cxld; +} +static ssize_t devtype_show (struct device *dev,struct device_attribute *attr,char *buf) {@@ -280,6 +321,11 @@ bool is_root_decoder(struct device *dev) }EXPORT_SYMBOL_NS_GPL (is_root_decoder,CXL ); +static bool is_cxl_decoder (struct device *dev) +{ +return dev->type->release == cxl_decoder_release; +} +struct cxl_decoder *to_cxl_decoder (struct device *dev) { if (dev_WARN_ONCE(dev, dev->type->release != cxl_decoder_release,@@ -327,6 +373,7 @@ struct cxl_port *to_cxl_port(struct device *dev)return NULL ; return container_of(dev, struct cxl_port, dev); } +EXPORT_SYMBOL_NS_GPL (to_cxl_port,CXL );struct cxl_dport *cxl_get_root_dport (struct device *dev) {@@ -735,6 +782,24 @@ EXPORT_SYMBOL_NS_GPL(cxl_decoder_add, CXL);static void cxld_unregister (void *dev) { +struct cxl_decoder *plat_decoder, *cxld =to_cxl_decoder (dev); +resource_size_t base, size; + +if (is_root_decoder (dev)) { +device_unregister (dev); +return ; + } + + base = cxld->decoder_range.start; + size =range_len (&cxld->decoder_range); + +if (size) { + plat_decoder =cxl_find_root_decoder (base, size); +if (plat_decoder) +__release_region (&plat_decoder->platform_res, base, + size); + } +device_unregister (dev); }@@ -789,6 +854,8 @@ static int cxl_device_id(struct device *dev)return CXL_DEVICE_NVDIMM_BRIDGE ;if (dev->type == &cxl_nvdimm_type)return CXL_DEVICE_NVDIMM ; +if (dev->type == &cxl_port_type) +return CXL_DEVICE_PORT ;return 0 ; }drivers/cxl/core/regs.c +3
-3@@ -159,9 +159,8 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, }EXPORT_SYMBOL_NS_GPL (cxl_probe_device_regs,CXL ); -static void __iomem *devm_cxl_iomap_block (struct device *dev, -resource_size_t addr, -resource_size_t length) +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length) {void __iomem *ret_val;struct resource *res;@@ -180,6 +179,7 @@ static void __iomem *devm_cxl_iomap_block(struct device *dev,return ret_val; } +EXPORT_SYMBOL_NS_GPL (devm_cxl_iomap_block,CXL ); int cxl_map_component_regs(struct pci_dev *pdev, struct cxl_component_regs *regs,drivers/cxl/cxl.h +28
-6@@ -17,6 +17,9 @@* (port-driver, region-driver, nvdimm object-drivers... etc). */ +/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ +#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K +/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/ #define CXL_CM_OFFSET 0x1000#define CXL_CM_CAP_HDR_OFFSET 0x0@@ -36,11 +39,22 @@#define CXL_HDM_DECODER_CAP_OFFSET 0x0#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER0_BASE_LOW_OFFSET 0x10 -#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET 0x14 -#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET 0x18 -#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET 0x1c -#define CXL_HDM_DECODER0_CTRL_OFFSET 0x20 +#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) +#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) +#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE BIT(1) +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET (i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET (i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET (i) (0x20 * (i) + 0x18) +#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET (i) (0x20 * (i) + 0x1c) +#define CXL_HDM_DECODER0_CTRL_OFFSET (i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) +#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) +#define CXL_HDM_DECODER0_CTRL_TYPE BIT(12) +#define CXL_HDM_DECODER0_TL_LOW (i) (0x20 * (i) + 0x24) +#define CXL_HDM_DECODER0_TL_HIGH (i) (0x20 * (i) + 0x28)static inline int cxl_hdm_decoder_count (u32 cap_hdr) {@@ -148,6 +162,8 @@ int cxl_map_device_regs(struct pci_dev *pdev,enum cxl_regloc_type ;int cxl_find_regblock (struct pci_dev *pdev,enum cxl_regloc_type type,struct cxl_register_map *map); +void __iomem *devm_cxl_iomap_block (struct device *dev,resource_size_t addr, +resource_size_t length);#define CXL_RESOURCE_NONE ((resource_size_t) -1)#define CXL_TARGET_STRLEN 20@@ -165,7 +181,8 @@ void cxl_unregister_topology_host(struct device *host);#define CXL_DECODER_F_TYPE2 BIT(2)#define CXL_DECODER_F_TYPE3 BIT(3)#define CXL_DECODER_F_LOCK BIT(4) -#define CXL_DECODER_F_MASK GENMASK(4, 0) +#define CXL_DECODER_F_ENABLE BIT(5) +#define CXL_DECODER_F_MASK GENMASK(5, 0)enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,@@ -255,6 +272,8 @@ struct cxl_walk_context {* @dports: cxl_dport instances referenced by decoders * @decoder_ida: allocator for decoder ids * @component_reg_phys: component register capability base address (optional) +* @rescan_work: worker object for bus rescans after port additions +* @data: opaque data with driver specific usage */ struct cxl_port {struct device dev;@@ -263,6 +282,8 @@ struct cxl_port {struct list_head dports;struct ida decoder_ida;resource_size_t component_reg_phys; +struct work_struct rescan_work; +void *data; };/** @@ -325,6 +346,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_NVDIMM_BRIDGE 1#define CXL_DEVICE_NVDIMM 2 +#define CXL_DEVICE_PORT 3#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"drivers/cxl/port.c new file +174
-0@@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> + +#include "cxlmem.h" + +/** +* DOC: cxl port +* +* The port driver implements the set of functionality needed to allow full +* decoder enumeration and routing. A CXL port is an abstraction of a CXL +* component that implements some amount of CXL decoding of CXL.mem traffic. +* As of the CXL 2.0 spec, this includes: +* +* .. list-table:: CXL Components w/ Ports +* :widths: 25 25 50 +* :header-rows: 1 +* +* * - component +* - upstream +* - downstream +* * - Hostbridge +* - ACPI0016 +* - root port +* * - Switch +* - Switch Upstream Port +* - Switch Downstream Port +* * - Endpoint +* - Endpoint Port +* - N/A +* +* The primary service this driver provides is enumerating HDM decoders and +* presenting APIs to other drivers to utilize the decoders. +*/ + +static struct workqueue_struct *cxl_port_wq; + +struct cxl_port_data { +struct cxl_component_regs regs; + +struct port_caps { +unsigned int count; +unsigned int tc; +unsigned int interleave11_8; +unsigned int interleave14_12; + } caps; +}; + +static inline int to_interleave_granularity (u32 ctrl) +{ +int val =FIELD_GET (CXL_HDM_DECODER0_CTRL_IG_MASK , ctrl); + +return 256 << val; +} + +static inline int to_interleave_ways (u32 ctrl) +{ +int val =FIELD_GET (CXL_HDM_DECODER0_CTRL_IW_MASK , ctrl); + +return 1 << val; +} + +static void get_caps (struct cxl_port *port,struct cxl_port_data *cpd) +{ +void __iomem *hdm_decoder = cpd->regs.hdm_decoder; +struct port_caps *caps = &cpd->caps; +u32 hdm_cap; + + hdm_cap =readl (hdm_decoder +CXL_HDM_DECODER_CAP_OFFSET ); + + caps->count =cxl_hdm_decoder_count (hdm_cap); + caps->tc =FIELD_GET (CXL_HDM_DECODER_TARGET_COUNT_MASK , hdm_cap); + caps->interleave11_8 = +FIELD_GET (CXL_HDM_DECODER_INTERLEAVE_11_8 , hdm_cap); + caps->interleave14_12 = +FIELD_GET (CXL_HDM_DECODER_INTERLEAVE_14_12 , hdm_cap); +} + +static int map_regs (struct cxl_port *port,void __iomem *crb, +struct cxl_port_data *cpd) +{ +struct cxl_register_map map; +struct cxl_component_reg_map *comp_map = &map.component_map; + +cxl_probe_component_regs (&port->dev, crb, comp_map); +if (!comp_map->hdm_decoder.valid) { +dev_err (&port->dev,"HDM decoder registers invalid\n" ); +return -ENXIO ; + } + + cpd->regs.hdm_decoder = crb + comp_map->hdm_decoder.offset; + +return 0 ; +} + +static u64 get_decoder_size (void __iomem *hdm_decoder,int n) +{ +u32 ctrl =readl (hdm_decoder +CXL_HDM_DECODER0_CTRL_OFFSET (n)); + +if (ctrl &CXL_HDM_DECODER0_CTRL_COMMITTED ) +return 0 ; + +return ioread64_hi_lo (hdm_decoder + +CXL_HDM_DECODER0_SIZE_LOW_OFFSET (n)); +} + +static bool is_endpoint_port (struct cxl_port *port) +{ +/* Endpoints can't be ports... yet! */ +return false; +} + +static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" ); +} + +/* Minor layering violation */ +static int dvsec_range_used (struct cxl_port *port) +{ +struct cxl_endpoint_dvsec_info *info; +int i, ret =0 ; + +if (!is_endpoint_port (port)) +return 0 ; + + info = port->data; +for (i =0 ; i < info->ranges; i++) +if (info->range[i].size) + ret |=1 << i; + +return ret; +} + +static int enumerate_hdm_decoders (struct cxl_port *port, +struct cxl_port_data *portdata) +{ +void __iomem *hdm_decoder = portdata->regs.hdm_decoder; +bool global_enable; +u32 global_ctrl; +int i =0 ; + + global_ctrl =readl (hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + global_enable = global_ctrl &CXL_HDM_DECODER_ENABLE ; +if (!global_enable) { + i =dvsec_range_used (port); +if (i) { +dev_err (&port->dev, +"Couldn't add port because device is using DVSEC range registers\n" ); +return -EBUSY ; + } + } + +for (i =0 ; i < portdata->caps.count; i++) { +int rc, target_count = portdata->caps.tc; +struct cxl_decoder *cxld; +int *target_map =NULL ; +u64 size; + +if (is_endpoint_port (port)) + target_count =0 ; + + cxld =cxl_decoder_alloc (port, target_count); +if (IS_ERR (cxld)) { +dev_warn (&port->dev, +"Failed to allocate the decoder\n" ); +return PTR_ERR (cxld); + } + + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->interleave_ways =1 ;
Quoted 8 lines
drivers/cxl/port.c line 179
Quoted 5 lines
drivers/cxl/port.c new file +5
-0+ cxld->interleave_granularity =0 ; + + size =get_decoder_size (hdm_decoder, i); +if (size !=0 ) { +#define decoderN (reg, n) hdm_decoder + CXL_HDM_DECODER0_##reg(n)Perhaps this block in the if (size != 0) would be more readable if broken out to a utility function?
I don't get this comment, there is already get_decoder_size(). Can you please elaborate?
Quoted 2 lines
As normal I'm not keen on the macro magic if we can avoid it.
Yeah - just trying to not have to deal with wrapping long lines.
Quoted 13 lines
drivers/cxl/port.c line 187
Quoted 9 lines
drivers/cxl/port.c new file +8
-0+int temp[CXL_DECODER_MAX_INTERLEAVE ]; +u64 base; +u32 ctrl; +int j; +union { +u64 value; +char target_id[8 ]; + } target_list;I thought we tried to avoid this union usage in kernel because of the whole thing about c compilers being able to do what they like with it...
I wasn't aware of the restriction. I can change it back if it's required. It does look a lot nicer this way. Is there a reference to this issue somewhere?
Quoted 6 lines
drivers/cxl/port.c line 189
Quoted 2 lines
drivers/cxl/port.c new file +2
-0+ + target_map = temp;This is set to a variable that goes out of scope whilst
target_mapis still in use.
Yikes. I'm pretty surprised the compiler didn't catch this.
Quoted 33 lines
Quoted 19 lines
drivers/cxl/port.c new file +19
-0+ ctrl =readl (decoderN (CTRL_OFFSET , i)); + base =ioread64_hi_lo (decoderN (BASE_LOW_OFFSET , i)); + + cxld->decoder_range = (struct range ){ + .start = base, + .end = base + size -1 + }; + + cxld->flags =CXL_DECODER_F_ENABLE ; + cxld->interleave_ways =to_interleave_ways (ctrl); + cxld->interleave_granularity = +to_interleave_granularity (ctrl); + +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_TYPE , ctrl) ==0 ) + cxld->target_type =CXL_DECODER_ACCELERATOR ; + + target_list.value =ioread64_hi_lo (decoderN (TL_LOW , i)); +for (j =0 ; j < cxld->interleave_ways; j++) + target_map[j] = target_list.target_id[j];drivers/cxl/port.c line 218
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+#undef decoderN + } + + rc =cxl_decoder_add_locked (cxld, target_map); +if (rc) +put_device (&cxld->dev); +else + rc =cxl_decoder_autoremove (&port->dev, cxld); +if (rc) +dev_err (&port->dev,"Failed to add decoder\n" );If that fails on the autoremove registration (unlikely) this message will be rather confusing - as the add was fine...
Quoted 2 lines
This nest of carrying on when we have an error is getting ever deeper...
Yeah, this is not great. I will clean it up.
Thanks.
Quoted 17 lines
Quoted 17 lines
drivers/cxl/port.c new file +17
-0+else +dev_dbg (&cxld->dev,"Added to port %s\n" , +dev_name (&port->dev)); + } + +/* +* Turn on global enable now since DVSEC ranges aren't being used and +* we'll eventually want the decoder enabled. +*/ +if (!global_enable) { +dev_dbg (&port->dev,"Enabling HDM decode\n" ); +writel (global_ctrl |CXL_HDM_DECODER_ENABLE , hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + } + +return 0 ; +} +
Re: [PATCH 20/23] cxl/port: Introduce a port driver
On Mon, 22 Nov 2021 15:38:20 -0800
Ben Widawsky <[email protected]> wrote:
...
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
drivers/cxl/port.c new file +37
-0+static int enumerate_hdm_decoders (struct cxl_port *port, +struct cxl_port_data *portdata) +{ +void __iomem *hdm_decoder = portdata->regs.hdm_decoder; +bool global_enable; +u32 global_ctrl; +int i =0 ; + + global_ctrl =readl (hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + global_enable = global_ctrl &CXL_HDM_DECODER_ENABLE ; +if (!global_enable) { + i =dvsec_range_used (port); +if (i) { +dev_err (&port->dev, +"Couldn't add port because device is using DVSEC range registers\n" ); +return -EBUSY ; + } + } + +for (i =0 ; i < portdata->caps.count; i++) { +int rc, target_count = portdata->caps.tc; +struct cxl_decoder *cxld; +int *target_map =NULL ; +u64 size; + +if (is_endpoint_port (port)) + target_count =0 ; + + cxld =cxl_decoder_alloc (port, target_count); +if (IS_ERR (cxld)) { +dev_warn (&port->dev, +"Failed to allocate the decoder\n" ); +return PTR_ERR (cxld); + } + + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->interleave_ways =1 ;Quoted 8 lines
drivers/cxl/port.c line 179
Quoted 5 lines
drivers/cxl/port.c new file +5
-0+ cxld->interleave_granularity =0 ; + + size =get_decoder_size (hdm_decoder, i); +if (size !=0 ) { +#define decoderN (reg, n) hdm_decoder + CXL_HDM_DECODER0_##reg(n)Perhaps this block in the if (size != 0) would be more readable if broken out to a utility function?
Quoted 2 lines
I don't get this comment, there is already
get_decoder_size(). Can you please elaborate?
Sure. Just talking about having something like
if (size != 0) init_decoder() // or something better named
as an alternative to this deep nesting.
Quoted 20 lines
Quoted 3 lines
As normal I'm not keen on the macro magic if we can avoid it.
Yeah - just trying to not have to deal with wrapping long lines.
Quoted 13 lines
drivers/cxl/port.c line 187
Quoted 9 lines
drivers/cxl/port.c new file +8
-0+int temp[CXL_DECODER_MAX_INTERLEAVE ]; +u64 base; +u32 ctrl; +int j; +union { +u64 value; +char target_id[8 ]; + } target_list;I thought we tried to avoid this union usage in kernel because of the whole thing about c compilers being able to do what they like with it...
Quoted 2 lines
I wasn't aware of the restriction. I can change it back if it's required. It does look a lot nicer this way. Is there a reference to this issue somewhere?
Hmm. Seems I was out of date on this. There is a mess in the c99 standard that contradicts itself on whether you can do this or not.
https://davmac.wordpress.com/2010/02/26/c99-revisited/
The pull request for a patch form Andy got a Linus response...
https://lore.kernel.org/all/CAJZ5v0jq45atkapwSjJ4DkHhB1bfOA-Sh1TiA3dPXwKyFTBheA@mail.gmail.com/
Quoted 67 lines
Quoted 7 lines
drivers/cxl/port.c line 189
Quoted 3 lines
drivers/cxl/port.c new file +2
-0+ + target_map = temp;This is set to a variable that goes out of scope whilst
target_mapis still in use.Yikes. I'm pretty surprised the compiler didn't catch this.
Quoted 33 lines
Quoted 19 lines
drivers/cxl/port.c new file +19
-0+ ctrl =readl (decoderN (CTRL_OFFSET , i)); + base =ioread64_hi_lo (decoderN (BASE_LOW_OFFSET , i)); + + cxld->decoder_range = (struct range ){ + .start = base, + .end = base + size -1 + }; + + cxld->flags =CXL_DECODER_F_ENABLE ; + cxld->interleave_ways =to_interleave_ways (ctrl); + cxld->interleave_granularity = +to_interleave_granularity (ctrl); + +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_TYPE , ctrl) ==0 ) + cxld->target_type =CXL_DECODER_ACCELERATOR ; + + target_list.value =ioread64_hi_lo (decoderN (TL_LOW , i)); +for (j =0 ; j < cxld->interleave_ways; j++) + target_map[j] = target_list.target_id[j];drivers/cxl/port.c line 218
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+#undef decoderN + } + + rc =cxl_decoder_add_locked (cxld, target_map); +if (rc) +put_device (&cxld->dev); +else + rc =cxl_decoder_autoremove (&port->dev, cxld); +if (rc) +dev_err (&port->dev,"Failed to add decoder\n" );If that fails on the autoremove registration (unlikely) this message will be rather confusing - as the add was fine...
Quoted 2 lines
This nest of carrying on when we have an error is getting ever deeper...
Yeah, this is not great. I will clean it up.
Thanks.
Quoted 17 lines
Quoted 17 lines
drivers/cxl/port.c new file +17
-0+else +dev_dbg (&cxld->dev,"Added to port %s\n" , +dev_name (&port->dev)); + } + +/* +* Turn on global enable now since DVSEC ranges aren't being used and +* we'll eventually want the decoder enabled. +*/ +if (!global_enable) { +dev_dbg (&port->dev,"Enabling HDM decode\n" ); +writel (global_ctrl |CXL_HDM_DECODER_ENABLE , hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + } + +return 0 ; +} +
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 54 lines · Jonathan Cameron
On Mon, 22 Nov 2021 15:38:20 -0800
Ben Widawsky <[email protected]> wrote:...
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
drivers/cxl/port.c new file +37
-0+static int enumerate_hdm_decoders (struct cxl_port *port, +struct cxl_port_data *portdata) +{ +void __iomem *hdm_decoder = portdata->regs.hdm_decoder; +bool global_enable; +u32 global_ctrl; +int i =0 ; + + global_ctrl =readl (hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + global_enable = global_ctrl &CXL_HDM_DECODER_ENABLE ; +if (!global_enable) { + i =dvsec_range_used (port); +if (i) { +dev_err (&port->dev, +"Couldn't add port because device is using DVSEC range registers\n" ); +return -EBUSY ; + } + } + +for (i =0 ; i < portdata->caps.count; i++) { +int rc, target_count = portdata->caps.tc; +struct cxl_decoder *cxld; +int *target_map =NULL ; +u64 size; + +if (is_endpoint_port (port)) + target_count =0 ; + + cxld =cxl_decoder_alloc (port, target_count); +if (IS_ERR (cxld)) { +dev_warn (&port->dev, +"Failed to allocate the decoder\n" ); +return PTR_ERR (cxld); + } + + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->interleave_ways =1 ;Quoted 8 lines
drivers/cxl/port.c line 179
Quoted 5 lines
drivers/cxl/port.c new file +5
-0+ cxld->interleave_granularity =0 ; + + size =get_decoder_size (hdm_decoder, i); +if (size !=0 ) { +#define decoderN (reg, n) hdm_decoder + CXL_HDM_DECODER0_##reg(n)Perhaps this block in the if (size != 0) would be more readable if broken out to a utility function?
Quoted 2 lines
I don't get this comment, there is already
get_decoder_size(). Can you please elaborate?
Quoted 7 lines
Sure. Just talking about having something like
if (size != 0) init_decoder() // or something better namedas an alternative to this deep nesting.
Sounds good. I can combine it with the similar initialization done in cxl_acpi.
Quoted 20 lines
Quoted 20 lines
Quoted 3 lines
As normal I'm not keen on the macro magic if we can avoid it.
Yeah - just trying to not have to deal with wrapping long lines.
Quoted 13 lines
drivers/cxl/port.c line 187
Quoted 9 lines
drivers/cxl/port.c new file +8
-0+int temp[CXL_DECODER_MAX_INTERLEAVE ]; +u64 base; +u32 ctrl; +int j; +union { +u64 value; +char target_id[8 ]; + } target_list;I thought we tried to avoid this union usage in kernel because of the whole thing about c compilers being able to do what they like with it...
Quoted 7 lines
Quoted 2 lines
I wasn't aware of the restriction. I can change it back if it's required. It does look a lot nicer this way. Is there a reference to this issue somewhere?
Hmm. Seems I was out of date on this. There is a mess in the c99 standard that contradicts itself on whether you can do this or not.
Thanks for the link.
Quoted 5 lines
The pull request for a patch form Andy got a Linus response...
https://lore.kernel.org/all/CAJZ5v0jq45atkapwSjJ4DkHhB1bfOA-Sh1TiA3dPXwKyFTBheA@mail.gmail.com/
That was a fun read :-)
I'll defer to Dan on this. This was actually his code that he suggested in review of the RFC.
Quoted 69 lines
Quoted 68 lines
Quoted 8 lines
drivers/cxl/port.c line 189
Quoted 4 lines
drivers/cxl/port.c new file +2
-0+ + target_map = temp;This is set to a variable that goes out of scope whilst
target_mapis still in use.Yikes. I'm pretty surprised the compiler didn't catch this.
Quoted 33 lines
Quoted 19 lines
drivers/cxl/port.c new file +19
-0+ ctrl =readl (decoderN (CTRL_OFFSET , i)); + base =ioread64_hi_lo (decoderN (BASE_LOW_OFFSET , i)); + + cxld->decoder_range = (struct range ){ + .start = base, + .end = base + size -1 + }; + + cxld->flags =CXL_DECODER_F_ENABLE ; + cxld->interleave_ways =to_interleave_ways (ctrl); + cxld->interleave_granularity = +to_interleave_granularity (ctrl); + +if (FIELD_GET (CXL_HDM_DECODER0_CTRL_TYPE , ctrl) ==0 ) + cxld->target_type =CXL_DECODER_ACCELERATOR ; + + target_list.value =ioread64_hi_lo (decoderN (TL_LOW , i)); +for (j =0 ; j < cxld->interleave_ways; j++) + target_map[j] = target_list.target_id[j];drivers/cxl/port.c line 218
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+#undef decoderN + } + + rc =cxl_decoder_add_locked (cxld, target_map); +if (rc) +put_device (&cxld->dev); +else + rc =cxl_decoder_autoremove (&port->dev, cxld); +if (rc) +dev_err (&port->dev,"Failed to add decoder\n" );If that fails on the autoremove registration (unlikely) this message will be rather confusing - as the add was fine...
Quoted 2 lines
This nest of carrying on when we have an error is getting ever deeper...
Yeah, this is not great. I will clean it up.
Thanks.
Quoted 17 lines
Quoted 17 lines
drivers/cxl/port.c new file +17
-0+else +dev_dbg (&cxld->dev,"Added to port %s\n" , +dev_name (&port->dev)); + } + +/* +* Turn on global enable now since DVSEC ranges aren't being used and +* we'll eventually want the decoder enabled. +*/ +if (!global_enable) { +dev_dbg (&port->dev,"Enabling HDM decode\n" ); +writel (global_ctrl |CXL_HDM_DECODER_ENABLE , hdm_decoder +CXL_HDM_DECODER_CTRL_OFFSET ); + } + +return 0 ; +} +
Re: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Ben WidawskyQuoted 7 lines · Ben Widawsky
The CXL port driver is responsible for managing the decoder resources contained within the port. It will also provide APIs that other drivers will consume for managing these resources.
There are 4 types of ports in a system:
1. Platform port. This is a non-programmable entity. Such a port is named rootX. It is enumerated bycxl_acpiin an ACPI based system.
Can you mention the ACPI source (static table, namespace PNP ID) here?
Quoted 1 line
2. Hostbridge port.
Is "hostbridge" styled as a single word in the spec? I've only seen "host bridge" elsewhere.
Quoted 2 lines
This ports register access is defined in a platform specific way (CHBS for ACPI platforms).
s/This ports/This port's/
This doesn't really make sense, though. Are you saying the register access *mechanism* is platform specific? Or merely that the enumeration method (ACPI table, ACPI namespace, DT, etc) is platform-specific?
I assume "CHBS" is an ACPI static table?
Quoted 2 lines
It has n downstream ports,
each of which are known as CXL 2.0 root ports.
This sounds like a "host bridge port *contains* these root ports." That doesn't sound right.
Quoted 10 lines
Once the platform
specific mechanism to get the offset to the registers is obtained it operates just like other CXL components. The enumeration of this component is started bycxl_acpiand completed bycxl_port.3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained within
cxl_port.
In PCIe, "Downstream Port" includes both Root Ports and Switch Downstream Ports. It sounds like that's not the case for CXL?
Well, except above you say that a Host Bridge Port has N Downstream Ports, so I guess "Downstream Port" probably includes both Host Bridge Ports and Switch Downstream Ports.
Maybe you should just omit the "The downstream ports for a switch are simply known as downstream ports" sentence.
Quoted 4 lines
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started by
cxl_pci, and completed bycxl_port.
Does CXL use an "Upstream Port" concept similar to PCIe? In PCIe, "Upstream Port" includes both Switch Upstream Ports and the Upstream Port on an Endpoint.
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
Documentation/driver-api/cxl/memory-devices.rst CXL Memory Device · line 34
Quoted 10 lines
Documentation/driver-api/cxl/memory-devices.rst +4
-0@@ -28,6 +28,11 @@ CXL Memory Device .. kernel-doc:: drivers/cxl/pci.c :internal: +CXL Port +-------- +.. kernel-doc:: drivers/cxl/port.c + :doc: cxl port
s/cxl port/CXL Port/ ? I don't know exactly how this gets rendered by ReST.
Quoted 12 lines
Documentation/driver-api/cxl/memory-devices.rst +0
-0CXL Core -------- .. kernel-doc:: drivers/cxl/cxl.hdrivers/cxl/Kconfig +1
-0@@ -77,4 +77,26 @@ config CXL_PMEM provisioning the persistent memory capacity of CXL memory expanders. If unsure say 'm'. +
drivers/cxl/Kconfig config CXL_PMEM · line 90
Quoted 10 lines
drivers/cxl/Kconfig +10
-0+config CXL_MEM + tristate "CXL.mem: Memory Devices" + select CXL_PORT + depends on CXL_PCI + default CXL_BUS + help + The CXL.mem protocol allows a device to act as a provider of "System + RAM" and/or "Persistent Memory" that is fully coherent as if the + memory was attached to the typical CPU memory controller. This is + known as HDM "Host-managed Device Memory".
s/was attached/were attached/
drivers/cxl/Kconfig config CXL_PMEM · line 98
Quoted 7 lines
drivers/cxl/Kconfig +7
-0+ Say 'y/m' to enable a driver that will attach to CXL.mem devices for + memory expansion and control of HDM. See Chapter 9.13 in the CXL 2.0 + specification for a detailed description of HDM. + + If unsure say 'm'. + +
Spurious blank line.
Quoted 8 lines
drivers/cxl/Kconfig +3
-0+config CXL_PORT + tristate + endif
drivers/cxl/port.c line 10
Quoted 10 lines
drivers/cxl/port.c new file +10
-0@@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> + +#include "cxlmem.h" + +/** +* DOC: cxl port
s/cxl port/CXL port/ (I'm assuming this should match usage below) or maybe "CXL Port" both places to match typical PCIe spec usage?
Capitalization is a useful hint that this term is defined by a spec.
Quoted 4 lines
drivers/cxl/port.c new file +4
-0+* +* The port driver implements the set of functionality needed to allow full +* decoder enumeration and routing. A CXL port is an abstraction of a CXL +* component that implements some amount of CXL decoding of CXL.mem traffic.
drivers/cxl/port.c line 24
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+* As of the CXL 2.0 spec, this includes: +* +* .. list-table:: CXL Components w/ Ports +* :widths: 25 25 50 +* :header-rows: 1 +* +* * - component +* - upstream +* - downstream +* * - Hostbridge
s/Hostbridge/Host bridge/
drivers/cxl/port.c line 26
Quoted 2 lines
drivers/cxl/port.c new file +2
-0+* - ACPI0016 +* - root port
s/root port/Root Port/ to match Switch Ports below (and spec usage).
drivers/cxl/port.c line 32
Quoted 6 lines
drivers/cxl/port.c new file +6
-0+* * - Switch +* - Switch Upstream Port +* - Switch Downstream Port +* * - Endpoint +* - Endpoint Port +* - N/A
What does "N/A" mean here? Is it telling us something useful?
drivers/cxl/port.c line 118
Quoted 4 lines
drivers/cxl/port.c new file +4
-0+static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" );
Needs some context. A bare "Failed to rescan" in the dmesg log without a clue about who emitted it is really annoying.
Maybe you defined pr_fmt() somewhere; I couldn't find it easily.
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 10 lines · Bjorn Helgaas
Quoted 7 lines · Ben Widawsky
The CXL port driver is responsible for managing the decoder resources contained within the port. It will also provide APIs that other drivers will consume for managing these resources.
There are 4 types of ports in a system:
1. Platform port. This is a non-programmable entity. Such a port is named rootX. It is enumerated bycxl_acpiin an ACPI based system.Can you mention the ACPI source (static table, namespace PNP ID) here?
Done.
Quoted 6 lines
Quoted 2 lines
2. Hostbridge port.
Is "hostbridge" styled as a single word in the spec? I've only seen "host bridge" elsewhere.
2 words. I'm sadly inconsistent with this particular word. CXL spec capitalizes it.
Quoted 12 lines
Quoted 2 lines
This ports register access is defined in a platform specific way (CHBS for ACPI platforms).
s/This ports/This port's/
This doesn't really make sense, though. Are you saying the register access *mechanism* is platform specific? Or merely that the enumeration method (ACPI table, ACPI namespace, DT, etc) is platform-specific?
I assume "CHBS" is an ACPI static table?
The registers are spec defined. The base address of those registers is defined in a platform specific manner. Enumeration is a better word. CHBS is an ACPI static table, yes.
Quoted 6 lines
Quoted 2 lines
It has n downstream ports,
each of which are known as CXL 2.0 root ports.This sounds like a "host bridge port *contains* these root ports." That doesn't sound right.
What sounds better? A CXL 2.0 Root Port is CXL capabilities on top of the PCIe component which has the PCI_EXP_TYPE_ROOT_PORT cap. In my mental model, a host bridge does contain the root ports. Perhaps I am wrong about that?
Quoted 5 lines
Quoted 4 lines
Once the platform
specific mechanism to get the offset to the registers is obtained it operates just like other CXL components. The enumeration of this component is started bycxl_acpiand completed bycxl_port.
Quoted 9 lines
Quoted 5 lines
3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained within
cxl_port.In PCIe, "Downstream Port" includes both Root Ports and Switch Downstream Ports. It sounds like that's not the case for CXL?
In CXL 2.0, it's fairly close to true that switch downstream ports and root ports are equivalent. From today's driver perspective they are equivalent. Root ports have some capabilities which switch downstream ports do not.
Quoted 3 lines
Well, except above you say that a Host Bridge Port has N Downstream Ports, so I guess "Downstream Port" probably includes both Host Bridge Ports and Switch Downstream Ports.
Yes, in that case I meant a port which is downstream - confusing indeed.
Quoted 4 lines
Maybe you should just omit the "The downstream ports for a switch are simply known as downstream ports" sentence.
Sounds good.
Quoted 8 lines
Quoted 4 lines
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started by
cxl_pci, and completed bycxl_port.Does CXL use an "Upstream Port" concept similar to PCIe? In PCIe, "Upstream Port" includes both Switch Upstream Ports and the Upstream Port on an Endpoint.
Not really. Endpoints aren't known as ports in the spec and they have a decent amount of divergence from upstream ports. The main area where they do converge is in the memory decoding capabilities. In fact, it might come to a point where we find adding an endpoint as a port does not make sense, but for now it does.
Quoted 5 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service cxl_port will provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.
It could be in CXL core just the same, but I don't see too much of a benefit since the code would be almost identical. One nice aspect of having the port driver outside of CXL core is it would allow CXL devices which do not need port services (type-1 and probably type-2) to not need to load the port driver. We do not have examples of such devices today but there's good evidence they are being built. Whether or not they will even want CXL core is another topic up for debate however.
I admit Dan and I did discuss putting this in either its own port driver, or within core as a port driver. My inclination is, less is more in CXL core; but perhaps you will be able to change my mind.
Quoted 7 lines
Quoted 7 lines
Documentation/driver-api/cxl/memory-devices.rst +0
-0@@ -28,6 +28,11 @@ CXL Memory Device .. kernel-doc:: drivers/cxl/pci.c :internal:
Quoted 7 lines
Documentation/driver-api/cxl/memory-devices.rst CXL Memory Device · line 34
Quoted 4 lines
Documentation/driver-api/cxl/memory-devices.rst +4
-0+CXL Port +-------- +.. kernel-doc:: drivers/cxl/port.c + :doc: cxl ports/cxl port/CXL Port/ ? I don't know exactly how this gets rendered by ReST.
I believe this is the specific tag as specified in the .c file. I can capitalize it, but we haven't done this for other tags.
Quoted 58 lines
Quoted 13 lines
Documentation/driver-api/cxl/memory-devices.rst +0
-0CXL Core -------- .. kernel-doc:: drivers/cxl/cxl.hdrivers/cxl/Kconfig +1
-0@@ -77,4 +77,26 @@ config CXL_PMEM provisioning the persistent memory capacity of CXL memory expanders. If unsure say 'm'. +drivers/cxl/Kconfig config CXL_PMEM · line 90
Quoted 10 lines
drivers/cxl/Kconfig +10
-0+config CXL_MEM + tristate "CXL.mem: Memory Devices" + select CXL_PORT + depends on CXL_PCI + default CXL_BUS + help + The CXL.mem protocol allows a device to act as a provider of "System + RAM" and/or "Persistent Memory" that is fully coherent as if the + memory was attached to the typical CPU memory controller. This is + known as HDM "Host-managed Device Memory".s/was attached/were attached/
drivers/cxl/Kconfig config CXL_PMEM · line 98
Quoted 7 lines
drivers/cxl/Kconfig +7
-0+ Say 'y/m' to enable a driver that will attach to CXL.mem devices for + memory expansion and control of HDM. See Chapter 9.13 in the CXL 2.0 + specification for a detailed description of HDM. + + If unsure say 'm'. + +Spurious blank line.
Quoted 8 lines
drivers/cxl/Kconfig +3
-0+config CXL_PORT + tristate + endifdrivers/cxl/port.c line 10
Quoted 10 lines
drivers/cxl/port.c new file +10
-0@@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> + +#include "cxlmem.h" + +/** +* DOC: cxl port s/cxl port/CXL port/ (I'm assuming this should match usage below) or maybe "CXL Port" both places to match typical PCIe spec usage?
Quoted 2 lines
Capitalization is a useful hint that this term is defined by a spec.
As above, I don't mind changing this at all, but this would be inconsistent with the other tags we have defined.
Quoted 22 lines
Quoted 4 lines
drivers/cxl/port.c new file +4
-0+* +* The port driver implements the set of functionality needed to allow full +* decoder enumeration and routing. A CXL port is an abstraction of a CXL +* component that implements some amount of CXL decoding of CXL.mem traffic. drivers/cxl/port.c line 24
Quoted 10 lines
drivers/cxl/port.c new file +10
-0+* As of the CXL 2.0 spec, this includes: +* +* .. list-table:: CXL Components w/ Ports +* :widths: 25 25 50 +* :header-rows: 1 +* +* * - component +* - upstream +* - downstream +* * - Hostbridge s/Hostbridge/Host bridge/
drivers/cxl/port.c line 26
Quoted 2 lines
drivers/cxl/port.c new file +2
-0+* - ACPI0016 +* - root port s/root port/Root Port/ to match Switch Ports below (and spec usage).
Quoted 8 lines
drivers/cxl/port.c line 32
Quoted 6 lines
drivers/cxl/port.c new file +6
-0+* * - Switch +* - Switch Upstream Port +* - Switch Downstream Port +* * - Endpoint +* - Endpoint Port +* - N/A What does "N/A" mean here? Is it telling us something useful?
This gets rendered into a table that looks like the following:
| component | upstream | downstream | | --------- | -------- | ---------- | | Hostbridge | ACPI0016 | Root Port | | Switch | Switch Upstream Port | Switch Downstream Port | | Endpoint | Endpoint Port | N/A |
Quoted 11 lines
drivers/cxl/port.c line 118
Quoted 5 lines
drivers/cxl/port.c new file +4
-0+static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" );Needs some context. A bare "Failed to rescan" in the dmesg log without a clue about who emitted it is really annoying.
Maybe you defined
pr_fmt()somewhere; I couldn't find it easily.
I actually didn't know about pr_fmt() trick, so thanks for that. I'll improve this message to be more useful and contextual.
Quoted 1 line
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
On Tue, Nov 23, 2021 at 2:03 PM Ben Widawsky <[email protected]> wrote:
[..]
Quoted 20 lines
Quoted 4 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service
cxl_portwill provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.It could be in CXL core just the same, but I don't see too much of a benefit since the code would be almost identical. One nice aspect of having the port driver outside of CXL core is it would allow CXL devices which do not need port services (type-1 and probably type-2) to not need to load the port driver. We do not have examples of such devices today but there's good evidence they are being built. Whether or not they will even want CXL core is another topic up for debate however.
Quoted 3 lines
I admit Dan and I did discuss putting this in either its own port driver, or within core as a port driver. My inclination is, less is more in CXL core; but perhaps you will be able to change my mind.
No, I don't think Bjorn is talking about whether the driver is integrated into cxl_core.ko vs its own cxl_port.ko. IIUC this goes back to the original contention about have /sys/bus/cxl at all:
https://lore.kernel.org/r/CAPcyv4iu8D-hJoujLXw8a4myS7trOE1FcUhESLB_imGMECVfrg@mail.gmail.com
Unlike pcieportdrv where the functionality is bounded to a single device instance with relatively simpler hierarchical coordination of the memory space and services. CXL interleaving is both a foreign concept to the PCIE core and an awkward fit for the pci_bus_type device model. CXL uses the cxl_bus_type and bus drivers to coordinate CXL operations that have cross PCI device implications.
Outside of that I think the auxiliary device driver model, of which the PCIE portdrv model is an open-coded form, is a useful construct for separation of concerns. That said, I do want to hear more about what trouble it has caused though to make sure that CXL does not trip over the same issues longer term.
[..]
Quoted 8 lines
Quoted 8 lines
drivers/cxl/port.c line 118
Quoted 4 lines
drivers/cxl/port.c new file +4
-0+static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" );Needs some context. A bare "Failed to rescan" in the dmesg log without a clue about who emitted it is really annoying.
Quoted 5 lines
Quoted 2 lines
Maybe you defined
pr_fmt()somewhere; I couldn't find it easily.I actually didn't know about
pr_fmt()trick, so thanks for that. I'll improve this message to be more useful and contextual.
I am skeptical that this implementation of the workqueue properly synchronizes with workqueue shutdown concerns, but I have not had a chance to dig in too deep on this patchset.
Regardless, it is not worth reporting a rescan failure, because those are to be expected here. The rescan attempts to rescan when a constraint changes, there is no guarantee that all constraints are met just because one constraint changed, so rescan failures (device_attach() failures) are not interesting to report. The individual driver probe failure reporting is sufficient.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 57 lines · Dan Williams
On Tue, Nov 23, 2021 at 2:03 PM Ben Widawsky <[email protected]> wrote:
[..]Quoted 20 lines
Quoted 4 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service
cxl_portwill provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.It could be in CXL core just the same, but I don't see too much of a benefit since the code would be almost identical. One nice aspect of having the port driver outside of CXL core is it would allow CXL devices which do not need port services (type-1 and probably type-2) to not need to load the port driver. We do not have examples of such devices today but there's good evidence they are being built. Whether or not they will even want CXL core is another topic up for debate however.
Quoted 3 lines
I admit Dan and I did discuss putting this in either its own port driver, or within core as a port driver. My inclination is, less is more in CXL core; but perhaps you will be able to change my mind.
No, I don't think Bjorn is talking about whether the driver is integrated into cxl_core.ko vs its own cxl_port.ko. IIUC this goes back to the original contention about have
/sys/bus/cxlat all:https://lore.kernel.org/r/CAPcyv4iu8D-hJoujLXw8a4myS7trOE1FcUhESLB_imGMECVfrg@mail.gmail.com
Unlike pcieportdrv where the functionality is bounded to a single device instance with relatively simpler hierarchical coordination of the memory space and services. CXL interleaving is both a foreign concept to the PCIE core and an awkward fit for the
pci_bus_typedevice model. CXL uses thecxl_bus_typeand bus drivers to coordinate CXL operations that have cross PCI device implications.Outside of that I think the auxiliary device driver model, of which the PCIE portdrv model is an open-coded form, is a useful construct for separation of concerns. That said, I do want to hear more about what trouble it has caused though to make sure that CXL does not trip over the same issues longer term.
[..]
Quoted 11 lines
Quoted 10 lines
drivers/cxl/port.c line 118
Quoted 4 lines
drivers/cxl/port.c new file +4
-0+static void rescan_ports (struct work_struct *work) +{ +if (bus_rescan_devices (&cxl_bus_type)) +pr_warn ("Failed to rescan\n" );Needs some context. A bare "Failed to rescan" in the dmesg log without a clue about who emitted it is really annoying.
Maybe you defined
pr_fmt()somewhere; I couldn't find it easily.
Quoted 6 lines
Quoted 2 lines
I actually didn't know about
pr_fmt()trick, so thanks for that. I'll improve this message to be more useful and contextual.I am skeptical that this implementation of the workqueue properly synchronizes with workqueue shutdown concerns, but I have not had a chance to dig in too deep on this patchset.
Yeah, we discussed this. Please do check that. I'm happy to send out v2 with all of Jonathan's fixes first, so you don't have to duplicate effort with what he has already uncovered.
Quoted 7 lines
Regardless, it is not worth reporting a rescan failure, because those are to be expected here. The rescan attempts to rescan when a constraint changes, there is no guarantee that all constraints are met just because one constraint changed, so rescan failures (
device_attach()failures) are not interesting to report. The individual driver probe failure reporting is sufficient.
Agreed.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Dan Williams[+cc Christoph, since he has opinions about portdrv; Greg, Rafael, since they have good opinions about sysfs structure]
Quoted 29 lines · Dan Williams
On Tue, Nov 23, 2021 at 2:03 PM Ben Widawsky <[email protected]> wrote:
[..]Quoted 22 lines
Quoted 4 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service
cxl_portwill provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.It could be in CXL core just the same, but I don't see too much of a benefit since the code would be almost identical. One nice aspect of having the port driver outside of CXL core is it would allow CXL devices which do not need port services (type-1 and probably type-2) to not need to load the port driver. We do not have examples of such devices today but there's good evidence they are being built. Whether or not they will even want CXL core is another topic up for debate however.
Quoted 4 lines
I admit Dan and I did discuss putting this in either its own port driver, or within core as a port driver. My inclination is, less is more in CXL core; but perhaps you will be able to change my mind.
Quoted 5 lines
No, I don't think Bjorn is talking about whether the driver is integrated into cxl_core.ko vs its own cxl_port.ko. IIUC this goes back to the original contention about have
/sys/bus/cxlat all:https://lore.kernel.org/r/CAPcyv4iu8D-hJoujLXw8a4myS7trOE1FcUhESLB_imGMECVfrg@mail.gmail.com
That question was about whether we want the same physical device to be represented both in the /sys/bus/pci hierarchy and the /sys/bus/cxl hierarchy. That still seems a little weird to me, but I don't know enough about the CXL constraints to really object to it.
My question here is more about whether you're going to use something like the pcie_port_service_register() model for supporting multiple drivers attached to the same physical device.
The PCIe portdrv creates a /sys/bus/pci_express hierarchy parallel to the /sys/bus/pci hierarchy. The pci_express hierarchy has a "device" for every service (hotplug, AER, DPC, PME, etc) (see pcie_device_init()). This device creation is quite complicated and depends on whether the Port advertises a Capability, whether the platform has granted control to the OS, whether support is compiled in, etc.
I think that was a mistake because these hotplug, AER, DPC, PME "devices" are not independent things. They are optional features that can be added to a variety of devices, and those devices might have their own drivers. For example, we want to write drivers for vendor-specific functionality like PMUs in switches, but we can't do that cleanly because portdrv claims them.
The portdrv features are fully specified by the PCIe spec, so nothing is vendor-specific. They share interrupts. They share power state. They cannot be reset independently. They are not addressable entities in the usual bus/device/function model. They can't be removed or added like the underlying device can. I wasn't there when they were designed, but from reading the spec, it seems like they were designed as optional features of a device, not as separate devices themselves.
Quoted 7 lines
Unlike pcieportdrv where the functionality is bounded to a single device instance with relatively simpler hierarchical coordination of the memory space and services. CXL interleaving is both a foreign concept to the PCIE core and an awkward fit for the
pci_bus_typedevice model. CXL uses thecxl_bus_typeand bus drivers to coordinate CXL operations that have cross PCI device implications.
Quoted 5 lines
Outside of that I think the auxiliary device driver model, of which the PCIE portdrv model is an open-coded form, is a useful construct for separation of concerns. That said, I do want to hear more about what trouble it has caused though to make sure that CXL does not trip over the same issues longer term.
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 64 lines · Bjorn Helgaas
[+cc Christoph, since he has opinions about portdrv; Greg, Rafael, since they have good opinions about sysfs structure]
Quoted 29 lines · Dan Williams
On Tue, Nov 23, 2021 at 2:03 PM Ben Widawsky <[email protected]> wrote:
[..]Quoted 22 lines
Quoted 4 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service
cxl_portwill provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.It could be in CXL core just the same, but I don't see too much of a benefit since the code would be almost identical. One nice aspect of having the port driver outside of CXL core is it would allow CXL devices which do not need port services (type-1 and probably type-2) to not need to load the port driver. We do not have examples of such devices today but there's good evidence they are being built. Whether or not they will even want CXL core is another topic up for debate however.
Quoted 4 lines
I admit Dan and I did discuss putting this in either its own port driver, or within core as a port driver. My inclination is, less is more in CXL core; but perhaps you will be able to change my mind.
Quoted 5 lines
No, I don't think Bjorn is talking about whether the driver is integrated into cxl_core.ko vs its own cxl_port.ko. IIUC this goes back to the original contention about have
/sys/bus/cxlat all:https://lore.kernel.org/r/CAPcyv4iu8D-hJoujLXw8a4myS7trOE1FcUhESLB_imGMECVfrg@mail.gmail.com
That question was about whether we want the same physical device to be represented both in the
/sys/bus/pcihierarchy and the/sys/bus/cxlhierarchy. That still seems a little weird to me, but I don't know enough about the CXL constraints to really object to it.My question here is more about whether you're going to use something like the
pcie_port_service_register()model for supporting multiple drivers attached to the same physical device.The PCIe portdrv creates a
/sys/bus/pci_expresshierarchy parallel to the/sys/bus/pcihierarchy. Thepci_expresshierarchy has a "device" for every service (hotplug, AER, DPC, PME, etc) (seepcie_device_init()). This device creation is quite complicated and depends on whether the Port advertises a Capability, whether the platform has granted control to the OS, whether support is compiled in, etc.I think that was a mistake because these hotplug, AER, DPC, PME "devices" are not independent things. They are optional features that can be added to a variety of devices, and those devices might have their own drivers. For example, we want to write drivers for vendor-specific functionality like PMUs in switches, but we can't do that cleanly because portdrv claims them.
Quoted 7 lines
The portdrv features are fully specified by the PCIe spec, so nothing is vendor-specific. They share interrupts. They share power state. They cannot be reset independently. They are not addressable entities in the usual bus/device/function model. They can't be removed or added like the underlying device can. I wasn't there when they were designed, but from reading the spec, it seems like they were designed as optional features of a device, not as separate devices themselves.
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet. Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
Another example, the Native PCIe Enclosure Management (NPEM) specification defines a handful of registers that can appear anywhere in the PCIe hierarchy. How can you write a common driver that is generically applicable to any given NPEM instance?
The auxiliary bus answer to those questions is to register an auxiliary device to be driven by a common auxiliary driver across hard-device generations from the same vendor or even across vendors.
For your example about a PCIe port PMU driver it ultimately requires something to enumerate that capability and a library of code to exercise that functionality. What is a more natural fit than a Linux "device" and a Linux driver to coordinate attaching enabling code to a standalone hardware capability?
PCIe portdrv may be awkward because there was never a real native driver for the device to begin with and it all could have handled by 'pcie_portdriver' directly without registering more Linux devices, but assigning self contained features to Linux devices otherwise seems a useful idiom to me.
As for CXL, there is no analog of the PCIe portdrv pattern of just having a device act as a multiplexer of features to other Linux devices.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Quoted 6 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
Quoted 4 lines
Another example, the Native PCIe Enclosure Management (NPEM) specification defines a handful of registers that can appear anywhere in the PCIe hierarchy. How can you write a common driver that is generically applicable to any given NPEM instance?
Another totally messed up spec. But then pretty much everything coming from the PCIe SIG in terms of interface tends to be really, really broken lately.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Yes, I was being too coarse, point taken. However, I don't think that changes the observation that multiple vendors are using aux bus to share a feature driver across multiple base Ethernet drivers.
Quoted 11 lines
Quoted 7 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Quoted 8 lines
Quoted 4 lines
Another example, the Native PCIe Enclosure Management (NPEM) specification defines a handful of registers that can appear anywhere in the PCIe hierarchy. How can you write a common driver that is generically applicable to any given NPEM instance?
Another totally messed up spec. But then pretty much everything coming from the PCIe SIG in terms of interface tends to be really, really broken lately.
DVSEC and DOE is more of the same in terms of composing add-on features into devices. Hardware vendors want to mix multiple hard-IPs into a single device, aux bus is one response. Topology specific buses like /sys/bus/cxl are another.
This CXL port driver is offering enumeration, link management, and memory decode setup services to the rest of the topology. I see it as similar to management protocol services offered by libsas.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 2 lines · Dan Williams
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Given that I've only brought in in the last mail I have no idea what the original proposal even is.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Dan Williams
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Given that I've only brought in in the last mail I have no idea what the original proposal even is.
Neither do I :(
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 10 lines · Greg Kroah-Hartman
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Dan Williams
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Given that I've only brought in in the last mail I have no idea what the original proposal even is.
Neither do I :(
To be clear I am also trying to get to the root of Bjorn's concern.
The proposal in $SUBJECT is to build on / treat a CXL topology as a Linux device topology on /sys/bus/cxl that references devices on /sys/bus/platform (CXL ACPI topology root and Host Bridges) and /sys/bus/pci (CXL Switches and Endpoints). This CXL port device topology has already been shipping for a few kernel cycles. What is on the table now is a driver for CXL port devices (a logical Linux construct). The driver handles discovery of "component registers" either by ACPI table or PCI DVSEC and offers services to proxision CXL regions. CXL 'regions' are also proposed as Linux devices that represent an active CXL memory range which can interleave multiple endpoints across multiple switches and host bridges.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 11 lines · Dan Williams
Quoted 9 lines · Greg Kroah-Hartman
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Dan Williams
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Given that I've only brought in in the last mail I have no idea what the original proposal even is.
Quoted 8 lines
Quoted 1 line
Neither do I :(
To be clear I am also trying to get to the root of Bjorn's concern.
The proposal in $SUBJECT is to build on / treat a CXL topology as a Linux device topology on
/sys/bus/cxlthat references devices on/sys/bus/platform(CXL ACPI topology root and Host Bridges) and/sys/bus/pci(CXL Switches and Endpoints).
Wait, I am confused.
A bus lists devices that are on that bus. It does not list devices that are on other busses.
Now a device in a bus list can have a parent be on different types of busses, as the parent device does not matter, but the device itself can not be of different types.
So I do not understand what you are describing here at all. Do you have an example output of sysfs that shows this situation?
Quoted 2 lines
This CXL port device topology has already been shipping for a few kernel cycles.
So it's always been broken? :)
Quoted 5 lines
What is on
the table now is a driver for CXL port devices (a logical Linux construct). The driver handles discovery of "component registers" either by ACPI table or PCI DVSEC and offers services to proxision CXL regions.
So a normal bus controller device that creates new devices of a bus type, right? What is special about that?
Quoted 3 lines
CXL 'regions' are also proposed as Linux devices that represent an active CXL memory range which can interleave multiple endpoints across multiple switches and host bridges.
As long as these 'devices' have drivers and do not mess with the resources of any other device in the system, I do not understand the problem here.
Or is the issue that you are again trying to carve up "real" devices into tiny devices that then somehow need to be aware of the resources being touched by different drivers at the same time?
I'm still confused, sorry.
greg k-h
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 27 lines · Greg Kroah-Hartman
Quoted 11 lines · Dan Williams
Quoted 9 lines · Greg Kroah-Hartman
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Dan Williams
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
Given that I've only brought in in the last mail I have no idea what the original proposal even is.
Quoted 8 lines
Quoted 1 line
Neither do I :(
To be clear I am also trying to get to the root of Bjorn's concern.
The proposal in $SUBJECT is to build on / treat a CXL topology as a Linux device topology on
/sys/bus/cxlthat references devices on/sys/bus/platform(CXL ACPI topology root and Host Bridges) and/sys/bus/pci(CXL Switches and Endpoints).Wait, I am confused.
A bus lists devices that are on that bus. It does not list devices that are on other busses.
Quoted 6 lines
Now a device in a bus list can have a parent be on different types of busses, as the parent device does not matter, but the device itself can not be of different types.
So I do not understand what you are describing here at all. Do you have an example output of sysfs that shows this situation?
Commit 40ba17afdfab ("cxl/acpi: Introduce cxl_decoder objects")
...has an example of the devices registered on the CXL bus by the cxl_acpi driver.
Quoted 4 lines
Quoted 2 lines
This CXL port device topology has already been shipping for a few kernel cycles.
So it's always been broken? :)
Kidding aside, CXL has different moving pieces than PCI and the Linux device-driver model is how we are organizing that complexity. CXL is also symbiotically attached to PCI as it borrows the enumeration mechanism while building up a parallel CXL.mem universe to live alongside PCI.config and PCI.mmio. The CXL subsystem is similar to MD / DM where virtual devices are built up from other devices.
Quoted 8 lines
Quoted 5 lines
What is on
the table now is a driver for CXL port devices (a logical Linux construct). The driver handles discovery of "component registers" either by ACPI table or PCI DVSEC and offers services to proxision CXL regions.So a normal bus controller device that creates new devices of a bus type, right? What is special about that?
Yes, which is the root of my confusion about Bjorn's concern.
Quoted 7 lines
Quoted 3 lines
CXL 'regions' are also proposed as Linux devices that represent an active CXL memory range which can interleave multiple endpoints across multiple switches and host bridges.
As long as these 'devices' have drivers and do not mess with the resources of any other device in the system, I do not understand the problem here.
Correct, these drivers manage CXL.mem resources and leave PCI.mmio resource management to PCI.
Quoted 3 lines
Or is the issue that you are again trying to carve up "real" devices into tiny devices that then somehow need to be aware of the resources being touched by different drivers at the same time?
No, there's no multiplexing of resources across devices / drivers that requires cross-driver awareness.
Quoted 1 line
I'm still confused, sorry.
No worries, appreciate the attention to make sure this implementation is idiomatic and avoids any architectural dragons.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Dan WilliamsQuoted 20 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Yes, I was being too coarse, point taken. However, I don't think that changes the observation that multiple vendors are using aux bus to share a feature driver across multiple base Ethernet drivers.
Quoted 7 lines
Quoted 6 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
Quoted 6 lines
Quoted 3 lines
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
For the case of PCIe portdrv, the functionality involved is Power Management Events (PME), Advanced Error Reporting (AER), PCIe native hotplug, Downstream Port Containment (DPC), and Bandwidth Notifications.
Currently each has a separate "port service driver" with .probe(), .remove(), .suspend(), .resume(), etc.
The services share interrupt vectors. It's quite complicated to set them up, and it has to be done in the portdrv, not in the individual drivers.
They also share power state (D0, D3hot, etc).
In my mind these are not separate devices from the underlying PCI device, and I don't think splitting the support into "service drivers" made things better. I think it would be simpler if these were just added to pci_init_capabilities() like other optional pieces of PCI functionality.
Sysfs looks like this:
/sys/devices/pci0000:00/0000:00:1c.0/ # Root Port /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/ # AER "device" /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie010/ # BW notif
/sys/bus/pci/devices/0000:00:1c.0 -> ../../../devices/pci0000:00/0000:00:1c.0/ /sys/bus/pci_express/devices/0000:00:1c.0:pcie002 -> ../../../devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/
The "pcie002" names (hex for PCIE_PORT_SERVICE_AER, etc.) are unintelligible. I don't know why we have a separate /sys/bus/pci_express hierarchy.
IIUC, CXL devices will be enumerated by the usual PCI enumeration, so there will be a struct pci_dev for them, and they will appear under /sys/devices/pci*/.
They will have the usual PCI Power Management, MSI, AER, DPC, and similar Capabilites, so the PCI core will manage them.
CXL devices have lots of fancy additional features. Does that merit making a separate struct device and a separate sysfs hierarchy for them? I don't know.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Another example, the Native PCIe Enclosure Management (NPEM) specification defines a handful of registers that can appear anywhere in the PCIe hierarchy. How can you write a common driver that is generically applicable to any given NPEM instance?
Another totally messed up spec. But then pretty much everything coming from the PCIe SIG in terms of interface tends to be really, really broken lately.
Hotplug is more central to PCI than NPEM is, but NPEM is a little bit like PCIe native hotplug in concept: hotplug has a few registers that control downstream indicators, interlock, and power controller; NPEM has registers that control downstream indicators.
Both are prescribed by the PCIe spec and presumably designed to work alongside the usual device-specific drivers for bridges, SSDs, etc.
I would at least explore the idea of doing common support by integrating NPEM into the PCI core. There would have to be some hook for the enclosure-specific bits, but I think it's fair for the details of sending commands and polling for command completed to be part of the PCI core.
Quoted 4 lines
DVSEC and DOE is more of the same in terms of composing add-on features into devices. Hardware vendors want to mix multiple hard-IPs into a single device, aux bus is one response. Topology specific buses like
/sys/bus/cxlare another.
VSEC and DVSEC are pretty much wild cards since the PCIe spec says nothing about what registers they may contain or how they should work.
DOE *is* specified by PCIe, at least in terms of the data transfer protocol (interrupt usage, read/write mailbox, etc). I think that, and the fact that it's not specific to CXL, means we need some kind of PCI core interface to do the transfers.
Quoted 3 lines
This CXL port driver is offering enumeration, link management, and memory decode setup services to the rest of the topology. I see it as similar to management protocol services offered by libsas.
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
[ add Stuart for NPEM feedback ]
Quoted 65 lines · Bjorn Helgaas
Quoted 20 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Yes, I was being too coarse, point taken. However, I don't think that changes the observation that multiple vendors are using aux bus to share a feature driver across multiple base Ethernet drivers.
Quoted 7 lines
Quoted 6 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
Quoted 6 lines
Quoted 3 lines
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
For the case of PCIe portdrv, the functionality involved is Power Management Events (PME), Advanced Error Reporting (AER), PCIe native hotplug, Downstream Port Containment (DPC), and Bandwidth Notifications.
Currently each has a separate "port service driver" with .probe(), .remove(), .suspend(), .resume(), etc.
The services share interrupt vectors. It's quite complicated to set them up, and it has to be done in the portdrv, not in the individual drivers.
They also share power state (D0, D3hot, etc).
In my mind these are not separate devices from the underlying PCI device, and I don't think splitting the support into "service drivers" made things better. I think it would be simpler if these were just added to
pci_init_capabilities()like other optional pieces of PCI functionality.Sysfs looks like this:
/sys/devices/pci0000:00/0000:00:1c.0/ # Root Port /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/ # AER "device" /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie010/ # BW notif/sys/bus/pci/devices/0000:00:1c.0 -> ../../../devices/pci0000:00/0000:00:1c.0/ /sys/bus/pci_express/devices/0000:00:1c.0:pcie002 -> ../../../devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/
The "pcie002" names (hex for
PCIE_PORT_SERVICE_AER, etc.) are unintelligible. I don't know why we have a separate/sys/bus/pci_expresshierarchy.IIUC, CXL devices will be enumerated by the usual PCI enumeration, so there will be a struct
pci_devfor them, and they will appear under/sys/devices/pci*/.
Quoted 6 lines
They will have the usual PCI Power Management, MSI, AER, DPC, and similar Capabilites, so the PCI core will manage them.
CXL devices have lots of fancy additional features. Does that merit making a separate struct device and a separate sysfs hierarchy for them? I don't know.
Thank you, this framing really helps.
So, if I look at this through the lens of the "can this just be handled by pci_init_capabilities()?" guardband, where the capability is device-scoped and shares resources that a driver for the same device would use, then yes, PCIe port services do not merit a separate bus.
CXL memory expansion is distinctly not in that category. CXL is a distinct specification (i.e. unlike PCIe port services which are literally in the PCI Sig purview), distinct transport/data layer (effectively a different physical connection on the wire), and is composable in ways that PCIe is not.
For example, if you trigger FLR on a PCI device you would expect the entirety of pci_init_capabilities() needs to be saved / restored, CXL state is not affected by FLR.
The separate link layer for CXL means that mere device visibility is not sufficient to determine CXL operation. Whereas with a PCI driver if you can see the device you know that the device is ready to be the target of config, io, and mmio cycles, CXL.mem operation may not be available when the device is visible to enumeration.
The composability of CXL places the highest demand for an independent 'struct bus_type' and registering CXL devices for their corresponding PCIe devices. The bus is a rendezvous for all the moving pieces needed to validate and provision a CXL memory region that may span multiple endpoints, switches and host bridges. An action like reprogramming memory decode of an endpoint needs to be coordinated across the entire topology.
The fact that the PCI core remains blissfully unaware of all these interdependencies is a feature because CXL is effectively a super-set of PCIe for fast-path CXL.mem operation, even though it is derivative for enumeration and slow-path manageability.
So I hope you see CXL's need to create some PCIe-symbiotic objects in order to compose CXL objects (like interleaved memory regions) that have no PCIe analog.
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Another example, the Native PCIe Enclosure Management (NPEM) specification defines a handful of registers that can appear anywhere in the PCIe hierarchy. How can you write a common driver that is generically applicable to any given NPEM instance?
Another totally messed up spec. But then pretty much everything coming from the PCIe SIG in terms of interface tends to be really, really broken lately.
Hotplug is more central to PCI than NPEM is, but NPEM is a little bit like PCIe native hotplug in concept: hotplug has a few registers that control downstream indicators, interlock, and power controller; NPEM has registers that control downstream indicators.
Quoted 8 lines
Both are prescribed by the PCIe spec and presumably designed to work alongside the usual device-specific drivers for bridges, SSDs, etc.
I would at least explore the idea of doing common support by integrating NPEM into the PCI core. There would have to be some hook for the enclosure-specific bits, but I think it's fair for the details of sending commands and polling for command completed to be part of the PCI core.
The problem with NPEM compared to hotplug LED signaling is that it has the strange property that an NPEM higher in the topology might override one lower in the topology. This is to support things like NVME enclosures where the NVME device itself may have an NPEM instance, but that instance is of course not suitable for signaling that the device is not present. So, instead the system may be designed to have an NPEM in the upstream switch port and disable the NPEM instances in the device. Platform firmware decides which NPEM is in use.
It also has the "fun" property of additionally being overridden by ACPI.
Stuart, have a look at collapsing the auxiliary-device approach into pci_init_capabilities() and whether that can still coordinate with the enclosure code.
One of the nice properties of the auxiliary organization is well defined module boundaries. Will NPEM in the PCI core now force CONFIG_ENCLOSURE_SERVICES to be built-in? That seems an unwanted side effect to me.
Quoted 5 lines
Quoted 4 lines
DVSEC and DOE is more of the same in terms of composing add-on features into devices. Hardware vendors want to mix multiple hard-IPs into a single device, aux bus is one response. Topology specific buses like
/sys/bus/cxlare another.
Quoted 7 lines
VSEC and DVSEC are pretty much wild cards since the PCIe spec says nothing about what registers they may contain or how they should work.
DOE *is* specified by PCIe, at least in terms of the data transfer protocol (interrupt usage, read/write mailbox, etc). I think that, and the fact that it's not specific to CXL, means we need some kind of PCI core interface to do the transfers.
DOE transfer code belongs in drivers/pci/ period, but does it really need to be in drivers/pci/built-in.a for everyone regardless of whether it is being used or not?
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 89 lines · Dan Williams
[ add Stuart for NPEM feedback ]
Quoted 65 lines · Bjorn Helgaas
Quoted 20 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Yes, I was being too coarse, point taken. However, I don't think that changes the observation that multiple vendors are using aux bus to share a feature driver across multiple base Ethernet drivers.
Quoted 7 lines
Quoted 6 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
Quoted 6 lines
Quoted 3 lines
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
For the case of PCIe portdrv, the functionality involved is Power Management Events (PME), Advanced Error Reporting (AER), PCIe native hotplug, Downstream Port Containment (DPC), and Bandwidth Notifications.
Currently each has a separate "port service driver" with .probe(), .remove(), .suspend(), .resume(), etc.
The services share interrupt vectors. It's quite complicated to set them up, and it has to be done in the portdrv, not in the individual drivers.
They also share power state (D0, D3hot, etc).
In my mind these are not separate devices from the underlying PCI device, and I don't think splitting the support into "service drivers" made things better. I think it would be simpler if these were just added to
pci_init_capabilities()like other optional pieces of PCI functionality.Sysfs looks like this:
/sys/devices/pci0000:00/0000:00:1c.0/ # Root Port /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/ # AER "device" /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie010/ # BW notif/sys/bus/pci/devices/0000:00:1c.0 -> ../../../devices/pci0000:00/0000:00:1c.0/ /sys/bus/pci_express/devices/0000:00:1c.0:pcie002 -> ../../../devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/
The "pcie002" names (hex for
PCIE_PORT_SERVICE_AER, etc.) are unintelligible. I don't know why we have a separate/sys/bus/pci_expresshierarchy.IIUC, CXL devices will be enumerated by the usual PCI enumeration, so there will be a struct
pci_devfor them, and they will appear under/sys/devices/pci*/.Quoted 6 lines
They will have the usual PCI Power Management, MSI, AER, DPC, and similar Capabilites, so the PCI core will manage them.
CXL devices have lots of fancy additional features. Does that merit making a separate struct device and a separate sysfs hierarchy for them? I don't know.
Thank you, this framing really helps.
So, if I look at this through the lens of the "can this just be handled by
pci_init_capabilities()?" guardband, where the capability is device-scoped and shares resources that a driver for the same device would use, then yes, PCIe port services do not merit a separate bus.CXL memory expansion is distinctly not in that category. CXL is a distinct specification (i.e. unlike PCIe port services which are literally in the PCI Sig purview), distinct transport/data layer (effectively a different physical connection on the wire), and is composable in ways that PCIe is not.
Quoted 8 lines
For example, if you trigger FLR on a PCI device you would expect the entirety of
pci_init_capabilities()needs to be saved / restored, CXL state is not affected by FLR.The separate link layer for CXL means that mere device visibility is not sufficient to determine CXL operation. Whereas with a PCI driver if you can see the device you know that the device is ready to be the target of config, io, and mmio cycles,
Not strictly true. A PCI device must respond to config transactions within a short time after reset, but it does not respond to IO or MEM transactions until a driver sets PCI_COMMAND_IO or PCI_COMMAND_MEMORY.
Quoted 10 lines
... CXL.mem operation may not be available when the device is visible to enumeration.
The composability of CXL places the highest demand for an independent 'struct
bus_type' and registering CXL devices for their corresponding PCIe devices. The bus is a rendezvous for all the moving pieces needed to validate and provision a CXL memory region that may span multiple endpoints, switches and host bridges. An action like reprogramming memory decode of an endpoint needs to be coordinated across the entire topology.
I guess software uses the CXL DVSEC to distinguish a CXL device from a PCIe device, at least for 00.0. Not sure about non-Dev 0 Func 0 devices; maybe this implies other functions in the same device are part of the same CXL device?
A CXL device may comprise several PCIe devices, and I think they may have non-CXL Capabilities, so I assume you need a struct pci_dev for each PCIe device?
Looks like a single CXL DVSEC controls the entire CXL device (including several PCIe devices), so I assume you want some kind of struct cxl_dev to represent that aggregation?
Quoted 4 lines
The fact that the PCI core remains blissfully unaware of all these interdependencies is a feature because CXL is effectively a super-set of PCIe for fast-path CXL.mem operation, even though it is derivative for enumeration and slow-path manageability.
I don't know how blissfully unaware PCI can be. Can a user remove a PCIe device that's part of a CXL device via sysfs? Is the PCIe device available for drivers to claim? Do we need coordination between cxl_driver_register() and pci_register_driver()? Does CXL impose new constraints on PCI power management?
Quoted 18 lines
So I hope you see CXL's need to create some PCIe-symbiotic objects in order to compose CXL objects (like interleaved memory regions) that have no PCIe analog.
Quoted 5 lines
Hotplug is more central to PCI than NPEM is, but NPEM is a little bit like PCIe native hotplug in concept: hotplug has a few registers that control downstream indicators, interlock, and power controller; NPEM has registers that control downstream indicators.
Quoted 8 lines
Both are prescribed by the PCIe spec and presumably designed to work alongside the usual device-specific drivers for bridges, SSDs, etc.
I would at least explore the idea of doing common support by integrating NPEM into the PCI core. There would have to be some hook for the enclosure-specific bits, but I think it's fair for the details of sending commands and polling for command completed to be part of the PCI core.
Quoted 6 lines
The problem with NPEM compared to hotplug LED signaling is that it has the strange property that an NPEM higher in the topology might override one lower in the topology. This is to support things like NVME enclosures where the NVME device itself may have an NPEM instance, but that instance is of course not suitable for signaling that the device is not present.
I would envision the PCI core providing only a bare-bones "send this command and wait for it to complete" sort of interface. Everything else, including deciding which device to use, would go elsewhere.
Quoted 3 lines
So, instead the system may be designed to have an NPEM in the upstream switch port and disable the NPEM instances in the device. Platform firmware decides which NPEM is in use.
Since you didn't mention a firmware interface to communicate which NPEM is in use, I assume firmware does this by preventing other devices from advertising NPEM support?
Quoted 6 lines
It also has the "fun" property of additionally being overridden by ACPI.
...
One of the nice properties of the auxiliary organization is well defined module boundaries. Will NPEM in the PCI core now forceCONFIG_ENCLOSURE_SERVICESto be built-in? That seems an unwanted side effect to me.
If the PCI core provides only the mechanism, and the smarts of NPEM are in something analogous to drivers/scsi/ses.c, I don't think it would force CONFIG_ENCLOSURE_SERVICES to be built-in.
Quoted 8 lines
Quoted 4 lines
DOE *is* specified by PCIe, at least in terms of the data transfer protocol (interrupt usage, read/write mailbox, etc). I think that, and the fact that it's not specific to CXL, means we need some kind of PCI core interface to do the transfers.
DOE transfer code belongs in
drivers/pci/period, but does it really need to be indrivers/pci/built-in.afor everyone regardless of whether it is being used or not?
I think my opinion would depend on how small the DOE transfer code could be made and how much it would complicate things to make it a module. And also how we could enforce whatever mutual exclusion we need to make it safe.
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 95 lines · Bjorn Helgaas
Quoted 93 lines · Dan Williams
[ add Stuart for NPEM feedback ]
Quoted 65 lines · Bjorn Helgaas
Quoted 20 lines · Dan Williams
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams
Let me ask a clarifying question coming from the other direction that resulted in the creation of the auxiliary bus architecture. Some background. RDMA is a protocol that may run on top of Ethernet.
No, RDMA is a concept. Linux supports 2 and a half RDMA protocols that run over ethernet (RoCE v1 and v2 and iWarp).
Yes, I was being too coarse, point taken. However, I don't think that changes the observation that multiple vendors are using aux bus to share a feature driver across multiple base Ethernet drivers.
Quoted 7 lines
Quoted 6 lines
Consider the case where you have multiple generations of Ethernet adapter devices, but they all support common RDMA functionality. You only have the one PCI device to attach a unique Ethernet driver. What is an idiomatic way to deploy a module that automatically loads and attaches to the exported common functionality across adapters that otherwise have a unique native driver for the hardware device?
Quoted 6 lines
Quoted 3 lines
The whole aux bus drama is mostly because the intel design for these is really fucked up. All the sane HCAs do not use this model. All this attchment crap really should not be there.
I am missing the counter proposal in both Bjorn's and your distaste for aux bus and PCIe portdrv?
For the case of PCIe portdrv, the functionality involved is Power Management Events (PME), Advanced Error Reporting (AER), PCIe native hotplug, Downstream Port Containment (DPC), and Bandwidth Notifications.
Currently each has a separate "port service driver" with .probe(), .remove(), .suspend(), .resume(), etc.
The services share interrupt vectors. It's quite complicated to set them up, and it has to be done in the portdrv, not in the individual drivers.
They also share power state (D0, D3hot, etc).
In my mind these are not separate devices from the underlying PCI device, and I don't think splitting the support into "service drivers" made things better. I think it would be simpler if these were just added to
pci_init_capabilities()like other optional pieces of PCI functionality.Sysfs looks like this:
/sys/devices/pci0000:00/0000:00:1c.0/ # Root Port /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/ # AER "device" /sys/devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie010/ # BW notif/sys/bus/pci/devices/0000:00:1c.0 -> ../../../devices/pci0000:00/0000:00:1c.0/ /sys/bus/pci_express/devices/0000:00:1c.0:pcie002 -> ../../../devices/pci0000:00/0000:00:1c.0/0000:00:1c.0:pcie002/
The "pcie002" names (hex for
PCIE_PORT_SERVICE_AER, etc.) are unintelligible. I don't know why we have a separate/sys/bus/pci_expresshierarchy.IIUC, CXL devices will be enumerated by the usual PCI enumeration, so there will be a struct
pci_devfor them, and they will appear under/sys/devices/pci*/.Quoted 6 lines
They will have the usual PCI Power Management, MSI, AER, DPC, and similar Capabilites, so the PCI core will manage them.
CXL devices have lots of fancy additional features. Does that merit making a separate struct device and a separate sysfs hierarchy for them? I don't know.
Thank you, this framing really helps.
So, if I look at this through the lens of the "can this just be handled by
pci_init_capabilities()?" guardband, where the capability is device-scoped and shares resources that a driver for the same device would use, then yes, PCIe port services do not merit a separate bus.CXL memory expansion is distinctly not in that category. CXL is a distinct specification (i.e. unlike PCIe port services which are literally in the PCI Sig purview), distinct transport/data layer (effectively a different physical connection on the wire), and is composable in ways that PCIe is not.
For example, if you trigger FLR on a PCI device you would expect the entirety of
pci_init_capabilities()needs to be saved / restored, CXL state is not affected by FLR.
Quoted 8 lines
Quoted 4 lines
The separate link layer for CXL means that mere device visibility is not sufficient to determine CXL operation. Whereas with a PCI driver if you can see the device you know that the device is ready to be the target of config, io, and mmio cycles,
Not strictly true. A PCI device must respond to config transactions within a short time after reset, but it does not respond to IO or MEM transactions until a driver sets
PCI_COMMAND_IOorPCI_COMMAND_MEMORY.
Right, what I was attempting to convey is that it's not like CXL.mem. While flipping a bit on the device turns on PCI.mmio target support, there's additional polling and status checking to be done after the device is enabled to be a target for CXL.mem. I.e. the CXL.mem configuration is done via PCI.mmio* (*for CXL 2.0 devices) and only after the device negotiates a CXL link beyond the base PCIe link. It is also the case that the device may not be ready for CXL.mem until all the devices that compose the same range are available as well.
Quoted 12 lines
Quoted 11 lines
... CXL.mem operation may not be available when the device is visible to enumeration.
The composability of CXL places the highest demand for an independent 'struct
bus_type' and registering CXL devices for their corresponding PCIe devices. The bus is a rendezvous for all the moving pieces needed to validate and provision a CXL memory region that may span multiple endpoints, switches and host bridges. An action like reprogramming memory decode of an endpoint needs to be coordinated across the entire topology.
Quoted 2 lines
I guess software uses the CXL DVSEC to distinguish a CXL device from a PCIe device, at least for 00.0.
driver/cxl/pci.c attaches to: "PCI_DEVICE_CLASS((PCI_CLASS_MEMORY_CXL << 8 | CXL_MEMORY_PROGIF), ~0)"
I am not aware of any restriction for that class code to appear at function0?
Quoted 3 lines
Not sure about non-Dev 0 Func 0
devices; maybe this implies other functions in the same device are part of the same CXL device?
DVSEC is really only telling us the layout of the MMIO register space. A CXL Memory Device (CXL.mem endpoint) implements so-called "device" registers and "component" registers. The "device" registers control things that a traditional PCI driver would control like a mailbox interface and device local status registers. The "component" registers are what mediate access to the CXL.mem decode space.
It is somewhat confusing because CXL 1.1 devices used the DVSEC configuration registers directly for CXL.mem configuration, but CXL 2.0 ditched that organization. The Linux driver is only targeting CXL 2.0+ devices as platform firmware owns setting up CXL 1.1 memory expanders. As far as Linux is concerned CXL 1.1 looks like DDR i.e. it's just address space set up by the BIOS and populated into EFI and ACPI tables. CXL 1.1 is also implementing a 1:1 device-to-memory-range association. CXL 2.0 allows for N:1 devices-to-memory-range and dynamic configuration of that by the OS. CXL 1.1 platform firmware locks out the OS from making configuration changes.
Quoted 7 lines
A CXL device may comprise several PCIe devices, and I think they may have non-CXL Capabilities, so I assume you need a struct
pci_devfor each PCIe device?Looks like a single CXL DVSEC controls the entire CXL device (including several PCIe devices), so I assume you want some kind of struct
cxl_devto represent that aggregation?
We have 3 classes of a 'cxl_dev' in drivers/cxl:
"mem" devices (PCIe endpoints)
"port" devices (PCIe Upstream Switch Ports, PCIe Root Ports, ACPI0016 Host Bridge devices, and ACPI0017 CXL Root devices) "region" devices (aggregate devices composed of multiple mem devices).
The "mem" driver is tasked with enumerating all "ports" in the path between itself and the ACPI0017 root and validating that a CXL link is up at each hop before enumerating "region" devices.
Quoted 8 lines
Quoted 5 lines
The fact that the PCI core remains blissfully unaware of all these interdependencies is a feature because CXL is effectively a super-set of PCIe for fast-path CXL.mem operation, even though it is derivative for enumeration and slow-path manageability.
I don't know how blissfully unaware PCI can be. Can a user remove a PCIe device that's part of a CXL device via sysfs?
Yes. If that PCIe device is an endpoint then the corresponding "mem" driver will get a ->remove() event because it is registered as a child of that parent PCIe device. That in turn will tear down the relevant part of the CXL port topology.
However, a gap (deliberate design choice?) in the PCI hotplug implementation I currently see is an ability for an endpoint PCI driver to dynamically deny hotplug requests based on the state of the device. pci_ignore_hotplug() seems inadequate for the job of making sure that someone first disables all participation of a "mem" device in all regions before asking for its physical removal. However, if someone force removes a device the driver and CXL subsystem will do the right thing and go fail that memory region for all users. I'm working with other DAX developers on a range based memory_failure() api for this case.
Quoted 1 line
Is the PCIe device available for drivers to claim?
drivers/cxl/pci.c *is* a native PCI driver for CXL memory expanders. You might be thinking of CXL accelerator devices, where the CXL.cache and CXL.mem capabilities are incremental capabilities for the accelerator. So, for example, a GPU with CXL.mem capabilities, would be mostly ignored by the drivers/cxl/ stack by default. Only if that device+driver wanted to emulate a generic memory expander and share its memory space with the host might it instantiate mem, port, and region objects. Otherwise CXL for accelerators is mostly a transparent capability that the OS may not need to manage. Rather than copy data back and forth between the host a CXL enabled GPU's driver can just assign pointers to its local memory space and trust that it is coherent. That's a gross oversimplification, but I want to convey that there are CXL devices like accelerators that are the responsibility of each accelerator PCI driver, vs CXL memory expander devices which are generic providers of "System RAM" and "Persistent Memory" resources and need the "mem", "port", "region" schema.
Quoted 1 line
Do we need coordination between
cxl_driver_register()andpci_register_driver()?
They do not collide, and I think this goes back to the concern about whether drivers/cxl/ is scanning for all CXL DVSECs. No, it only cares about the CXL DVSEC in CXL memory expander endpoints with the aforementioned class code, and the CXL DVSEC in every upstream switch port in the ancestry up to the CXL root device (ACPI0017). CXL accelerator drivers will always use pci_register_driver() and can decide to register their DVSEC with the CXL core, or keep it private to themselves. I imagine a GPU accelerator might register a "mem" device if it needs to get CXL.mem decode set up after a hotplug event, but if it's only using CXL.cache, or if its memory space was established by platform firmware then drivers/cxl/ is not involved.
Quoted 1 line
Does CXL impose new constraints on PCI power management?
Recall that CXL is built such that it could be supported by a legacy operating system where platform firmware owns the setup of devices, just like DDR memory does not need a driver. This is where CXL 1.1 played until CXL 2.0 added so much configuration complexity (hotplug, interleave, persistent memory) that it started to need OS help. The Linux PCIe PM code will not notice a difference, but behind the scenes the device, if it is offering coherent memory to the CPU, will be coordinating with the CPU like it was part of the CPU package and not a discrete device. I do not see new PM software enabling required in my reading of "Section 10.0 Power Management" of the CXL specification.
Quoted 29 lines
Quoted 18 lines
So I hope you see CXL's need to create some PCIe-symbiotic objects in order to compose CXL objects (like interleaved memory regions) that have no PCIe analog.
Quoted 5 lines
Hotplug is more central to PCI than NPEM is, but NPEM is a little bit like PCIe native hotplug in concept: hotplug has a few registers that control downstream indicators, interlock, and power controller; NPEM has registers that control downstream indicators.
Quoted 8 lines
Both are prescribed by the PCIe spec and presumably designed to work alongside the usual device-specific drivers for bridges, SSDs, etc.
I would at least explore the idea of doing common support by integrating NPEM into the PCI core. There would have to be some hook for the enclosure-specific bits, but I think it's fair for the details of sending commands and polling for command completed to be part of the PCI core.
Quoted 6 lines
The problem with NPEM compared to hotplug LED signaling is that it has the strange property that an NPEM higher in the topology might override one lower in the topology. This is to support things like NVME enclosures where the NVME device itself may have an NPEM instance, but that instance is of course not suitable for signaling that the device is not present.
I would envision the PCI core providing only a bare-bones "send this command and wait for it to complete" sort of interface. Everything else, including deciding which device to use, would go elsewhere.
Quoted 7 lines
Quoted 3 lines
So, instead the system may be designed to have an NPEM in the upstream switch port and disable the NPEM instances in the device. Platform firmware decides which NPEM is in use.
Since you didn't mention a firmware interface to communicate which NPEM is in use, I assume firmware does this by preventing other devices from advertising NPEM support?
That's also my assumption. If the OS sees a disabled NPEM in the topology it just needs to assume firmware knew what it was doing when it disabled it. I wish NPEM was better specified than "trust firmware to do the right thing via an ambiguous signal".
Quoted 11 lines
Quoted 7 lines
It also has the "fun" property of additionally being overridden by ACPI.
...
One of the nice properties of the auxiliary organization is well defined module boundaries. Will NPEM in the PCI core now forceCONFIG_ENCLOSURE_SERVICESto be built-in? That seems an unwanted side effect to me.If the PCI core provides only the mechanism, and the smarts of NPEM are in something analogous to
drivers/scsi/ses.c, I don't think it would forceCONFIG_ENCLOSURE_SERVICESto be built-in.
What is that dynamic thing that glues CONFIG_ENCLOSURE_SERVICES to the PCI core that also does not require statically linking that glue to every driver that wants to talk to NPEM? I don't mind that the base hardware access mechanism library is in the PCI core, but what does NVME and the CXL memory expander driver register to get NPEM service and associate their block / memory device with an enclosure slot? To me that glue for these one-off ancillary features is what aux-bus is all about, but I'm open to it not being aux-bus in the end. Maybe Stuart has a proposal here?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
DOE *is* specified by PCIe, at least in terms of the data transfer protocol (interrupt usage, read/write mailbox, etc). I think that, and the fact that it's not specific to CXL, means we need some kind of PCI core interface to do the transfers.
Quoted 8 lines
Quoted 3 lines
DOE transfer code belongs in
drivers/pci/period, but does it really need to be indrivers/pci/built-in.afor everyone regardless of whether it is being used or not?I think my opinion would depend on how small the DOE transfer code could be made and how much it would complicate things to make it a module. And also how we could enforce whatever mutual exclusion we need to make it safe.
At least for the mutual exclusion aspect I'm thinking of typical request_region() style exclusion where the aux-driver claims the configuration address register range.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 2 lines · Dan Williams
Quoted 1 line · Bjorn Helgaas
On Thu, Dec 02, 2021 at 05:38:17PM -0800, Dan Williams wrote:
I'm cutting out a bunch of details, not because they're unimportant, but because I don't know enough yet for them to make sense to me.
Quoted 8 lines
Quoted 2 lines
I guess software uses the CXL DVSEC to distinguish a CXL device from a PCIe device, at least for 00.0.
driver/cxl/pci.cattaches to: "PCI_DEVICE_CLASS((PCI_CLASS_MEMORY_CXL << 8 | CXL_MEMORY_PROGIF), ~0)"I am not aware of any restriction for that class code to appear at function0?
Maybe it's not an actual restriction; I'm reading CXL r2.0, sec 8.1.3, where it says:
The PCIe configuration space of Device 0, Function 0 shall include the CXL PCI Express Designated Vendor-Specific Extended Capability (DVSEC) as shown in Figure 126. The capability, status and control fields in Device 0, Function 0 DVSEC control the CXL functionality of the entire CXL device.
...
Software may use the presence of this DVSEC to differentiate between a CXL device and a PCIe device. As such, a standard PCIe device must not expose this DVSEC.
Sections 9.11.5 and 9.12.2 also talk about looking for CXL DVSEC on dev 0, func 0 to identify CXL devices.
Quoted 5 lines
Quoted 2 lines
Not sure about non-Dev 0 Func 0 devices; maybe this implies other functions in the same device are part of the same CXL device?
DVSEC is really only telling us the layout of the MMIO register space. ...
And DVSEC also apparently tells us that "this is a CXL device, not just an ordinary PCIe device"? It's not clear to me how you identify other PCIe functions that are also part of the same CXL device.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The fact that the PCI core remains blissfully unaware of all these interdependencies is a feature because CXL is effectively a super-set of PCIe for fast-path CXL.mem operation, even though it is derivative for enumeration and slow-path manageability.
Quoted 7 lines
Quoted 2 lines
I don't know how blissfully unaware PCI can be. Can a user remove a PCIe device that's part of a CXL device via sysfs?
Yes. If that PCIe device is an endpoint then the corresponding "mem" driver will get a ->remove() event because it is registered as a child of that parent PCIe device. That in turn will tear down the relevant part of the CXL port topology.
The CXL device may include several PCIe devices. "mem" is a CXL driver that's bound to one of them (?) Is that what you mean by "mem" being a "child of the the parent PCIe device"?
Quoted 4 lines
However, a gap (deliberate design choice?) in the PCI hotplug implementation I currently see is an ability for an endpoint PCI driver to dynamically deny hotplug requests based on the state of the device. ...
PCI allows surprise remove, so drivers generally can't deny hot unplugs. PCIe *does* provide for an Electromechanical Interlock (see PCI_EXP_SLTCTL_EIC), but we don't currently support it.
Quoted 6 lines
Quoted 1 line
Is the PCIe device available for drivers to claim?
drivers/cxl/pci.c*is* a native PCI driver for CXL memory expanders. You might be thinking of CXL accelerator devices, where the CXL.cache and CXL.mem capabilities are incremental capabilities for the accelerator. ...
No, I'm not nearly sophisticated enough to be thinking of specific types of CXL things :)
Quoted 5 lines
Quoted 2 lines
Do we need coordination between
cxl_driver_register()andpci_register_driver()?They do not collide, and I think this goes back to the concern about whether
drivers/cxl/is scanning for all CXL DVSECs. ...
Sorry, I don't remember what this concern was, and I don't know why they don't collide. I *would* know that if I knew that the set of things cxl_driver_register() binds to doesn't intersect the set of pci_devs, but it's not clear to me what those things are.
The PCI core enumerates devices by doing config reads of the Vendor ID for every possible bus and device number. It allocs a pci_dev for each device it finds, and those are the things pci_register_driver() binds drivers to based on Vendor ID, Device ID, etc.
How does CXL enumeration work? Do you envision it being integrated into PCI enumeration? Does it build a list/tree/etc of cxl_devs?
cxl_driver_register() associates a driver with something. What exactly is the thing the driver is associated with? A pci_dev? A cxl_dev? Some kind of aggregate CXL device composed of several PCIe devices?
I expected cxl_driver.probe() to take a "struct cxl_dev *" or similar, but it takes a "struct device *". I'm trying to apply my knowledge of how other buses work in Linux, but obviously it's not working very well.
Quoted 13 lines
Quoted 1 line
Does CXL impose new constraints on PCI power management?
Recall that CXL is built such that it could be supported by a legacy operating system where platform firmware owns the setup of devices, just like DDR memory does not need a driver. This is where CXL 1.1 played until CXL 2.0 added so much configuration complexity (hotplug, interleave, persistent memory) that it started to need OS help. The Linux PCIe PM code will not notice a difference, but behind the scenes the device, if it is offering coherent memory to the CPU, will be coordinating with the CPU like it was part of the CPU package and not a discrete device. I do not see new PM software enabling required in my reading of "Section 10.0 Power Management" of the CXL specification.
So if Linux PM decides to suspend a PCIe device that's part of a CXL device and put it in D3hot, this is not a problem for CXL? I guess if a CXL driver binds to the PCIe device, it can control what PM will do. But I thought CXL drivers would bind to a CXL thing, not a PCIe thing.
I see lots of mentions of LTR in sec 10, which really scares me because I'm pretty confident that Linux handling of LTR is broken, and I have no idea how to fix it.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
So, instead the system may be designed to have an NPEM in the upstream switch port and disable the NPEM instances in the device. Platform firmware decides which NPEM is in use.
Quoted 8 lines
Quoted 3 lines
Since you didn't mention a firmware interface to communicate which NPEM is in use, I assume firmware does this by preventing other devices from advertising NPEM support?
That's also my assumption. If the OS sees a disabled NPEM in the topology it just needs to assume firmware knew what it was doing when it disabled it. I wish NPEM was better specified than "trust firmware to do the right thing via an ambiguous signal".
If we enumerate a device with a capability that is disabled, we normally don't treat that as a signal that it cannot be enabled. There are lots of enable bits in PCI capabilities, and I don't know of any cases where Linux assumes "Enable == 0" means "don't use this feature." Absent some negotiation like _OSC or some documented restriction, e.g., in the PCI Firmware spec, Linux normally just enables features when it decides to use them.
Bjorn
Re: [PATCH 20/23] cxl/port: Introduce a port driver
Quoted 30 lines · Bjorn Helgaas
Quoted 2 lines · Dan Williams
Quoted 1 line · Bjorn Helgaas
On Thu, Dec 02, 2021 at 05:38:17PM -0800, Dan Williams wrote:
I'm cutting out a bunch of details, not because they're unimportant, but because I don't know enough yet for them to make sense to me.
Quoted 8 lines
Quoted 2 lines
I guess software uses the CXL DVSEC to distinguish a CXL device from a PCIe device, at least for 00.0.
driver/cxl/pci.cattaches to: "PCI_DEVICE_CLASS((PCI_CLASS_MEMORY_CXL << 8 | CXL_MEMORY_PROGIF), ~0)"I am not aware of any restriction for that class code to appear at function0?
Maybe it's not an actual restriction; I'm reading CXL r2.0, sec 8.1.3, where it says:
The PCIe configuration space of Device 0, Function 0 shall include the CXL PCI Express Designated Vendor-Specific Extended Capability (DVSEC) as shown in Figure 126. The capability, status and control fields in Device 0, Function 0 DVSEC control the CXL functionality of the entire CXL device.
...
Software may use the presence of this DVSEC to differentiate between a CXL device and a PCIe device. As such, a standard PCIe device must not expose this DVSEC.
Quoted 2 lines
Sections 9.11.5 and 9.12.2 also talk about looking for CXL DVSEC on dev 0, func 0 to identify CXL devices.
Ah, I did not internalize that when reading because if the DVSEC shows up on other functions on the same device it should "just work" as far as Linux is concerned, but I guess it simplifies implementations to constrain where the capability appears.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Not sure about non-Dev 0 Func 0 devices; maybe this implies other functions in the same device are part of the same CXL device?
DVSEC is really only telling us the layout of the MMIO register space. ...
And DVSEC also apparently tells us that "this is a CXL device, not just an ordinary PCIe device"? It's not clear to me how you identify other PCIe functions that are also part of the same CXL device.
I have not encountered any flows where the driver would care if it was enabling another instance of the CXL DVSEC on the same device.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
The fact that the PCI core remains blissfully unaware of all these interdependencies is a feature because CXL is effectively a super-set of PCIe for fast-path CXL.mem operation, even though it is derivative for enumeration and slow-path manageability.
I don't know how blissfully unaware PCI can be. Can a user remove a PCIe device that's part of a CXL device via sysfs?
Quoted 8 lines
Quoted 4 lines
Yes. If that PCIe device is an endpoint then the corresponding "mem" driver will get a ->remove() event because it is registered as a child of that parent PCIe device. That in turn will tear down the relevant part of the CXL port topology.
The CXL device may include several PCIe devices. "mem" is a CXL driver that's bound to one of them (?) Is that what you mean by "mem" being a "child of the the parent PCIe device"?
Right, cxl_pci_probe() does device_add() of a "mem" device on the cxl_bus_type, and cxl_pci_remove() does device_del() of that same child device. Just like xhci_pci_probe() arranges for device_add() of a child USB device on the usb_dev_type.
Quoted 9 lines
Quoted 5 lines
However, a gap (deliberate design choice?) in the PCI hotplug implementation I currently see is an ability for an endpoint PCI driver to dynamically deny hotplug requests based on the state of the device. ...
PCI allows surprise remove, so drivers generally can't deny hot unplugs. PCIe *does* provide for an Electromechanical Interlock (see
PCI_EXP_SLTCTL_EIC), but we don't currently support it.
Ok, I guess people will just need to be careful then.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Is the PCIe device available for drivers to claim?
drivers/cxl/pci.c*is* a native PCI driver for CXL memory expanders. You might be thinking of CXL accelerator devices, where the CXL.cache and CXL.mem capabilities are incremental capabilities for the accelerator. ...No, I'm not nearly sophisticated enough to be thinking of specific types of CXL things :)
Ah, apologies I was reading too much into your line of questions.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Do we need coordination between
cxl_driver_register()andpci_register_driver()?They do not collide, and I think this goes back to the concern about whether
drivers/cxl/is scanning for all CXL DVSECs. ...Sorry, I don't remember what this concern was, and I don't know why they don't collide. I *would* know that if I knew that the set of things
cxl_driver_register()binds to doesn't intersect the set ofpci_devs, but it's not clear to me what those things are.
cxl_driver_register() registers a driver on the cxl_bus_type, it can not bind to devices on the pci_bus_type.
Quoted 7 lines
The PCI core enumerates devices by doing config reads of the Vendor ID for every possible bus and device number. It allocs a
pci_devfor each device it finds, and those are the thingspci_register_driver()binds drivers to based on Vendor ID, Device ID, etc.How does CXL enumeration work? Do you envision it being integrated into PCI enumeration? Does it build a list/tree/etc of
cxl_devs?
The cxl_pci driver, native PCI driver, attaches to endpoints that emit the CXL Memory Device class code. That driver walks up the PCI topology and adds a cxl_port device for each CXL DVSEC it finds in each PCIE Upstream Port up to the hostbridge. Additionally there is a cxl_acpi driver that enumerates platform CXL root resources and is called the 'root' cxl_port. Once root-to-endpoint connectivity is established then the endpoint is considered ready for CXL.mem operation.
Quoted 4 lines
cxl_driver_register()associates a driver with something. What exactly is the thing the driver is associated with? Apci_dev? Acxl_dev? Some kind of aggregate CXL device composed of several PCIe devices?
The aggregate device is a cxl_region composed of 1 or more endpoint (registered by cxl_pci_probe()) devices. Like a RAID array it aggregates multiple devices to produce a new memory range.
Quoted 4 lines
I expected
cxl_driver.probe()to take a "structcxl_dev*" or similar, but it takes a "struct device *". I'm trying to apply my knowledge of how other buses work in Linux, but obviously it's not working very well.
There's no common attributes between "mem" "port" and "region" devices so the drivers just do container_of() on the device and skip defining a generic 'struct cxl_dev'.
Quoted 14 lines
Quoted 13 lines
Quoted 1 line
Does CXL impose new constraints on PCI power management?
Recall that CXL is built such that it could be supported by a legacy operating system where platform firmware owns the setup of devices, just like DDR memory does not need a driver. This is where CXL 1.1 played until CXL 2.0 added so much configuration complexity (hotplug, interleave, persistent memory) that it started to need OS help. The Linux PCIe PM code will not notice a difference, but behind the scenes the device, if it is offering coherent memory to the CPU, will be coordinating with the CPU like it was part of the CPU package and not a discrete device. I do not see new PM software enabling required in my reading of "Section 10.0 Power Management" of the CXL specification.
Quoted 4 lines
So if Linux PM decides to suspend a PCIe device that's part of a CXL device and put it in D3hot, this is not a problem for CXL? I guess if a CXL driver binds to the PCIe device, it can control what PM will do. But I thought CXL drivers would bind to a CXL thing, not a PCIe thing.
Recall that this starts with a native PCI driver for endpoints, that driver can coordinate PM for its children on the CXL bus if it is needed.
Quoted 3 lines
I see lots of mentions of LTR in sec 10, which really scares me because I'm pretty confident that Linux handling of LTR is broken, and I have no idea how to fix it.
Ok, I will keep that in mind.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
So, instead the system may be designed to have an NPEM in the upstream switch port and disable the NPEM instances in the device. Platform firmware decides which NPEM is in use.
Quoted 8 lines
Quoted 3 lines
Since you didn't mention a firmware interface to communicate which NPEM is in use, I assume firmware does this by preventing other devices from advertising NPEM support?
That's also my assumption. If the OS sees a disabled NPEM in the topology it just needs to assume firmware knew what it was doing when it disabled it. I wish NPEM was better specified than "trust firmware to do the right thing via an ambiguous signal".
Quoted 7 lines
If we enumerate a device with a capability that is disabled, we normally don't treat that as a signal that it cannot be enabled. There are lots of enable bits in PCI capabilities, and I don't know of any cases where Linux assumes "Enable == 0" means "don't use this feature." Absent some negotiation like
_OSCor some documented restriction, e.g., in the PCI Firmware spec, Linux normally just enables features when it decides to use them.
If the LEDs are not routed to a given NPEM instance, no amount of enabling can get that instance operational.
Re: [PATCH 20/23] cxl/port: Introduce a port driver
In reply to Ben WidawskyQuoted 8 lines · Ben Widawsky
Quoted 7 lines · Bjorn Helgaas
Quoted 4 lines · Ben Widawsky
2. Hostbridge port.
...
It has n downstream ports,
each of which are known as CXL 2.0 root ports.
Quoted 7 lines
Quoted 2 lines
This sounds like a "host bridge port *contains* these root ports." That doesn't sound right.
What sounds better? A CXL 2.0 Root Port is CXL capabilities on top of the PCIe component which has the
PCI_EXP_TYPE_ROOT_PORTcap. In my mental model, a host bridge does contain the root ports. Perhaps I am wrong about that?
"A host bridge contains the root ports" makes sense to me. "A host bridge *port* contains root ports" does not.
The PCIe spec would describe this as a "Root Complex may support one or more [Root] Ports" (see PCIe r5.0, sec 1.3.1).
In PCIe, a "Port" is "an interface between a component and a PCI Express Link." It doesn't contain other Ports.
Sounds like CXL is the same here, and using the same terminology (assuming that's what the CXL spec does) will reduce confusion.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
3. Switch port. A switch port is similar to a hostbridge port except register access is defined in the CXL specification in a platform agnostic way. The downstream ports for a switch are simply known as downstream ports. The enumeration of these are entirely contained within
cxl_port.
Quoted 7 lines
Quoted 2 lines
In PCIe, "Downstream Port" includes both Root Ports and Switch Downstream Ports. It sounds like that's not the case for CXL?
In CXL 2.0, it's fairly close to true that switch downstream ports and root ports are equivalent. From today's driver perspective they are equivalent. Root ports have some capabilities which switch downstream ports do not.
Same as PCIe.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
4. Endpoint port. Endpoint ports are similar to switch ports with the exception that they have no downstream ports, only the underlying media on the device. The enumeration of these are started by
cxl_pci, and completed bycxl_port.Does CXL use an "Upstream Port" concept similar to PCIe? In PCIe, "Upstream Port" includes both Switch Upstream Ports and the Upstream Port on an Endpoint.
Quoted 5 lines
Not really. Endpoints aren't known as ports in the spec and they have a decent amount of divergence from upstream ports. The main area where they do converge is in the memory decoding capabilities. In fact, it might come to a point where we find adding an endpoint as a port does not make sense, but for now it does.
Since a PCIe "Port" refers to the interface between a component and a Link, PCIe Endpoints have Upstream Ports just like switches do. I'm guessing CXL is the same.
The part about "endpoint ports have no downstream ports" is what doesn't read well to me because ports don't have other ports.
This section is about the four types of ports in a system. I'm trying to match those up with spec terms, either PCIe or CXL. It sounds like you intend bullet 3 to include both Switch Upstream Ports and Switch Downstream Ports, and bullet 4 to be only Endpoint Ports (which are Upstream Ports).
Quoted 5 lines
Quoted 4 lines
I hope this driver is not modeled on the PCI portdrv. IMO, that was a design error, and the "port service drivers" (PME, hotplug, AER, etc) should be directly integrated into the PCI core instead of pretending to be independent drivers.
Quoted 7 lines
I'd like to understand a bit about why you think it was a design error. The port driver is intended to be a port services driver, however I see the services provided as quite different than the ones you mention. The primary service
cxl_portwill provide from here on out is the ability to manage HDM decoder resources for a port. Other independent drivers that want to use HDM decoder resources would rely on the port driver for this.
I'll continue this part in the later part of the thread.
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
+ * * - Switch
+ * - Switch Upstream Port
+ * - Switch Downstream Port
+ * * - Endpoint
+ * - Endpoint Port
+ * - N/AWhat does "N/A" mean here? Is it telling us something useful?
Quoted 7 lines
This gets rendered into a table that looks like the following:
| component | upstream | downstream | | --------- | -------- | ---------- | | Hostbridge | ACPI0016 | Root Port | | Switch | Switch Upstream Port | Switch Downstream Port | | Endpoint | Endpoint Port | N/A |
Makes sense, thanks. I didn't know how to read the ReST and thought this was just a list, where N/A wouldn't make much sense.
Bjorn
PATCH 21/23 cxl: Unify port enumeration for decoders
In reply to Ben WidawskyThe port driver exists to do proper enumeration of the component registers for ports, including HDM decoder resources. Any port which follows the CXL specification to implement HDM decoder registers should be handled by the port driver. This includes host bridge registers that are currently handled within the cxl_acpi driver.
In moving the responsibility from cxl_acpi to cxl_port, three primary things are accomplished here:
1. Multi-port host bridges are now handled by the port driver
2. Single port host bridges are handled by the port driver
3. Single port switches without component registers will be handled by the port driver.
While it's tempting to remove decoder APIs from cxl_core entirely, it is still required that platform specific drivers are able to add decoders which aren't specified in CXL 2.0+. An example of this is the CFMWS parsing which is implementing in cxl_acpi.
---
Changes since RFCv2:
- Renamed subject
- Reworded commit message
- Handle *all* host bridge port enumeration in cxl_port (Dan)
- Handle passthrough decoding in cxl_port
--- drivers/cxl/acpi.c | 41 +++----------------------------- drivers/cxl/core/bus.c | 6 +++-- drivers/cxl/cxl.h | 2 ++ drivers/cxl/port.c | 54 +++++++++++++++++++++++++++++++++++++++++- 4 files changed, 62 insertions(+), 41 deletions(-)
drivers/cxl/acpi.c +3 -38
@@ -210,8 +210,6 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct acpi_device *bridge =to_cxl_host_bridge (host, match);struct acpi_pci_root *pci_root;struct cxl_walk_context ctx; -int single_port_map[1 ], rc; -struct cxl_decoder *cxld;struct cxl_dport *dport;struct cxl_port *port;
@@ -245,43 +243,9 @@ static int add_host_bridge_uport(struct device *match, void *arg)return -ENODEV ;if (ctx.error)return ctx.error; -if (ctx.count >1 ) -return 0 ; -/* TODO: Scan CHBCR for HDM Decoder resources */ - -/* -* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability -* Structure) single ported host-bridges need not publish a decoder -* capability when a passthrough decode can be assumed, i.e. all -* transactions that the uport sees are claimed and passed to the single -* dport. Disable the range until the first CXL region is enumerated / -* activated. -*/ - cxld =cxl_decoder_alloc (port,1 ); -if (IS_ERR (cxld)) -return PTR_ERR (cxld); - - cxld->interleave_ways =1 ; - cxld->interleave_granularity =PAGE_SIZE ; - cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); - -device_lock (&port->dev); - dport =list_first_entry (&port->dports,typeof (*dport), list); -device_unlock (&port->dev); - - single_port_map[0 ] = dport->port_id; - - rc =cxl_decoder_add (cxld, single_port_map); -if (rc) -put_device (&cxld->dev); -else - rc =cxl_decoder_autoremove (host, cxld); - -if (rc ==0 ) -dev_dbg (host,"add: %s\n" ,dev_name (&cxld->dev)); -return rc; +/* Host bridge ports are enumerated by the port driver. */ +return 0 ; }struct cxl_chbs_context {
@@ -448,3 +412,4 @@ module_platform_driver(cxl_acpi_driver);MODULE_LICENSE ("GPL v2" );MODULE_IMPORT_NS (CXL );MODULE_IMPORT_NS (ACPI ); +MODULE_SOFTDEP ("pre: cxl_port" );
drivers/cxl/core/bus.c +4 -2
@@ -62,7 +62,7 @@ void cxl_unregister_topology_host(struct device *host) }EXPORT_SYMBOL_NS_GPL (cxl_unregister_topology_host,CXL ); -static struct device *get_cxl_topology_host (void ) +struct device *get_cxl_topology_host (void ) {down_read (&topology_host_sem);if (cxl_topology_host)
@@ -70,12 +70,14 @@ static struct device *get_cxl_topology_host(void)up_read (&topology_host_sem);return NULL ; } +EXPORT_SYMBOL_NS_GPL (get_cxl_topology_host,CXL ); -static void put_cxl_topology_host (struct device *dev) +void put_cxl_topology_host (struct device *dev) {WARN_ON (dev != cxl_topology_host);up_read (&topology_host_sem); } +EXPORT_SYMBOL_NS_GPL (put_cxl_topology_host,CXL );static int decoder_match (struct device *dev,void *data) {
drivers/cxl/cxl.h +2 -0
@@ -170,6 +170,8 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,int cxl_register_topology_host (struct device *host);void cxl_unregister_topology_host (struct device *host); +struct device *get_cxl_topology_host (void ); +void put_cxl_topology_host (struct device *dev);/* * cxl_decoder flags that define the type of memory / devices this
drivers/cxl/port.c +53 -1
65 lines not on this page · raw message
Signature · 1 line
2.34.0
Re: [PATCH 21/23] cxl: Unify port enumeration for decoders
Quoted 13 lines · On Fri, 19 Nov 2021 16:02:48 -0800 Ben Widawsky <[email protected]> wrote:
The port driver exists to do proper enumeration of the component registers for ports, including HDM decoder resources. Any port which follows the CXL specification to implement HDM decoder registers should be handled by the port driver. This includes host bridge registers that are currently handled within the
cxl_acpidriver.In moving the responsibility from
cxl_acpitocxl_port, three primary things are accomplished here:
1. Multi-port host bridges are now handled by the port driver
2. Single port host bridges are handled by the port driver
3. Single port switches without component registers will be handled by the port driver.
Quoted 6 lines
While it's tempting to remove decoder APIs from
cxl_coreentirely, it is still required that platform specific drivers are able to add decoders which aren't specified in CXL 2.0+. An example of this is the CFMWS parsing which is implementing incxl_acpi.Signed-off-by Ben Widawsky <[email protected]>
One trivial suggestion inline, but looks fine to me otherwise.
Quoted 148 lines
---
Changes since RFCv2:
- Renamed subject
- Reworded commit message
- Handle *all* host bridge port enumeration incxl_port(Dan)
- Handle passthrough decoding incxl_port--- drivers/cxl/acpi.c | 41 +++----------------------------- drivers/cxl/core/bus.c | 6 +++-- drivers/cxl/cxl.h | 2 ++ drivers/cxl/port.c | 54 +++++++++++++++++++++++++++++++++++++++++- 4 files changed, 62 insertions(+), 41 deletions(-)drivers/cxl/acpi.c +3
-38@@ -210,8 +210,6 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct acpi_device *bridge =to_cxl_host_bridge (host, match);struct acpi_pci_root *pci_root;struct cxl_walk_context ctx; -int single_port_map[1 ], rc; -struct cxl_decoder *cxld;struct cxl_dport *dport;struct cxl_port *port;@@ -245,43 +243,9 @@ static int add_host_bridge_uport(struct device *match, void *arg)return -ENODEV ;if (ctx.error)return ctx.error; -if (ctx.count >1 ) -return 0 ; -/* TODO: Scan CHBCR for HDM Decoder resources */ - -/* -* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability -* Structure) single ported host-bridges need not publish a decoder -* capability when a passthrough decode can be assumed, i.e. all -* transactions that the uport sees are claimed and passed to the single -* dport. Disable the range until the first CXL region is enumerated / -* activated. -*/ - cxld =cxl_decoder_alloc (port,1 ); -if (IS_ERR (cxld)) -return PTR_ERR (cxld); - - cxld->interleave_ways =1 ; - cxld->interleave_granularity =PAGE_SIZE ; - cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); - -device_lock (&port->dev); - dport =list_first_entry (&port->dports,typeof (*dport), list); -device_unlock (&port->dev); - - single_port_map[0 ] = dport->port_id; - - rc =cxl_decoder_add (cxld, single_port_map); -if (rc) -put_device (&cxld->dev); -else - rc =cxl_decoder_autoremove (host, cxld); - -if (rc ==0 ) -dev_dbg (host,"add: %s\n" ,dev_name (&cxld->dev)); -return rc; +/* Host bridge ports are enumerated by the port driver. */ +return 0 ; }struct cxl_chbs_context {@@ -448,3 +412,4 @@ module_platform_driver(cxl_acpi_driver);MODULE_LICENSE ("GPL v2" );MODULE_IMPORT_NS (CXL );MODULE_IMPORT_NS (ACPI ); +MODULE_SOFTDEP ("pre: cxl_port" );drivers/cxl/core/bus.c +4
-2@@ -62,7 +62,7 @@ void cxl_unregister_topology_host(struct device *host) }EXPORT_SYMBOL_NS_GPL (cxl_unregister_topology_host,CXL ); -static struct device *get_cxl_topology_host (void ) +struct device *get_cxl_topology_host (void ) {down_read (&topology_host_sem);if (cxl_topology_host)@@ -70,12 +70,14 @@ static struct device *get_cxl_topology_host(void)up_read (&topology_host_sem);return NULL ; } +EXPORT_SYMBOL_NS_GPL (get_cxl_topology_host,CXL ); -static void put_cxl_topology_host (struct device *dev) +void put_cxl_topology_host (struct device *dev) {WARN_ON (dev != cxl_topology_host);up_read (&topology_host_sem); } +EXPORT_SYMBOL_NS_GPL (put_cxl_topology_host,CXL );static int decoder_match (struct device *dev,void *data) {drivers/cxl/cxl.h +2
-0@@ -170,6 +170,8 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,int cxl_register_topology_host (struct device *host);void cxl_unregister_topology_host (struct device *host); +struct device *get_cxl_topology_host (void ); +void put_cxl_topology_host (struct device *dev);/* * cxl_decoder flags that define the type of memory / devices this drivers/cxl/port.c +20
-0@@ -233,12 +233,64 @@ static int enumerate_hdm_decoders(struct cxl_port *port,return 0 ; } +/* +* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability Structure) +* single ported host-bridges need not publish a decoder capability when a +* passthrough decode can be assumed, i.e. all transactions that the uport sees +* are claimed and passed to the single dport. Disable the range until the first +* CXL region is enumerated / activated. +*/ +static int add_passthrough_decoder (struct cxl_port *port) +{ +int single_port_map[1 ], rc; +struct cxl_decoder *cxld; +struct cxl_dport *dport; + +device_lock_assert (&port->dev); + + cxld =cxl_decoder_alloc (port,1 ); +if (IS_ERR (cxld)) +return PTR_ERR (cxld); + + cxld->interleave_ways =1 ;
drivers/cxl/port.c static int enumerate_hdm_decoders(struct cxl_port *port, · line 265
Quoted 10 lines
drivers/cxl/port.c +10
-0+ cxld->interleave_granularity =PAGE_SIZE ; + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); + + dport =list_first_entry (&port->dports,typeof (*dport), list); + single_port_map[0 ] = dport->port_id; + + rc =cxl_decoder_add_locked (cxld, single_port_map); +if (rc) +put_device (&cxld->dev);
I would handle this error path entirely here, or use a goto rather than messing up the good path with conditionals on each element, particularly as there isn't much to do in the error paths. I guess this might get more complicated in later patches though.
Obviously that tidy up would make this more complex than simply moving the code. (I might have commented on this before, but too long ago to remember ;)
if (rc) {
put_device(&cxld->dev);
return rc;
}
rc = cxl_decoder...
if (rc)
return rc;
dev_dbg(..
return 0;Quoted 9 lines
drivers/cxl/port.c +9
-0+else + rc =cxl_decoder_autoremove (&port->dev, cxld); + +if (rc ==0 ) +dev_dbg (&port->dev,"add: %s\n" ,dev_name (&cxld->dev)); + +return rc; +} +
Re: [PATCH 21/23] cxl: Unify port enumeration for decoders
Quoted 193 lines · Jonathan Cameron
Quoted 13 lines · On Fri, 19 Nov 2021 16:02:48 -0800 Ben Widawsky <[email protected]> wrote:
The port driver exists to do proper enumeration of the component registers for ports, including HDM decoder resources. Any port which follows the CXL specification to implement HDM decoder registers should be handled by the port driver. This includes host bridge registers that are currently handled within the
cxl_acpidriver.In moving the responsibility from
cxl_acpitocxl_port, three primary things are accomplished here:
1. Multi-port host bridges are now handled by the port driver
2. Single port host bridges are handled by the port driver
3. Single port switches without component registers will be handled by the port driver.Quoted 6 lines
While it's tempting to remove decoder APIs from
cxl_coreentirely, it is still required that platform specific drivers are able to add decoders which aren't specified in CXL 2.0+. An example of this is the CFMWS parsing which is implementing incxl_acpi.Signed-off-by Ben Widawsky <[email protected]>One trivial suggestion inline, but looks fine to me otherwise.
Reviewed-by Jonathan Cameron <[email protected]>Quoted 148 lines
---
Changes since RFCv2:
- Renamed subject
- Reworded commit message
- Handle *all* host bridge port enumeration incxl_port(Dan)
- Handle passthrough decoding incxl_port--- drivers/cxl/acpi.c | 41 +++----------------------------- drivers/cxl/core/bus.c | 6 +++-- drivers/cxl/cxl.h | 2 ++ drivers/cxl/port.c | 54 +++++++++++++++++++++++++++++++++++++++++- 4 files changed, 62 insertions(+), 41 deletions(-)drivers/cxl/acpi.c +3
-38@@ -210,8 +210,6 @@ static int add_host_bridge_uport(struct device *match, void *arg)struct acpi_device *bridge =to_cxl_host_bridge (host, match);struct acpi_pci_root *pci_root;struct cxl_walk_context ctx; -int single_port_map[1 ], rc; -struct cxl_decoder *cxld;struct cxl_dport *dport;struct cxl_port *port;@@ -245,43 +243,9 @@ static int add_host_bridge_uport(struct device *match, void *arg)return -ENODEV ;if (ctx.error)return ctx.error; -if (ctx.count >1 ) -return 0 ; -/* TODO: Scan CHBCR for HDM Decoder resources */ - -/* -* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability -* Structure) single ported host-bridges need not publish a decoder -* capability when a passthrough decode can be assumed, i.e. all -* transactions that the uport sees are claimed and passed to the single -* dport. Disable the range until the first CXL region is enumerated / -* activated. -*/ - cxld =cxl_decoder_alloc (port,1 ); -if (IS_ERR (cxld)) -return PTR_ERR (cxld); - - cxld->interleave_ways =1 ; - cxld->interleave_granularity =PAGE_SIZE ; - cxld->target_type =CXL_DECODER_EXPANDER ; - cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); - -device_lock (&port->dev); - dport =list_first_entry (&port->dports,typeof (*dport), list); -device_unlock (&port->dev); - - single_port_map[0 ] = dport->port_id; - - rc =cxl_decoder_add (cxld, single_port_map); -if (rc) -put_device (&cxld->dev); -else - rc =cxl_decoder_autoremove (host, cxld); - -if (rc ==0 ) -dev_dbg (host,"add: %s\n" ,dev_name (&cxld->dev)); -return rc; +/* Host bridge ports are enumerated by the port driver. */ +return 0 ; }struct cxl_chbs_context {@@ -448,3 +412,4 @@ module_platform_driver(cxl_acpi_driver);MODULE_LICENSE ("GPL v2" );MODULE_IMPORT_NS (CXL );MODULE_IMPORT_NS (ACPI ); +MODULE_SOFTDEP ("pre: cxl_port" );drivers/cxl/core/bus.c +4
-2@@ -62,7 +62,7 @@ void cxl_unregister_topology_host(struct device *host) }EXPORT_SYMBOL_NS_GPL (cxl_unregister_topology_host,CXL ); -static struct device *get_cxl_topology_host (void ) +struct device *get_cxl_topology_host (void ) {down_read (&topology_host_sem);if (cxl_topology_host)@@ -70,12 +70,14 @@ static struct device *get_cxl_topology_host(void)up_read (&topology_host_sem);return NULL ; } +EXPORT_SYMBOL_NS_GPL (get_cxl_topology_host,CXL ); -static void put_cxl_topology_host (struct device *dev) +void put_cxl_topology_host (struct device *dev) {WARN_ON (dev != cxl_topology_host);up_read (&topology_host_sem); } +EXPORT_SYMBOL_NS_GPL (put_cxl_topology_host,CXL );static int decoder_match (struct device *dev,void *data) {drivers/cxl/cxl.h +2
-0@@ -170,6 +170,8 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,int cxl_register_topology_host (struct device *host);void cxl_unregister_topology_host (struct device *host); +struct device *get_cxl_topology_host (void ); +void put_cxl_topology_host (struct device *dev);/* * cxl_decoder flags that define the type of memory / devices this drivers/cxl/port.c +20
-0@@ -233,12 +233,64 @@ static int enumerate_hdm_decoders(struct cxl_port *port,return 0 ; } +/* +* Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability Structure) +* single ported host-bridges need not publish a decoder capability when a +* passthrough decode can be assumed, i.e. all transactions that the uport sees +* are claimed and passed to the single dport. Disable the range until the first +* CXL region is enumerated / activated. +*/ +static int add_passthrough_decoder (struct cxl_port *port) +{ +int single_port_map[1 ], rc; +struct cxl_decoder *cxld; +struct cxl_dport *dport; + +device_lock_assert (&port->dev); + + cxld =cxl_decoder_alloc (port,1 ); +if (IS_ERR (cxld)) +return PTR_ERR (cxld); + + cxld->interleave_ways =1 ;drivers/cxl/port.c static int enumerate_hdm_decoders(struct cxl_port *port, · line 265
Quoted 10 lines
drivers/cxl/port.c +10
-0+ cxld->interleave_granularity =PAGE_SIZE ; + cxld->target_type =CXL_DECODER_EXPANDER ; + cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); + + dport =list_first_entry (&port->dports,typeof (*dport), list); + single_port_map[0 ] = dport->port_id; + + rc =cxl_decoder_add_locked (cxld, single_port_map); +if (rc) +put_device (&cxld->dev);I would handle this error path entirely here, or use a goto rather than messing up the good path with conditionals on each element, particularly as there isn't much to do in the error paths. I guess this might get more complicated in later patches though.
Obviously that tidy up would make this more complex than simply moving the code. (I might have commented on this before, but too long ago to remember ;)
Quoted 12 lines
if (rc) {
put_device(&cxld->dev);
return rc;
}
rc = cxl_decoder...
if (rc)
return rc;
dev_dbg(..
return 0;Since I changed this in v2 for the port driver, in the last patch, I think it makes sense to do the move, and then cleanup.
Thanks.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/port.c +9
-0+else + rc =cxl_decoder_autoremove (&port->dev, cxld); + +if (rc ==0 ) +dev_dbg (&port->dev,"add: %s\n" ,dev_name (&cxld->dev)); + +return rc; +} +