Skip to content

Add drivers for CXL ports and mem devices

Ben Widawsky linux-cxl 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest14 patches 7 repliesIn review
  2. v123 patches 98 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/14cxl/core: Add, document, and tighten up decoder APIsdrivers/cxl/acpi.c drivers/cxl/core/bus.c drivers/cxl/cxl.h+79 -121 reply
2/14cxl: Introduce endpoint decodersdrivers/cxl/core/bus.c drivers/cxl/cxl.h+48 -70 replies
3/14cxl/core: Move target population locking to callerdrivers/cxl/core/bus.c drivers/cxl/cxl.h+47 -150 replies
4/14cxl: Introduce topology host registrationdrivers/cxl/acpi.c drivers/cxl/core/bus.c drivers/cxl/cxl.h+73 -73 replies
5/14cxl/core: Store global list of root portsdrivers/cxl/acpi.c drivers/cxl/core/bus.c drivers/cxl/cxl.h +1 more+41 -60 replies
6/14cxl/pci: Cache device DVSEC offsetdrivers/cxl/cxlmem.h drivers/cxl/pci.c+11 -00 replies
7/14cxl: Cache and pass DVSEC rangesdrivers/cxl/cxlmem.h drivers/cxl/pci.c drivers/cxl/pci.h+149 -00 replies
8/14cxl/pci: Implement wait for media activedrivers/cxl/cxlmem.h drivers/cxl/pci.c+60 -00 replies
9/14cxl/pci: Store component register base in cxldsdrivers/cxl/cxlmem.h drivers/cxl/pci.c+13 -00 replies
10/14cxl: Make passthrough decoder init implicitdrivers/cxl/acpi.c drivers/cxl/core/bus.c+8 -60 replies
11/14cxl/port: Introduce a port driverDocumentation/driver-api/cxl/memory-devices.rst drivers/cxl/Kconfig drivers/cxl/Makefile +4 more+442 -91 reply
12/14cxl: Unify port enumeration for decodersdrivers/cxl/acpi.c drivers/cxl/core/bus.c drivers/cxl/cxl.h +1 more+57 -360 replies
13/14cxl/port: Cleanup adding passthrough decodersdrivers/cxl/port.c+2 -70 replies
14/14cxl/mem: Introduce cxl_mem driverDocumentation/ABI/testing/sysfs-bus-cxl Documentation/driver-api/cxl/memory-devices.rst drivers/cxl/Makefile +12 more+578 -191 reply

Reviews trailers per patch

5 of 14 patches have a Reviewed-by or Acked-by.

PatchJonathan CameronDan Williams
1/14R-b
2/14R-b
3/14R-b R-b
4/14R-b
5/14
6/14
7/14
8/14
9/14
10/14
11/14
12/14R-b
13/14
14/14

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 00/14] Add drivers for CXL ports and mem devices

Significant changes since v1 [1]:
- Reworked decoder alloc APIs to be specific to component type
- cxl_mem handles DVSEC checks instead of cxl_port
- buggy switch handling in cxl_mem is #if 0'd out
- cxl_pci will fail if the Device DVSEC can't be found

There were a few improvements from the RFC, v1, and discussions on IRC (goto #cxl on OFTC). Dan has some patches that apply either on top of, or squashed in with these. Those are:
1. Global root port list doesn't need to exist.
2. Port enumeration doesn't rely on PCIe concepts to walk topology
3. Port driver should handle enumerating dports

Here I've also dropped the prep patches which are mostly reviewed. Those are here [2].

What follows is the original cover letter from v1 [1]

[1]: https://lore.kernel.org/linux-cxl/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/
---

This is the first set of patches from the RFC [1] for region creation. The patches enable port enumeration for endpoint devices, and enumeration of decoder resources for ports. In the RFC [1], I felt it necessary to post the consumer of this work, the region driver, so that it was clear why these patches were necessary. Because the region patches patches are less baked, and received no review in the RFC, they are excluded here. If you find yourself unclear about why these patches are interesting, go look at the RFC [1].

Each patch contains the list of changes from RFCv2. IMHO the following are the high level most important changes:
1. Rework cxl_pci to fix mailbox handling and allow for wait media ready.
2. DVSEC range information is passed from cxl_pci and checked

linux-pci is on the Cc since CXL lives in a parallel universe to PCI and some PCI mechanisms are reused here. Feedback from experts in that domain is very welcome.

What was requested and not changed:
1. Dropping global list of root ports.
2. Improving find_parent_cxl_port()

---

Summary
=======

Two new drivers are introduced to support Compute Express Link 2.0 [2] HDM decoder enumeration. While the existing cxl_acpi and cxl_pci drivers already create some of the necessary devices, they did not do full enumeration of decoders, and they did not do port enumeration for switches. Additionally, CXL 2.0 Root Port component registers are now handled as well.

cxl_port
========

The cxl_port driver is implemented within the cxl_port module. While loading of this module is optional, the other new drivers depend, and cxl_acpi depend on it for complete enumeration. The port driver is responsible for all activities around HDM decoder enumeration and programming. Introduced earlier, the concept of a port is an abstraction over CXL components with an upstream port, every host bridge, switch, and endpoint.

cxl_mem
=======

The cxl_mem driver's main job is to walk up the hierarchy to make the determination if it is CXL.mem routed, meaning, all components above it in the hierarchy are participating in the CXL.mem protocol. It is implemented within the cxl_mem module. As the host bridge ports are added by a platform specific driver, such as cxl_acpi, the scope of the mem driver can be reduced to scan for switches and ask cxl_core to work on enumerating them. With this done, the determination as to whether a device is CXL.mem routed can be done simply by checking if the struct device has a driver bound to it.

Results
=======

Running these patches should yield new devices and new drivers under /sys/bus/cxl/devices and /sys/bus/cxl/drivers. For example, in a standard QEMU run, using run_qemu [3]

/sys/bus/cxl/devices (new):
# The host bridge CHBS decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder1.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/decoder1.0 # mem0's decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder2.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2/decoder2.0 # mem1's decoder
lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder3.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3/decoder3.0 # mem0's port
lrwxrwxrwx 1 root root 0 Nov 19 15:23 port2 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2 # mem1's port
lrwxrwxrwx 1 root root 0 Nov 19 15:23 port3 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3

/sys/bus/cxl/drivers:
drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_mem

[1]: https://lore.kernel.org/linux-cxl/[email protected]/T/#t
[2]: https://www.computeexpresslink.org/download-the-specification
[3]: https://github.com/pmem/run_qemu/
drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_port
Ben Widawsky (14):
cxl/core: Add, document, and tighten up decoder APIs
cxl: Introduce endpoint decoders
cxl/core: Move target population locking to caller
cxl: Introduce topology host registration
cxl/core: Store global list of root ports
cxl/pci: Cache device DVSEC offset
cxl: Cache and pass DVSEC ranges
cxl/pci: Implement wait for media active
cxl/pci: Store component register base in cxlds
cxl: Make passthrough decoder init implicit
cxl/port: Introduce a port driver
cxl: Unify port enumeration for decoders
cxl/port: Cleanup adding passthrough decoders
cxl/mem: Introduce cxl_mem driver

 Documentation/ABI/testing/sysfs-bus-cxl       |   9 +
 .../driver-api/cxl/memory-devices.rst         |  14 +
 drivers/cxl/Kconfig                           |  21 +
 drivers/cxl/Makefile                          |   4 +
 drivers/cxl/acpi.c                            |  82 +--
 drivers/cxl/core/Makefile                     |   1 +
 drivers/cxl/core/bus.c                        | 479 ++++++++++++++++--
 drivers/cxl/core/core.h                       |   3 +
 drivers/cxl/core/memdev.c                     |   2 +-
 drivers/cxl/core/pci.c                        | 117 +++++
 drivers/cxl/core/regs.c                       |   6 +-
 drivers/cxl/cxl.h                             |  59 ++-
 drivers/cxl/cxlmem.h                          |  23 +
 drivers/cxl/mem.c                             | 285 +++++++++++
 drivers/cxl/pci.c                             | 200 ++++++++
 drivers/cxl/pci.h                             |  16 +
 drivers/cxl/port.c                            | 368 ++++++++++++++
 tools/testing/cxl/Kbuild                      |   1 +
 tools/testing/cxl/mock_acpi.c                 |   4 +-
 19 files changed, 1589 insertions(+), 105 deletions(-)
 create mode 100644 drivers/cxl/core/pci.c
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/port.c

base-commit: c19a248a84b950ef53c6768751cc917ec2c6b6cf

Signature · 1 line
2.34.1