Skip to content

RFC Introduce CXL 2.0 Emulation

Ben Widawsky qemu-arm qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest25 patches 38 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.include/standard-headers/linux/pci_regs.h+1 -00 replies
2/25hw/pci/cxl: Add a CXL component type (interface)hw/pci/pci.c include/hw/pci/pci.h+18 -00 replies
3/25hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)MAINTAINERS hw/Kconfig hw/cxl/Kconfig +7 more+537 -04 replies
4/25hw/cxl/device: Introduce a CXL device (8.2.8)include/hw/cxl/cxl.h include/hw/cxl/cxl_device.h+194 -03 replies
5/25hw/cxl/device: Implement the CAP array (8.2.8.1-2)hw/cxl/cxl-device-utils.c hw/cxl/meson.build+74 -02 replies
6/25hw/cxl/device: Add device status (8.2.8.3)hw/cxl/cxl-device-utils.c include/hw/cxl/cxl_device.h+60 -343 replies
7/25hw/cxl/device: Implement basic mailbox (8.2.8.4)hw/cxl/cxl-device-utils.c hw/cxl/cxl-mailbox-utils.c hw/cxl/meson.build +2 more+237 -12 replies
8/25hw/cxl/device: Add memory devices (8.2.8.5)hw/cxl/cxl-device-utils.c hw/cxl/cxl-mailbox-utils.c include/hw/cxl/cxl_device.h+109 -23 replies
9/25hw/pxb: Use a type for realizing expandershw/pci-bridge/pci_expander_bridge.c+7 -40 replies
10/25hw/pci/cxl: Create a CXL bus typehw/pci-bridge/pci_expander_bridge.c include/hw/pci/pci_bus.h+15 -10 replies
11/25hw/pxb: Allow creation of a CXL PXB (host bridge)hw/pci-bridge/pci_expander_bridge.c hw/pci/pci.c include/hw/pci/pci.h+78 -23 replies
12/25acpi/pci: Consolidate host bridge setuphw/i386/acpi-build.c+17 -142 replies
13/25hw/pci: Plumb _UID through host bridgeshw/i386/acpi-build.c hw/pci-bridge/pci_expander_bridge.c hw/pci/pci.c +2 more+34 -10 replies
14/25hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)hw/pci-bridge/pci_expander_bridge.c include/hw/cxl/cxl.h+54 -10 replies
15/25acpi/pxb/cxl: Reserve host bridge MMIOhw/i386/acpi-build.c+19 -31 reply
16/25hw/pxb/cxl: Add "windows" for host bridgeshw/pci-bridge/pci_expander_bridge.c include/hw/cxl/cxl.h+62 -42 replies
17/25hw/cxl/rp: Add a root porthw/pci-bridge/Kconfig hw/pci-bridge/cxl_root_port.c hw/pci-bridge/meson.build +2 more+245 -20 replies
18/25hw/cxl/device: Add a memory device (8.2.8.5)hw/core/numa.c hw/i386/pc.c hw/mem/Kconfig +8 more+343 -06 replies
19/25hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)hw/mem/cxl_type3.c+77 -50 replies
20/25acpi/cxl: Add _OSC implementation (9.14.2)hw/acpi/Kconfig hw/acpi/cxl.c hw/acpi/meson.build +2 more+144 -10 replies
21/25acpi/cxl: Introduce a compat-driver UUID for CXL _OSChw/acpi/cxl.c+38 -160 replies
22/25acpi/cxl: Create the CEDT (9.14.1)hw/acpi/cxl.c hw/i386/acpi-build.c hw/pci-bridge/pci_expander_bridge.c +2 more+107 -212 replies
23/25Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)hw/i386/acpi-build.c+19 -00 replies
24/25WIP: i386/cxl: Initialize a host bridgehw/arm/virt.c hw/core/machine.c hw/i386/acpi-build.c +5 more+44 -10 replies
25/25qtest/cxl: Add very basic sanity teststests/qtest/cxl-test.c tests/qtest/meson.build+97 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/25] Introduce CXL 2.0 Emulation

Introduce emulation of Compute Express Link 2.0, which was released today at https://www.computeexpresslink.org/.

I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0

The emulation has been critical to get the Linux enabling started
(https://lore.kernel.org/linux-cxl/), it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories. I am new to QEMU development, so please forgive and point me in the right direction if I severely misinterpreted where a piece of infrastructure belongs.

Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).

CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).

We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs

As mentioned above there are three benefits to carrying this enabling in upstream QEMU:

1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.

2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (nfit_test). In retrospect a QEMU emulation of nfit_test capabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.

3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?

There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity

The flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.

The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-11: Create the concept of a CXL bus and plumb into PXB 12-16: Implement host bridges
17: Implement a root port
18: Implement a memory device
19: Implement HDM decoders
20-24: ACPI bits
25: Start working on enabling the main host bridge

Ben Widawsky (23):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5)
hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge) acpi/pci: Consolidate host bridge setup
hw/pci: Plumb _UID through host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2)
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests

Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.

Vishal Verma (1):
acpi/cxl: Introduce a compat-driver UUID for CXL _OSC

 MAINTAINERS                               |   6 +
 hw/Kconfig                                |   1 +
 hw/acpi/Kconfig                           |   5 +
 hw/acpi/cxl.c                             | 198 +++++++++++++
 hw/acpi/meson.build                       |   1 +
 hw/arm/virt.c                             |   1 +
 hw/core/machine.c                         |  26 ++
 hw/core/numa.c                            |   3 +
 hw/cxl/Kconfig                            |   3 +
 hw/cxl/cxl-component-utils.c              | 192 +++++++++++++
 hw/cxl/cxl-device-utils.c                 | 293 +++++++++++++++++++
 hw/cxl/cxl-mailbox-utils.c                | 139 +++++++++
 hw/cxl/meson.build                        |   5 +
 hw/i386/acpi-build.c                      |  87 +++++-
 hw/i386/microvm.c                         |   1 +
 hw/i386/pc.c                              |   2 +
 hw/mem/Kconfig                            |   5 +
 hw/mem/cxl_type3.c                        | 334 ++++++++++++++++++++++
 hw/mem/meson.build                        |   1 +
 hw/meson.build                            |   1 +
 hw/pci-bridge/Kconfig                     |   5 +
 hw/pci-bridge/cxl_root_port.c             | 231 +++++++++++++++
 hw/pci-bridge/meson.build                 |   1 +
 hw/pci-bridge/pci_expander_bridge.c       | 209 +++++++++++++-
 hw/pci-bridge/pcie_root_port.c            |   6 +-
 hw/pci/pci.c                              |  32 ++-
 hw/pci/pcie.c                             |  30 ++
 hw/ppc/spapr.c                            |   2 +
 include/hw/acpi/cxl.h                     |  27 ++
 include/hw/boards.h                       |   2 +
 include/hw/cxl/cxl.h                      |  30 ++
 include/hw/cxl/cxl_component.h            | 181 ++++++++++++
 include/hw/cxl/cxl_device.h               | 199 +++++++++++++
 include/hw/cxl/cxl_pci.h                  | 155 ++++++++++
 include/hw/pci/pci.h                      |  15 +
 include/hw/pci/pci_bridge.h               |  25 ++
 include/hw/pci/pci_bus.h                  |   8 +
 include/hw/pci/pci_ids.h                  |   1 +
 include/standard-headers/linux/pci_regs.h |   1 +
 monitor/hmp-cmds.c                        |  15 +
 qapi/machine.json                         |   1 +
 tests/qtest/cxl-test.c                    |  93 ++++++
 tests/qtest/meson.build                   |   4 +
 43 files changed, 2547 insertions(+), 30 deletions(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/cxl-device-utils.c
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 hw/mem/cxl_type3.c
 create mode 100644 hw/pci-bridge/cxl_root_port.c
 create mode 100644 include/hw/acpi/cxl.h
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_device.h
 create mode 100644 include/hw/cxl/cxl_pci.h
 create mode 100644 tests/qtest/cxl-test.c
Signature · 1 line
2.29.2