Skip to contentSkip to conversation

arm/virt: CXL support via pxb_cxl

PATCH Applied Jonathan Cameron 3 participants 9 messages Jul 3, 2025 – Jul 10, 2025 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

v17 cover arm/virt: CXL support via pxb_cxl

v17: Thanks to Eric for review
- Add a comment to the high memory map to reduce the chance of nasty surprises in the future as similar to device_memory, the CXL Fixed Memory Windows are of variable size as so can't be represented by explicit entries in the map.
- Updated a couple of patch descriptions to provide answers to questions Eric raised.

v16:
- Mostly additional documentation and descriptive text in patch titles.
- Update test to reflect changes to other tests.
- Update physmem_max in virt to include the CXL memory.

Updated cover letter

Back in 2022, this series stalled on the absence of a solution to device tree support for PCI Expander Bridges (PXB) and we ended up only having x86 support upstream. I've been carrying the arm64 support out of tree since then, with occasional nasty surprises (e.g. UNIMP + DT issue seen a few weeks ago) and a fair number of fiddly rebases. gitlab.com/jic23/qemu cxl-<latest date>. Will update shortly with this series.

A recent discussion with Peter Maydell indicated that there are various other ACPI only features now, so in general he might be more relaxed about DT support being necessary. The upcoming vSMMUv3 support would run into this problem as well.

I presented the background to the PXB issue at Linaro connect 2022. In short the issue is that PXBs steal MMIO space from the main PCI root bridge. The challenge is knowing how much to steal.

On ACPI platforms, we can rely on EDK2 to perform an enumeration and configuration of the PCI topology and QEMU can update the ACPI tables after EDK2 has done this when it can simply read the space used by the root ports. On device tree, there is no entity to figure out that enumeration so we don't know how to size the stolen region.

Three approaches were discussed:
1) Enumerating in QEMU. Horribly complex and the last thing we want is a 3rd enumeration implementation that ends up out of sync with EDK2 and the kernel (there are frequent issues because of how those existing implementations differ.
2) Figure out how to enumerate in kernel. I never put a huge amount of work into this, but it seemed likely to involve a nasty dance with similar very specific code to that EDK2 is carrying and would very challenging to upstream (given the lack of clarity on real use cases for PXBs and DT).
3) Hack it based on the control we have which is bus numbers. No one liked this but it worked :)

The other little wrinkle would be the need to define full bindings for CXL on DT + implement a fairly complex kernel stack as equivalent in ACPI involves a static table, CEDT, new runtime queries via _DSM and a description of various components. Doable, but so far there is no interest on physical platforms. Worth noting that for now, the QEMU CXL emulation is all about testing and developing the OS stack, not about virtualization (performance is terrible except in some very contrived situations!)

There is only a very simple test in here, because my intent is not to duplicate what we have on x86, but just to do a smoke test that everything is hooked up. In general we need much more comprehensive end to end CXL tests but that requires a reaonsably stable guest software stack. A few people have expressed interest in working on that, but we aren't there yet.

Note that this series has a very different use case to that in the proposed SBSA-ref support:
https://lore.kernel.org/qemu-devel/[email protected]/

SBSA-ref is a good choice if you want a relatively simple mostly fixed configuration. That works well with the limited host system discoverability etc as EDK2 can be build against a known configuration.

My interest with this support in arm/virt is support host software stack development (we have a wide range of contributors, most of whom are working on emulation + the kernel support). I care about the weird corners. As such I need to be able to bring up variable numbers of host bridges, multiple CXL Fixed Memory Windows with varying characteristics (interleave etc), complex NUMA topologies with wierd performance characteristics etc. We can do that on x86 upstream today, or my gitlab tree. Note that we need arm support for some arch specific features in the near future (cache flushing). Doing kernel development with this need for flexibility on SBSA-ref is not currently practical. SBSA-ref CXL support is an excellent thing, just not much use to me for this work.

Also, we are kicking off some work on DCD virtualization, particularly to support inter-host shared memory being presented up into a VM. That will need upstream support on arm64 as it is built on top of the existing CXL emulation to avoid the need for a separate guest software stack.

Note this is TCG only - it is possible to support limited use with KVM but that needs additional patches not yet ready for upstream. The challenge is interleave - and the solution is don't interleave if you want to run with KVM.

Jonathan Cameron (5):
hw/cxl-host: Add an index field to CXLFixedMemoryWindow hw/cxl: Make the CXL fixed memory windows devices. hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
pxb-cxl
docs/cxl: Add an arm/virt example.
qtest/cxl: Add aarch64 virt test for CXL

 docs/system/arm/virt.rst    |   9 ++
 docs/system/devices/cxl.rst |  11 +++
 include/hw/arm/virt.h       |   4 +
 include/hw/cxl/cxl.h        |   5 +-
 include/hw/cxl/cxl_host.h   |   5 +-
 hw/acpi/cxl.c               |  76 ++++++++--------
 hw/arm/virt-acpi-build.c    |  34 +++++++
 hw/arm/virt.c               |  30 +++++++
 hw/cxl/cxl-host-stubs.c     |   7 +-
 hw/cxl/cxl-host.c           | 174 +++++++++++++++++++++++++++++-------
 hw/i386/pc.c                |  50 +++++------
 tests/qtest/cxl-test.c      |  58 +++++++++---
 tests/qtest/meson.build     |   1 +
 13 files changed, 354 insertions(+), 110 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH v17 1/5 hw/cxl-host: Add an index field to CXLFixedMemoryWindow

To enable these to be found in a fixed order, that order needs to be known. This will later be used to sort a list of these structures so that address map and ACPI table entries are predictable.

---
v17: Tag from Eric.

---
 include/hw/cxl/cxl.h | 1 +
 hw/cxl/cxl-host.c    | 9 ++++++---
 2 files changed, 7 insertions(+), 3 deletions(-)
include/hw/cxl/cxl.h +1 -0
index 3ae4303383..83096b2300 100644
@@ -28,6 +28,7 @@
 typedef struct PXBCXLDev PXBCXLDev;
 
 typedef struct CXLFixedWindow {
+    int index;
     uint64_t size;
     char **targets;
     PXBCXLDev *target_hbs[16];
hw/cxl/cxl-host.c +6 -3
index e010163174..b7aa429ddf 100644
@@ -24,13 +24,15 @@
 
 static void cxl_fixed_memory_window_config(CXLState *cxl_state,
                                            CXLFixedMemoryWindowOptions *object,
-                                           Error **errp)
+                                           int index, Error **errp)
 {
     ERRP_GUARD();
     g_autofree CXLFixedWindow *fw = g_malloc0(sizeof(*fw));
     strList *target;
     int i;
 
+    fw->index = index;
+
     for (target = object->targets; target; target = target->next) {
         fw->num_targets++;
     }
@@ -325,14 +327,15 @@ static void machine_set_cfmw(Object *obj, Visitor *v, const char *name,
     CXLState *state = opaque;
     CXLFixedMemoryWindowOptionsList *cfmw_list = NULL;
     CXLFixedMemoryWindowOptionsList *it;
+    int index;
 
     visit_type_CXLFixedMemoryWindowOptionsList(v, name, &cfmw_list, errp);
     if (!cfmw_list) {
         return;
     }
 
-    for (it = cfmw_list; it; it = it->next) {
-        cxl_fixed_memory_window_config(state, it->value, errp);
+    for (it = cfmw_list, index = 0; it; it = it->next, index++) {
+        cxl_fixed_memory_window_config(state, it->value, index, errp);
     }
     state->cfmw_list = cfmw_list;
 }
Signature · 1 line
2.48.1
#2 lore raw

PATCH v17 2/5 hw/cxl: Make the CXL fixed memory windows devices.

In reply to Jonathan Cameron

Previously these somewhat device like structures were tracked using a list in the CXLState in each machine. This is proving restrictive in a few cases where we need to iterate through these without being aware of the machine type. Just make them sysbus devices.

Restrict them to not user created as they need to be visible to early stages of machine init given effects on the memory map.

This change both simplifies state tracking and enables features needed for performance optimization and hotness tracking by making it possible to retrieve the fixed memory window on actions elsewhere in the topology.

In some cases the ordering of the Fixed Memory Windows matters. For those utility functions provide a GSList sorted by the window index. This ensures that we get consistency across:
- ordering in the command line
- ordering of the host PA ranges
- ordering of ACPI CEDT structures describing the CFMWS.

Other aspects don't have this constraint. For those direct iteration of the underlying hash structures is fine.

In the setup path for the memory map in pc_memory_init() split the operations into two calls. The first, cxl_fmws_set_mmemap(), loops over fixed memory windows in order and assigns their addresses. The second, cxl_fmws_update_mmio() actually sets up the mmio for each window. This is obviously less efficient than a single loop but this split design is needed to put the logic in two different places in the arm64 support and it is not a hot enough path to justify an x86 only implementation.

---
v16: Add a comment about there being no dynamic state so no reset
or migration support is needed. (Peter Maydell)

---
 include/hw/cxl/cxl.h      |   4 +-
 include/hw/cxl/cxl_host.h |   5 +-
 hw/acpi/cxl.c             |  76 +++++++++--------
 hw/cxl/cxl-host-stubs.c   |   7 +-
 hw/cxl/cxl-host.c         | 167 +++++++++++++++++++++++++++++++-------
 hw/i386/pc.c              |  50 +++++-------
 6 files changed, 214 insertions(+), 95 deletions(-)
include/hw/cxl/cxl.h +3 -1
index 83096b2300..998f495a98 100644
@@ -28,6 +28,7 @@
 typedef struct PXBCXLDev PXBCXLDev;
 
 typedef struct CXLFixedWindow {
+    SysBusDevice parent_obj;
     int index;
     uint64_t size;
     char **targets;
@@ -39,12 +40,13 @@ typedef struct CXLFixedWindow {
     MemoryRegion mr;
     hwaddr base;
 } CXLFixedWindow;
+#define TYPE_CXL_FMW "cxl-fmw"
+OBJECT_DECLARE_SIMPLE_TYPE(CXLFixedWindow, CXL_FMW)
 
 typedef struct CXLState {
     bool is_enabled;
     MemoryRegion host_mr;
     unsigned int next_mr_idx;
-    GList *fixed_windows;
     CXLFixedMemoryWindowOptionsList *cfmw_list;
 } CXLState;
 
include/hw/cxl/cxl_host.h +4 -1
index c9bc9c7c50..cd3c368c86 100644
@@ -14,8 +14,11 @@
 #define CXL_HOST_H
 
 void cxl_machine_init(Object *obj, CXLState *state);
-void cxl_fmws_link_targets(CXLState *stat, Error **errp);
+void cxl_fmws_link_targets(Error **errp);
 void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp);
+hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr);
+void cxl_fmws_update_mmio(void);
+GSList *cxl_fmws_get_all_sorted(void);
 
 extern const MemoryRegionOps cfmws_ops;
 
hw/acpi/cxl.c +40 -36

114 lines not on this page · raw message

hw/cxl/cxl-host-stubs.c +6 -1
index cae4afcdde..c015baac81 100644
@@ -8,8 +8,13 @@
 #include "hw/cxl/cxl.h"
 #include "hw/cxl/cxl_host.h"
 
-void cxl_fmws_link_targets(CXLState *stat, Error **errp) {};
+void cxl_fmws_link_targets(Error **errp) {};
 void cxl_machine_init(Object *obj, CXLState *state) {};
 void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp) {};
+hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr)
+{
+    return base;
+};
+void cxl_fmws_update_mmio(void) {};
 
 const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c +139 -28

196 lines not on this page · raw message

hw/i386/pc.c +22 -28

72 lines not on this page · raw message

Signature · 1 line
2.48.1
#3 lore raw

PATCH v17 3/5 hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl

In reply to Jonathan Cameron

Code based on i386/pc enablement.
The memory layout places space for 16 host bridge register regions after the GIC_REDIST2 in the extended memmap. This is a hole in the current map so adding them here has no impact on placement of other memory regions (tested with enough CPUs for GIC_REDIST2 to be in use.)
The high memory map is GiB aligned so the hole is there whatever the size of memory or device_memory below this point.

The CFMWs are placed above the extended memmap. Note the existing variable highest_gpa is the highest GPA that has been allocated at a particular point in setting up the memory map. Whilst this caused some confusion in review there are existing comments explaining this so nothing is added.

The cxl_devices_state.host_mr provides a small space in which to place the individual host bridge register regions for whatever host bridges are allocated via -device pxb-cxl on the command line. The existing dynamic sysbus infrastructure is not reused because pxb-cxl is a PCI device not a sysbus one but these registers are directly in the main memory map, not the PCI address space.

Only create the CEDT table if cxl=on set for the machine. Default to off.

---
v17: Updated patch description to cover some feedback from Eric.
Added a comment in the high memory map where the CXL Fixed Memory Windows will be placed. Given the size can only be established at runtime, explicit entries don't work. This is similar to device_memory.

Perhaps unresolved feedback. Peter raised a concern about the direct initialization of vms->cxl_devices_state.host_mr. I've added more commentary about that to the patch description. Whilst it seems unnecessary I could wrap the relevant 3 lines of code up in a utility function rather than open coding them here and in i386/pc.

---
 docs/system/arm/virt.rst |  9 +++++++++
 include/hw/arm/virt.h    |  4 ++++
 hw/arm/virt-acpi-build.c | 34 ++++++++++++++++++++++++++++++++++
 hw/arm/virt.c            | 30 ++++++++++++++++++++++++++++++
 4 files changed, 77 insertions(+)
docs/system/arm/virt.rst +9 -0
index 6a719b9586..10cbffc8a7 100644
@@ -31,6 +31,7 @@ Supported devices
 The virt board supports:
 
 - PCI/PCIe devices
+- CXL Fixed memory windows, root bridges and devices.
 - Flash memory
 - Either one or two PL011 UARTs for the NonSecure World
 - An RTC
@@ -189,6 +190,14 @@ ras
 acpi
   Set ``on``/``off``/``auto`` to enable/disable ACPI.
 
+cxl
+  Set  ``on``/``off`` to enable/disable CXL. More details in
+  :doc:`../devices/cxl`. The default is off.
+
+cxl-fmw
+  Array of CXL fixed memory windows describing fixed address routing to
+  target CXL host bridges. See :doc:`../devices/cxl`.
+
 dtb-randomness
   Set ``on``/``off`` to pass random seeds via the guest DTB
   rng-seed and kaslr-seed nodes (in both "/chosen" and
include/hw/arm/virt.h +4 -0
index 9a1b0f53d2..4375819ea0 100644
@@ -36,6 +36,7 @@
 #include "hw/arm/boot.h"
 #include "hw/arm/bsa.h"
 #include "hw/block/flash.h"
+#include "hw/cxl/cxl.h"
 #include "system/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
 #include "qom/object.h"
@@ -85,6 +86,7 @@ enum {
 /* indices of IO regions located after the RAM */
 enum {
     VIRT_HIGH_GIC_REDIST2 =  VIRT_LOWMEMMAP_LAST,
+    VIRT_CXL_HOST,
     VIRT_HIGH_PCIE_ECAM,
     VIRT_HIGH_PCIE_MMIO,
 };
@@ -140,6 +142,7 @@ struct VirtMachineState {
     bool secure;
     bool highmem;
     bool highmem_compact;
+    bool highmem_cxl;
     bool highmem_ecam;
     bool highmem_mmio;
     bool highmem_redists;
@@ -174,6 +177,7 @@ struct VirtMachineState {
     char *oem_id;
     char *oem_table_id;
     bool ns_el2_virt_timer_irq;
+    CXLState cxl_devices_state;
 };
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +34 -0
index cd90c47976..c3b9b3f6ea 100644
@@ -39,10 +39,12 @@
 #include "hw/acpi/aml-build.h"
 #include "hw/acpi/utils.h"
 #include "hw/acpi/pci.h"
+#include "hw/acpi/cxl.h"
 #include "hw/acpi/memory_hotplug.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/acpi/tpm.h"
 #include "hw/acpi/hmat.h"
+#include "hw/cxl/cxl.h"
 #include "hw/pci/pcie_host.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bus.h"
@@ -119,10 +121,29 @@ static void acpi_dsdt_add_flash(Aml *scope, const MemMapEntry *flash_memmap)
     aml_append(scope, dev);
 }
 
+static void build_acpi0017(Aml *table)
+{
+    Aml *dev, *scope, *method;
+
+    scope =  aml_scope("_SB");
+    dev = aml_device("CXLM");
+    aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+    method = aml_method("_STA", 0, AML_NOTSERIALIZED);
+    aml_append(method, aml_return(aml_int(0x0B)));
+    aml_append(dev, method);
+    build_cxl_dsm_method(dev);
+
+    aml_append(scope, dev);
+    aml_append(table, scope);
+}
+
 static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
                               uint32_t irq, VirtMachineState *vms)
 {
     int ecam_id = VIRT_ECAM_ID(vms->highmem_ecam);
+    bool cxl_present = false;
+    PCIBus *bus = vms->bus;
     struct GPEXConfig cfg = {
         .mmio32 = memmap[VIRT_PCIE_MMIO],
         .pio    = memmap[VIRT_PCIE_PIO],
@@ -136,6 +157,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
     }
 
     acpi_dsdt_add_gpex(scope, &cfg);
+    QLIST_FOREACH(bus, &vms->bus->child, sibling) {
+        if (pci_bus_is_cxl(bus)) {
+            cxl_present = true;
+        }
+    }
+    if (cxl_present) {
+        build_acpi0017(scope);
+    }
 }
 
 static void acpi_dsdt_add_gpio(Aml *scope, const MemMapEntry *gpio_memmap,
@@ -1027,6 +1056,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->cxl_devices_state.is_enabled) {
+        cxl_build_cedt(table_offsets, tables_blob, tables->linker,
+                       vms->oem_id, vms->oem_table_id, &vms->cxl_devices_state);
+    }
+
     if (ms->nvdimms_state->is_enabled) {
         nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
                           ms->nvdimms_state, ms->ram_slots, vms->oem_id,
hw/arm/virt.c +30 -0
index 3bcdf92e2f..394e8b5301 100644
@@ -57,6 +57,7 @@
 #include "qemu/error-report.h"
 #include "qemu/module.h"
 #include "hw/pci-host/gpex.h"
+#include "hw/pci-bridge/pci_expander_bridge.h"
 #include "hw/virtio/virtio-pci.h"
 #include "hw/core/sysbus-fdt.h"
 #include "hw/platform-bus.h"
@@ -86,6 +87,8 @@
 #include "hw/virtio/virtio-md-pci.h"
 #include "hw/virtio/virtio-iommu.h"
 #include "hw/char/pl011.h"
+#include "hw/cxl/cxl.h"
+#include "hw/cxl/cxl_host.h"
 #include "qemu/guest-random.h"
 
 static GlobalProperty arm_virt_compat[] = {
@@ -220,9 +223,11 @@ static const MemMapEntry base_memmap[] = {
 static MemMapEntry extended_memmap[] = {
     /* Additional 64 MB redist region (can contain up to 512 redistributors) */
     [VIRT_HIGH_GIC_REDIST2] =   { 0x0, 64 * MiB },
+    [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
     [VIRT_HIGH_PCIE_ECAM] =     { 0x0, 256 * MiB },
     /* Second PCIe window */
     [VIRT_HIGH_PCIE_MMIO] =     { 0x0, DEFAULT_HIGH_PCIE_MMIO_SIZE },
+    /* Any CXL Fixed memory windows come here */
 };
 
 static const int a15irqmap[] = {
@@ -1623,6 +1628,17 @@ static void create_pcie(VirtMachineState *vms)
     }
 }
 
+static void create_cxl_host_reg_region(VirtMachineState *vms)
+{
+    MemoryRegion *sysmem = get_system_memory();
+    MemoryRegion *mr = &vms->cxl_devices_state.host_mr;
+
+    memory_region_init(mr, OBJECT(vms), "cxl_host_reg",
+                       vms->memmap[VIRT_CXL_HOST].size);
+    memory_region_add_subregion(sysmem, vms->memmap[VIRT_CXL_HOST].base, mr);
+    vms->highmem_cxl = true;
+}
+
 static void create_platform_bus(VirtMachineState *vms)
 {
     DeviceState *dev;
@@ -1739,6 +1755,12 @@ void virt_machine_done(Notifier *notifier, void *data)
     struct arm_boot_info *info = &vms->bootinfo;
     AddressSpace *as = arm_boot_address_space(cpu, info);
 
+    cxl_hook_up_pxb_registers(vms->bus, &vms->cxl_devices_state,
+                              &error_fatal);
+
+    if (vms->cxl_devices_state.is_enabled) {
+        cxl_fmws_link_targets(&error_fatal);
+    }
     /*
      * If the user provided a dtb, we assume the dynamic sysbus nodes
      * already are integrated there. This corresponds to a use case where
@@ -1785,6 +1807,7 @@ static inline bool *virt_get_high_memmap_enabled(VirtMachineState *vms,
 {
     bool *enabled_array[] = {
         &vms->highmem_redists,
+        &vms->highmem_cxl,
         &vms->highmem_ecam,
         &vms->highmem_mmio,
     };
@@ -1892,6 +1915,9 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits)
     if (device_memory_size > 0) {
         machine_memory_devices_init(ms, device_memory_base, device_memory_size);
     }
+    vms->highest_gpa = cxl_fmws_set_memmap(ROUND_UP(vms->highest_gpa + 1,
+                                                    256 * MiB),
+                                           BIT_ULL(pa_bits)) - 1;
 }
 
 static VirtGICType finalize_gic_version_do(const char *accel_name,
@@ -2343,6 +2369,8 @@ static void machvirt_init(MachineState *machine)
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base,
                                 machine->ram);
 
+    cxl_fmws_update_mmio();
+
     virt_flash_fdt(vms, sysmem, secure_sysmem ?: sysmem);
 
     create_gic(vms, sysmem);
@@ -2398,6 +2426,7 @@ static void machvirt_init(MachineState *machine)
     create_rtc(vms);
 
     create_pcie(vms);
+    create_cxl_host_reg_region(vms);
 
     if (has_ged && aarch64 && firmware_loaded && virt_is_acpi_enabled(vms)) {
         vms->acpi_dev = create_acpi_ged(vms);
@@ -3364,6 +3393,7 @@ static void virt_instance_init(Object *obj)
 
     vms->oem_id = g_strndup(ACPI_BUILD_APPNAME6, 6);
     vms->oem_table_id = g_strndup(ACPI_BUILD_APPNAME8, 8);
+    cxl_machine_init(obj, &vms->cxl_devices_state);
 }
 
 static const TypeInfo virt_machine_info = {
Signature · 1 line
2.48.1
#4 lore raw

PATCH v17 4/5 docs/cxl: Add an arm/virt example.

In reply to Jonathan Cameron

Only add one very simple example as all the i386/pc examples will work for arm/virt with a change to appropriate executable and appropriate standard launch line for arm/virt. Note that max cpu is used to ensure we have plenty of physical address space.

---
v17: Tag from Eric plus expanded on the intro text for the arm example.

---
 docs/system/devices/cxl.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
docs/system/devices/cxl.rst +11 -0
index e307caf3f8..ca15a0da1c 100644
@@ -384,6 +384,17 @@ An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave::
   -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,sn=0x4 \
   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k
 
+A simple arm/virt example featuring a single direct connected CXL Type 3
+Volatile Memory device::
+
+  qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8g,slots=4 -cpu max -smp 4 \
+  ...
+  -object memory-backend-ram,id=vmem0,share=on,size=256M \
+  -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
+  -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
+  -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
+  -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
+
 Deprecations
 ------------
 
Signature · 1 line
2.48.1
#5 lore raw

PATCH v17 5/5 qtest/cxl: Add aarch64 virt test for CXL

In reply to Jonathan Cameron

Add a single complex case for aarch64 virt machine.

Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continues to work.

---
v17: Tag and small fix for patch description (Eric).

---
 tests/qtest/cxl-test.c  | 58 ++++++++++++++++++++++++++++++++---------
 tests/qtest/meson.build |  1 +
 2 files changed, 46 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +45 -13
index a600331843..8fb7e58d4f 100644
@@ -19,6 +19,12 @@
     "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
     "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
     "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,51 @@ static void cxl_2pxb_4rp_4t3d(void)
     qtest_end();
     rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    g_autofree const char *tmpfs = NULL;
+
+    tmpfs = g_dir_make_tmp("cxl-test-XXXXXX", NULL);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
     return g_test_run();
 }
tests/qtest/meson.build +1 -0
index 8ad849054f..42e927b32a 100644
@@ -261,6 +261,7 @@ qtests_aarch64 = \
    config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
   (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
   (config_all_devices.has_key('CONFIG_NPCM8XX') ? qtests_npcm8xx : []) + \
+  qtests_cxl +                                                                                  \
   ['arm-cpu-features',
    'numa-test',
    'boot-serial-test',
Signature · 1 line
2.48.1
Tested-by on v17 5/5
#6 lore raw

Re: [PATCH qemu v17 5/5] qtest/cxl: Add aarch64 virt test for CXL

Quoted 105 lines · Jonathan Cameron

Add a single complex case for aarch64 virt machine.

Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continues to work.

---
v17: Tag and small fix for patch description (Eric).

---
 tests/qtest/cxl-test.c  | 58 ++++++++++++++++++++++++++++++++---------
 tests/qtest/meson.build |  1 +
 2 files changed, 46 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +45 -13
index a600331843..8fb7e58d4f 100644
@@ -19,6 +19,12 @@
     "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
     "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
     "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,51 @@ static void cxl_2pxb_4rp_4t3d(void)
     qtest_end();
     rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    g_autofree const char *tmpfs = NULL;
+
+    tmpfs = g_dir_make_tmp("cxl-test-XXXXXX", NULL);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
     return g_test_run();
 }
tests/qtest/meson.build +0 -0
index 8ad849054f..42e927b32a 100644

tests/qtest/meson.build qtests_aarch64 = \ · line 267

Quoted 7 lines
tests/qtest/meson.build +1 -0
@@ -261,6 +261,7 @@ qtests_aarch64 = \
    config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
   (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
   (config_all_devices.has_key('CONFIG_NPCM8XX') ? qtests_npcm8xx : []) + \
+  qtests_cxl +                                                                                  \
   ['arm-cpu-features',
    'numa-test',
    'boot-serial-test',

I checked out jic23/cxl-2025-07-03 branch and built fine for all the architectures, and did meson test qtest-aarch64/cxl-test returned OK.

Again, for the latest v17

Thanks,
Itaru.

Quoted 2 lines

2.48.1

Tested-by on the v17 series
#7 lore raw
In reply to Jonathan Cameron
Quoted 7 lines · Jonathan Cameron

v17: Thanks to Eric for review
- Add a comment to the high memory map to reduce the chance of nasty surprises in the future as similar to device_memory, the CXL Fixed Memory Windows are of variable size as so can't be represented by explicit entries in the map.
- Updated a couple of patch descriptions to provide answers to questions Eric raised.

I also tested the cxl/next kernel; it boots fine with CXL devices.

Quoted 115 lines

v16:
- Mostly additional documentation and descriptive text in patch titles.
- Update test to reflect changes to other tests.
- Update physmem_max in virt to include the CXL memory.

Updated cover letter

Back in 2022, this series stalled on the absence of a solution to device tree support for PCI Expander Bridges (PXB) and we ended up only having x86 support upstream. I've been carrying the arm64 support out of tree since then, with occasional nasty surprises (e.g. UNIMP + DT issue seen a few weeks ago) and a fair number of fiddly rebases. gitlab.com/jic23/qemu cxl-<latest date>. Will update shortly with this series.

A recent discussion with Peter Maydell indicated that there are various other ACPI only features now, so in general he might be more relaxed about DT support being necessary. The upcoming vSMMUv3 support would run into this problem as well.

I presented the background to the PXB issue at Linaro connect 2022. In short the issue is that PXBs steal MMIO space from the main PCI root bridge. The challenge is knowing how much to steal.

On ACPI platforms, we can rely on EDK2 to perform an enumeration and configuration of the PCI topology and QEMU can update the ACPI tables after EDK2 has done this when it can simply read the space used by the root ports. On device tree, there is no entity to figure out that enumeration so we don't know how to size the stolen region.

Three approaches were discussed:
1) Enumerating in QEMU. Horribly complex and the last thing we want is a 3rd enumeration implementation that ends up out of sync with EDK2 and the kernel (there are frequent issues because of how those existing implementations differ.
2) Figure out how to enumerate in kernel. I never put a huge amount of work into this, but it seemed likely to involve a nasty dance with similar very specific code to that EDK2 is carrying and would very challenging to upstream (given the lack of clarity on real use cases for PXBs and DT).
3) Hack it based on the control we have which is bus numbers. No one liked this but it worked :)

The other little wrinkle would be the need to define full bindings for CXL on DT + implement a fairly complex kernel stack as equivalent in ACPI involves a static table, CEDT, new runtime queries via _DSM and a description of various components. Doable, but so far there is no interest on physical platforms. Worth noting that for now, the QEMU CXL emulation is all about testing and developing the OS stack, not about virtualization (performance is terrible except in some very contrived situations!)

There is only a very simple test in here, because my intent is not to duplicate what we have on x86, but just to do a smoke test that everything is hooked up. In general we need much more comprehensive end to end CXL tests but that requires a reaonsably stable guest software stack. A few people have expressed interest in working on that, but we aren't there yet.

Note that this series has a very different use case to that in the proposed SBSA-ref support:
https://lore.kernel.org/qemu-devel/[email protected]/

SBSA-ref is a good choice if you want a relatively simple mostly fixed configuration. That works well with the limited host system discoverability etc as EDK2 can be build against a known configuration.

My interest with this support in arm/virt is support host software stack development (we have a wide range of contributors, most of whom are working on emulation + the kernel support). I care about the weird corners. As such I need to be able to bring up variable numbers of host bridges, multiple CXL Fixed Memory Windows with varying characteristics (interleave etc), complex NUMA topologies with wierd performance characteristics etc. We can do that on x86 upstream today, or my gitlab tree. Note that we need arm support for some arch specific features in the near future (cache flushing). Doing kernel development with this need for flexibility on SBSA-ref is not currently practical. SBSA-ref CXL support is an excellent thing, just not much use to me for this work.

Also, we are kicking off some work on DCD virtualization, particularly to support inter-host shared memory being presented up into a VM. That will need upstream support on arm64 as it is built on top of the existing CXL emulation to avoid the need for a separate guest software stack.

Note this is TCG only - it is possible to support limited use with KVM but that needs additional patches not yet ready for upstream. The challenge is interleave - and the solution is don't interleave if you want to run with KVM.

Jonathan Cameron (5):
hw/cxl-host: Add an index field to CXLFixedMemoryWindow hw/cxl: Make the CXL fixed memory windows devices. hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
pxb-cxl
docs/cxl: Add an arm/virt example.
qtest/cxl: Add aarch64 virt test for CXL

 docs/system/arm/virt.rst    |   9 ++
 docs/system/devices/cxl.rst |  11 +++
 include/hw/arm/virt.h       |   4 +
 include/hw/cxl/cxl.h        |   5 +-
 include/hw/cxl/cxl_host.h   |   5 +-
 hw/acpi/cxl.c               |  76 ++++++++--------
 hw/arm/virt-acpi-build.c    |  34 +++++++
 hw/arm/virt.c               |  30 +++++++
 hw/cxl/cxl-host-stubs.c     |   7 +-
 hw/cxl/cxl-host.c           | 174 +++++++++++++++++++++++++++++-------
 hw/i386/pc.c                |  50 +++++------
 tests/qtest/cxl-test.c      |  58 +++++++++---
 tests/qtest/meson.build     |   1 +
 13 files changed, 354 insertions(+), 110 deletions(-)
Signature · 1 line
2.48.1
#8 lore raw
In reply to Jonathan Cameron
Quoted 9 lines · Jonathan Cameron

v17: Thanks to Eric for review
- Add a comment to the high memory map to reduce the chance of nasty surprises in the future as similar to device_memory, the CXL Fixed Memory Windows are of variable size as so can't be represented by explicit entries in the map.
- Updated a couple of patch descriptions to provide answers to questions Eric raised.

Applied to target-arm.next, thanks.

-- PMM