Skip to contentSkip to conversation

arm/virt: CXL support via pxb_cxl

RFC PATCH Superseded Jonathan Cameron 2 participants 6 messages Feb 3, 2025 – Feb 4, 2025 Arm/Amlogic Meson SoC support

v12 of 17 Superseded Latest is v17 →

lore
Share & tools
lore
#0 lore raw

v12 cover arm/virt: CXL support via pxb_cxl

Back in 2022, this series stalled on the absence of a solution to device tree support for PCI Expander Bridges (PXB) and we ended up only having x86 support upstream. I've been carrying the arm64 support out of tree since then, with occasional nasty surprises (e.g. UNIMP + DT issue seen a few weeks ago) and a fair number of fiddly rebases. gitlab.com/jic23/qemu cxl-<latest date>

A recent discussion with Peter Maydell indicated that there are various other ACPI only features now, so in general he might be more relaxed about DT support being necessary. The upcoming vSMMUv3 support would run into this problem as well.

I presented the background to the PXB issue at Linaro connect 2022. In short the issue is that PXBs steal MMIO space from the main PCI root bridge. The challenge is knowing how much to steal.

On ACPI platforms, we can rely on EDK2 to perform an enumeration and configuration of the PCI topology and QEMU can update the ACPI tables after EDK2 has done this when it can simply read the space used by the root ports. On device tree, there is no entity to figure out that enumeration so we don't know how to size the stolen region.

Three approaches were discussed:
1) Enumerating in QEMU. Horribly complex and the last thing we want is a 3rd enumeration implementation that ends up out of sync with EDK2 and the kernel (there are frequent issues because of how those existing implementations differ.
2) Figure out how to enumerate in kernel. I never put a huge amount of work into this, but it seemed likely to involve a nasty dance with similar very specific code to that EDK2 is carrying and would very challenging to upstream (given the lack of clarity on real use cases for PXBs and DT).
3) Hack it based on the control we have which is bus numbers. No one liked this but it worked :)

The other little wrinkle would be the need to define full bindings for CXL on DT + implement a fairly complex kernel stack as equivalent in ACPI involves a static table, CEDT, new runtime queries via _DSM and a description of various components. Doable, but so far there is no interest on physical platforms. Worth noting that for now, the QEMU CXL emulation is all about testing and developing the OS stack, not about virtualization (performance is terrible except in some very contrived situations!)

Back to posting as an RFC because there was some discussion of approach to modelling the devices that may need a bit of redesign.
The discussion kind of died out on the back of DT issue and I doubt anyone can remember the details.

https://lore.kernel.org/qemu-devel/[email protected]/

There is only a very simple test in here, because my intent is not to duplicate what we have on x86, but just to do a smoke test that everything is hooked up. In general we need much more comprehensive end to end CXL tests but that requires a reaonsably stable guest software stack. A few people have expressed interest in working on that, but we aren't there yet.

Note that this series has a very different use case to that in the proposed SBSA-ref support:
https://lore.kernel.org/qemu-devel/[email protected]/

SBSA-ref is a good choice if you want a relatively simple mostly fixed configuration. That works well with the limited host system discoverability etc as EDK2 can be build against a known configuration.

My interest with this support in arm/virt is support host software stack development (we have a wide range of contributors, most of whom are working on emulation + the kernel support). I care about the weird corners. As such I need to be able to bring up variable numbers of host bridges, multiple CXL Fixed Memory Windows with varying characteristics (interleave etc), complex NUMA topologies with wierd performance characteristics etc. We can do that on x86 upstream today, or my gitlab tree. Note that we need arm support for some arch specific features in the near future (cache flushing). Doing kernel development with this need for flexibility on SBSA-ref is not currently practical. SBSA-ref CXL support is an excellent thing, just not much use to me for this work.

Thanks,

Jonathan

Jonathan Cameron (2):
hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
pxb-cxl
qtest/cxl: Add aarch64 virt test for CXL

 include/hw/arm/virt.h    |  4 +++
 hw/arm/virt-acpi-build.c | 34 +++++++++++++++++++++++
 hw/arm/virt.c            | 54 ++++++++++++++++++++++++++++++++++++
 tests/qtest/cxl-test.c   | 59 +++++++++++++++++++++++++++++++---------
 tests/qtest/meson.build  |  1 +
 5 files changed, 139 insertions(+), 13 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH v12 1/2 hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl

Code based on i386/pc enablement.
The memory layout places space for 16 host bridge register regions after the GIC_REDIST2 in the extended memmap. Note this is in a hole, so nothing should move.

The CFMWs are placed above the extended memmap.

Only create the CEDT table if cxl=on set for the machine.

---
 include/hw/arm/virt.h    |  4 +++
 hw/arm/virt-acpi-build.c | 34 +++++++++++++++++++++++++
 hw/arm/virt.c            | 54 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+)
include/hw/arm/virt.h +4 -0
index c8e94e6aed..3e83d9e06b 100644
@@ -36,6 +36,7 @@
 #include "hw/arm/boot.h"
 #include "hw/arm/bsa.h"
 #include "hw/block/flash.h"
+#include "hw/cxl/cxl.h"
 #include "system/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
 #include "qom/object.h"
@@ -85,6 +86,7 @@ enum {
 /* indices of IO regions located after the RAM */
 enum {
     VIRT_HIGH_GIC_REDIST2 =  VIRT_LOWMEMMAP_LAST,
+    VIRT_CXL_HOST,
     VIRT_HIGH_PCIE_ECAM,
     VIRT_HIGH_PCIE_MMIO,
 };
@@ -146,6 +148,7 @@ struct VirtMachineState {
     bool secure;
     bool highmem;
     bool highmem_compact;
+    bool highmem_cxl;
     bool highmem_ecam;
     bool highmem_mmio;
     bool highmem_redists;
@@ -180,6 +183,7 @@ struct VirtMachineState {
     char *oem_id;
     char *oem_table_id;
     bool ns_el2_virt_timer_irq;
+    CXLState cxl_devices_state;
 };
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +34 -0
index 3ac8f8e178..339aa60ce5 100644
@@ -39,10 +39,12 @@
 #include "hw/acpi/aml-build.h"
 #include "hw/acpi/utils.h"
 #include "hw/acpi/pci.h"
+#include "hw/acpi/cxl.h"
 #include "hw/acpi/memory_hotplug.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/acpi/tpm.h"
 #include "hw/acpi/hmat.h"
+#include "hw/cxl/cxl.h"
 #include "hw/pci/pcie_host.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bus.h"
@@ -119,10 +121,29 @@ static void acpi_dsdt_add_flash(Aml *scope, const MemMapEntry *flash_memmap)
     aml_append(scope, dev);
 }
 
+static void build_acpi0017(Aml *table)
+{
+    Aml *dev, *scope, *method;
+
+    scope =  aml_scope("_SB");
+    dev = aml_device("CXLM");
+    aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+    method = aml_method("_STA", 0, AML_NOTSERIALIZED);
+    aml_append(method, aml_return(aml_int(0x0B)));
+    aml_append(dev, method);
+    build_cxl_dsm_method(dev);
+
+    aml_append(scope, dev);
+    aml_append(table, scope);
+}
+
 static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
                               uint32_t irq, VirtMachineState *vms)
 {
     int ecam_id = VIRT_ECAM_ID(vms->highmem_ecam);
+    bool cxl_present = false;
+    PCIBus *bus = vms->bus;
     struct GPEXConfig cfg = {
         .mmio32 = memmap[VIRT_PCIE_MMIO],
         .pio    = memmap[VIRT_PCIE_PIO],
@@ -136,6 +157,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
     }
 
     acpi_dsdt_add_gpex(scope, &cfg);
+    QLIST_FOREACH(bus, &vms->bus->child, sibling) {
+        if (pci_bus_is_cxl(bus)) {
+            cxl_present = true;
+        }
+    }
+    if (cxl_present) {
+        build_acpi0017(scope);
+    }
 }
 
 static void acpi_dsdt_add_gpio(Aml *scope, const MemMapEntry *gpio_memmap,
@@ -967,6 +996,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->cxl_devices_state.is_enabled) {
+        cxl_build_cedt(table_offsets, tables_blob, tables->linker,
+                       vms->oem_id, vms->oem_table_id, &vms->cxl_devices_state);
+    }
+
     if (ms->nvdimms_state->is_enabled) {
         nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
                           ms->nvdimms_state, ms->ram_slots, vms->oem_id,
hw/arm/virt.c +54 -0

114 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

RFC PATCH v12 2/2 qtest/cxl: Add aarch64 virt test for CXL

In reply to Jonathan Cameron

Add a single complex case for aarch64 virt machine. Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continue to work.

---
 tests/qtest/cxl-test.c  | 59 ++++++++++++++++++++++++++++++++---------
 tests/qtest/meson.build |  1 +
 2 files changed, 47 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +46 -13
index a600331843..c7189d6222 100644
@@ -19,6 +19,12 @@
     "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
     "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
     "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,52 @@ static void cxl_2pxb_4rp_4t3d(void)
     qtest_end();
     rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
     return g_test_run();
 }
tests/qtest/meson.build +1 -0
index e60e92fe9d..f5e7fb060e 100644
@@ -257,6 +257,7 @@ qtests_aarch64 = \
   (config_all_accel.has_key('CONFIG_TCG') and                                            \
    config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
   (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
+  qtests_cxl +                                                                                  \
   ['arm-cpu-features',
    'numa-test',
    'boot-serial-test',
Signature · 1 line
2.43.0
#3 lore raw

Re: [RFC PATCH v12 qemu 2/2] qtest/cxl: Add aarch64 virt test for CXL

Jonathan,

Quoted 111 lines · Jonathan Cameron

Add a single complex case for aarch64 virt machine. Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continue to work.

---
tests/qtest/cxl-test.c | 59 ++++++++++++++++++++++++++++++++--------- tests/qtest/meson.build | 1 +

2 files changed, 47 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +46 -13
index a600331843..c7189d6222 100644
@@ -19,6 +19,12 @@
    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
    "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,52 @@ static void cxl_2pxb_4rp_4t3d(void)
    qtest_end();
    rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
    return g_test_run();
 }
tests/qtest/meson.build +1 -0
index e60e92fe9d..f5e7fb060e 100644
@@ -257,6 +257,7 @@ qtests_aarch64 = \
  (config_all_accel.has_key('CONFIG_TCG') and                                            \
   config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
  (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
+  qtests_cxl +                                                                                  \
  ['arm-cpu-features',
   'numa-test',
   'boot-serial-test',
Signature · 1 line
2.43.0

In Ubuntu 22.04 LTS, cxl-test applied on top of today’s QEMU upstream master branch cxl-test fails:

$ ./tests/qtest/cxl-test
# random seed: R02S2a8b02df7b32b79d086ce22f7f8ebeab
1..1
# Start of aarch64 tests
# Start of pci tests
# Start of cxl tests
# Start of virt tests
# starting QEMU: exec qemu-system-aarch64 -qtest unix:/tmp/qtest-568421.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-568421.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -machine virt,cxl=on -cpu max -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 -device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 -device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 -device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 -object memory-backend-file,id=cxl-mem0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp0,persistent-memdev=cxl-mem0,lsa=lsa0,id=pmem0 -object memory-backend-file,id=cxl-mem1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1,id=pmem1 -object memory-backend-file,id=cxl-mem2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2,id=pmem2 -object memory-backend-file,id=cxl-mem3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3,id=pmem3  -accel qtest
qemu-system-aarch64: -audio: invalid option
socket_accept failed: Resource temporarily unavailable
**
ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
Bail out! ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
../tests/qtest/libqtest.c:199: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)

Do I need set env vars when execute this test?

Itaru.

#4 lore raw

Re: [RFC PATCH v12 qemu 2/2] qtest/cxl: Add aarch64 virt test for CXL

Quoted 132 lines · On Tue, 4 Feb 2025 14:16:19 +0900 Itaru Kitayama <[email protected]>…

Jonathan,

Quoted 111 lines · Jonathan Cameron

Add a single complex case for aarch64 virt machine. Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continue to work.

---
tests/qtest/cxl-test.c | 59 ++++++++++++++++++++++++++++++++--------- tests/qtest/meson.build | 1 +

2 files changed, 47 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +46 -13
index a600331843..c7189d6222 100644
@@ -19,6 +19,12 @@
    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
    "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,52 @@ static void cxl_2pxb_4rp_4t3d(void)
    qtest_end();
    rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
    return g_test_run();
 }
tests/qtest/meson.build +1 -0
index e60e92fe9d..f5e7fb060e 100644
@@ -257,6 +257,7 @@ qtests_aarch64 = \
  (config_all_accel.has_key('CONFIG_TCG') and                                            \
   config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
  (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
+  qtests_cxl +                                                                                  \
  ['arm-cpu-features',
   'numa-test',
   'boot-serial-test',
Signature · 1 line
2.43.0

In Ubuntu 22.04 LTS, cxl-test applied on top of today’s QEMU upstream master branch cxl-test fails:

$ ./tests/qtest/cxl-test
# random seed: R02S2a8b02df7b32b79d086ce22f7f8ebeab
1..1
# Start of aarch64 tests
# Start of pci tests
# Start of cxl tests
# Start of virt tests
# starting QEMU: exec qemu-system-aarch64 -qtest unix:/tmp/qtest-568421.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-568421.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -machine virt,cxl=on -cpu max -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 -device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 -device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 -device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 -object memory-backend-file,id=cxl-mem0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp0,persistent-memdev=cxl-mem0,lsa=lsa0,id=pmem0 -object memory-backend-file,id=cxl-mem1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1,id=pmem1 -object memory-backend-file,id=cxl-mem2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2,id=pmem2 -object memory-backend-file,id=cxl-mem3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3,id=pmem3  -accel qtest
qemu-system-aarch64: -audio: invalid option
socket_accept failed: Resource temporarily unavailable
**
ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
Bail out! ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
../tests/qtest/libqtest.c:199: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
Quoted 1 line

Do I need set env vars when execute this test?

I've just been running it with make check-qtest and not seeing anything similar.

I'm not sure what infrastructure qtest puts round these but I guess it sets up that socket.

Jonathan

Quoted 2 lines

Itaru.

Tested-by
#5 lore raw

Re: [RFC PATCH v12 qemu 2/2] qtest/cxl: Add aarch64 virt test for CXL

Quoted 139 lines · Jonathan Cameron
Quoted 132 lines · On Tue, 4 Feb 2025 14:16:19 +0900 Itaru Kitayama <[email protected]>…

Jonathan,

Quoted 111 lines · Jonathan Cameron

Add a single complex case for aarch64 virt machine. Given existing much more comprehensive tests for x86 cover the common functionality, a single test should be enough to verify that the aarch64 part continue to work.

---
tests/qtest/cxl-test.c | 59 ++++++++++++++++++++++++++++++++--------- tests/qtest/meson.build | 1 +

2 files changed, 47 insertions(+), 13 deletions(-)
tests/qtest/cxl-test.c +46 -13
index a600331843..c7189d6222 100644
@@ -19,6 +19,12 @@
   "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
   "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
 
+#define QEMU_VIRT_2PXB_CMD \
+    "-machine virt,cxl=on -cpu max " \
+    "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+    "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+    "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "
+
 #define QEMU_RP \
   "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
 
@@ -197,25 +203,52 @@ static void cxl_2pxb_4rp_4t3d(void)
   qtest_end();
   rmdir(tmpfs);
 }
+
+static void cxl_virt_2pxb_4rp_4t3d(void)
+{
+    g_autoptr(GString) cmdline = g_string_new(NULL);
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D,
+                    tmpfs, tmpfs, tmpfs, tmpfs, tmpfs, tmpfs,
+                    tmpfs, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+    rmdir(tmpfs);
+}
 #endif /* CONFIG_POSIX */
 
 int main(int argc, char **argv)
 {
-    g_test_init(&argc, &argv, NULL);
+    const char *arch = qtest_get_arch();
 
-    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
-    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
-    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
-    qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
-    qtest_add_func("/pci/cxl/rp", cxl_root_port);
-    qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+    g_test_init(&argc, &argv, NULL);
+    if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
+        qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+        qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+        qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+        qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+        qtest_add_func("/pci/cxl/rp", cxl_root_port);
+        qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
 #ifdef CONFIG_POSIX
-    qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
-    qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
-    qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
-    qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
-    qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
-    qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+        qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated);
+        qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent);
+        qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile);
+        qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa);
+        qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+        qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4",
+                       cxl_2pxb_4rp_4t3d);
 #endif
+    } else if (strcmp(arch, "aarch64") == 0) {
+#ifdef CONFIG_POSIX
+        qtest_add_func("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4",
+                       cxl_virt_2pxb_4rp_4t3d);
+#endif
+    }
+
   return g_test_run();
 }
tests/qtest/meson.build +1 -0
index e60e92fe9d..f5e7fb060e 100644
@@ -257,6 +257,7 @@ qtests_aarch64 = \
 (config_all_accel.has_key('CONFIG_TCG') and                                            \
  config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
 (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
+  qtests_cxl +                                                                                  \
 ['arm-cpu-features',
  'numa-test',
  'boot-serial-test',
Signature · 1 line
2.43.0

In Ubuntu 22.04 LTS, cxl-test applied on top of today’s QEMU upstream master branch cxl-test fails:

$ ./tests/qtest/cxl-test
# random seed: R02S2a8b02df7b32b79d086ce22f7f8ebeab
1..1
# Start of aarch64 tests
# Start of pci tests
# Start of cxl tests
# Start of virt tests
# starting QEMU: exec qemu-system-aarch64 -qtest unix:/tmp/qtest-568421.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-568421.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -machine virt,cxl=on -cpu max -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 -device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 -device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 -device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 -object memory-backend-file,id=cxl-mem0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa0,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp0,persistent-memdev=cxl-mem0,lsa=lsa0,id=pmem0 -object memory-backend-file,id=cxl-mem1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa1,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1,id=pmem1 -object memory-backend-file,id=cxl-mem2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa2,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2,id=pmem2 -object memory-backend-file,id=cxl-mem3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -object memory-backend-file,id=lsa3,mem-path=/tmp/cxl-test-WdBn4X,size=256M -device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3,id=pmem3  -accel qtest
qemu-system-aarch64: -audio: invalid option
socket_accept failed: Resource temporarily unavailable
**
ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
Bail out! ERROR:../tests/qtest/libqtest.c:550:qtest_connect: assertion failed: (s->fd >= 0 && s->qmp_fd >= 0)
../tests/qtest/libqtest.c:199: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
Quoted 4 lines
Quoted 1 line

Do I need set env vars when execute this test?

I've just been running it with make check-qtest and not seeing anything similar.

Ah. make check-qtest rather than directly executing the binary made the updated cxl-test finished with other 26 “Ok” test 1 Skipped for aarch64

Tested-by Itaru Kitayama <[email protected] <mailto:[email protected]>>
Quoted 9 lines

I'm not sure what infrastructure qtest puts round these but I guess it sets up that socket.

Jonathan

Quoted 2 lines

Itaru.