hw/riscv/virt: Add CXL support to the RISC-V virt machine
cover hw/riscv/virt: Add CXL support to the RISC-V virt machine
This series adds CXL support to the RISC-V virt machine, following the approach used by the ARM virt machine: CXL host bridges (pxb-cxl) are described as ACPI0016 devices, an ACPI0017 (CXLM) device is added to the DSDT, and a CEDT table is built. A bios-tables test is added to pin down the generated ACPI tables.
Prerequisite
------------
The v2 posting depended on Alireza Sanaee's v8 series [1] (the performant non-interleaved CFMW lookup path). That series has since been merged upstream, so this version has no outstanding prerequisite and applies directly on current upstream master.
Changes since v2
----------------
- hw/riscv/virt: The MMIO-window patch no longer touches the common gpex code (Igor). The gpex_cfg.cxl_mmio32 field and the is_cxl static _CRS branch are dropped; the ACPI0016 _CRS now comes from the generic build_crs() path. Since EDK2 does not enumerate the pxb-cxl expander bridge, riscv simulates the firmware PCI initialization (reserved window + depth-first bridge bus numbers) and re-applies it via a reset handler.
- hw/riscv/virt: Drops the machine-global window/bus-range synthesis that did not scale past one pxb-cxl (Junjie). The series targets a single CXL host bridge for now, documented in the commit message and a TODO.
- hw/riscv/virt: CXL host register region and FMW setup folded into a single cxl_host_state_init() helper; redundant braces removed in create_fdt_pcie() (Daniel).
- hw/riscv/virt-acpi-build: the _DEP commit message documents the kernel-compatibility behaviour (Alistair).
- Carried review tags: Sunil V L's Reviewed-by on the _DEP patch and Alistair Francis's Acked-by on the test.
Changes since v1
---------------
- hw/riscv/virt: PCIBus *bus renamed to PCIBus *pci_bus (Jonathan).
- hw/riscv/virt: Dropped outer if (s->pci_bus) guard around cxl_hook_up_pxb_registers(); the function already handles a NULL bus internally (Jonathan).
- hw/riscv/virt-acpi-build: All s->bus references updated to
s->pci_bus; iasl -d decompiled DSDT fragment added to commit
message.
- hw/riscv/virt,gpex: Commit message expanded with PCI-to-PCI Bridge
Spec §3.2.5.8/9/10 citations (Jonathan). - Original patch 4 ("Map committed HDM decoder ranges as RAM for direct DMA") dropped; superseded by Alireza Sanaee's v8 series [1], which is now merged upstream and thus no longer a prerequisite.
- New patch 4: RISC-V ACPI bios-tables test for CXL, with golden AML files generated and included.
[1] https://lore.kernel.org/qemu-devel/[email protected]/
Test
----
Built riscv64-softmmu and ran the new bios-tables test together with the existing riscv64 ACPI tests; all pass. Also booted an RVA22 kernel with pxb-cxl + cxl-rp + cxl-type3 + a CFMW: the root port and Type3 device enumerate (0000:0c:00.0 / 0000:0d:00.0) and 'cxl list' reports the memdev (4 GiB) and the CFMWS root decoder.
QEMU invocation (CXL-relevant options shown):
qemu-system-riscv64 \
-M virt,aia=aplic-imsic,acpi=on,cxl=on \
-cpu rv64 -smp 2 -m 4G,maxmem=8G,slots=8 \
-object memory-backend-ram,id=vmem0,share=on,size=4G \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=rp0,chassis=0,slot=2 \
-device cxl-type3,bus=rp0,volatile-memdev=vmem0,id=cxl-vmem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
...Verification (total system memory grows by 4 GiB after onlining):
# cxl list
# cxl enable-memdev mem0
# cxl create-region -m -t ram -d decoder0.0 -w 1 mem0 -s 4G
# daxctl online-memory dax0.0
# free -hChen Pei (4):
hw/riscv/virt: Add CXL support to the RISC-V virt machine hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge
dependency
hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges tests/qtest: Add RISC-V ACPI bios tables test for CXL
hw/riscv/Kconfig | 2 + hw/riscv/virt-acpi-build.c | 52 +++++++ hw/riscv/virt.c | 191 +++++++++++++++++++++++++- include/hw/riscv/virt.h | 3 + tests/data/acpi/riscv64/virt/CEDT.cxl | Bin 0 -> 108 bytes tests/data/acpi/riscv64/virt/DSDT.cxl | Bin 0 -> 6212 bytes tests/qtest/bios-tables-test.c | 54 ++++++++ 7 files changed, 301 insertions(+), 1 deletion(-) create mode 100644 tests/data/acpi/riscv64/virt/CEDT.cxl create mode 100644 tests/data/acpi/riscv64/virt/DSDT.cxl
Signature · 1 line
2.50.1
PATCH 1/4 hw/riscv/virt: Add CXL support to the RISC-V virt machine
Enable CXL support on the RISC-V virt machine following the same approach used by the ARM virt machine:
- Add PXB and ACPI_CXL Kconfig selections
- Add CXLState and PCIBus pointer to RISCVVirtState
- Register CXL machine properties via cxl_machine_init()
- Create CXL host register region above the PCIe high MMIO region
- Call cxl_hook_up_pxb_registers() and cxl_fmws_link_targets() at machine_done time
- Map Fixed Memory Windows above the CXL host register region
- Add ACPI0017 device in DSDT and build CEDT table in virt-acpi-build.c
--- hw/riscv/Kconfig | 2 ++ hw/riscv/virt-acpi-build.c | 20 ++++++++++++++++++++ hw/riscv/virt.c | 37 +++++++++++++++++++++++++++++++++++++ include/hw/riscv/virt.h | 3 +++ 4 files changed, 62 insertions(+)
hw/riscv/Kconfig +2 -0
@@ -66,8 +66,10 @@ config RISCV_VIRT select VIRTIO_MMIO select FW_CFG_DMA select PLATFORM_BUS + select PXB select ACPI select ACPI_PCI + select ACPI_CXL config SHAKTI_C bool
hw/riscv/virt-acpi-build.c +20 -0
@@ -29,12 +29,16 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/pci.h" #include "hw/acpi/utils.h" +#include "hw/acpi/cxl.h" #include "hw/intc/riscv_aclint.h" #include "hw/nvram/fw_cfg_acpi.h" #include "hw/pci-host/gpex.h" +#include "hw/pci/pci_bus.h" #include "hw/riscv/virt.h" #include "hw/riscv/numa.h" #include "hw/virtio/virtio-acpi.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "kvm/kvm_riscv.h" #include "migration/vmstate.h" #include "qapi/error.h"
@@ -514,6 +518,17 @@ static void build_dsdt(GArray *table_data,acpi_dsdt_add_gpex_host (scope,PCIE_IRQ + s->num_sources *2 ); } +if (s->cxl_devices_state.is_enabled) { +Aml *cxl_dev =aml_device ("CXLM" ); +aml_append (cxl_dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); +Aml *method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED ); +aml_append (method,aml_return (aml_int (0x0B ))); +aml_append (cxl_dev, method); +build_cxl_dsm_method (cxl_dev); + +aml_append (scope, cxl_dev); + } +aml_append (dsdt, scope);/* copy AML table into ACPI tables blob and patch header there */
@@ -921,6 +936,11 @@ static void virt_acpi_build(RISCVVirtState *s, AcpiBuildTables *tables) s->oem_table_id); } +if (s->cxl_devices_state.is_enabled) { +cxl_build_cedt (table_offsets, tables_blob, tables->linker, + s->oem_id, s->oem_table_id, &s->cxl_devices_state); + } +if (ms->numa_state->num_nodes >0 ) {acpi_add_table (table_offsets, tables_blob);build_srat (tables_blob, tables->linker, s);
hw/riscv/virt.c +37 -0
@@ -55,6 +55,8 @@#include "hw/pci/pci.h" #include "hw/pci-host/gpex.h" #include "hw/display/ramfb.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "hw/acpi/aml-build.h" #include "qapi/qapi-visit-common.h" #include "hw/virtio/virtio-iommu.h"
@@ -1101,9 +1103,33 @@ static inline DeviceState *gpex_pcie_init(MemoryRegion *sys_mem, }GPEX_HOST (dev)->gpex_cfg.bus =PCI_HOST_BRIDGE (dev)->bus; + s->pci_bus =PCI_HOST_BRIDGE (dev)->bus;return dev; } +static void cxl_host_state_init (RISCVVirtState *s) +{ +MemoryRegion *sysmem =get_system_memory (); +MemoryRegion *mr = &s->cxl_devices_state.host_mr; +hwaddr base; + +if (!s->cxl_devices_state.is_enabled) { +return ; + } + + base = virt_high_pcie_memmap.base + virt_high_pcie_memmap.size; + base =ROUND_UP (base,64 * KiB); + +memory_region_init (mr,OBJECT (s),"cxl_host_reg" ,64 * KiB *16 ); +memory_region_add_subregion (sysmem, base, mr); + +/* Map the Fixed Memory Windows above the CXL host register region. */ + base +=memory_region_size (mr); + base =ROUND_UP (base,256 * MiB); +cxl_fmws_set_memmap (base,UINT64_MAX ); +cxl_fmws_update_mmio (); +} +static FWCfgState *create_fw_cfg (const MachineState *ms,hwaddr base) {FWCfgState *fw_cfg;
@@ -1205,6 +1231,13 @@ static void virt_machine_done(Notifier *notifier, void *data) machine_done);MachineState *machine =MACHINE (s);hwaddr start_addr = s->memmap[VIRT_DRAM ].base; + +cxl_hook_up_pxb_registers (s->pci_bus, &s->cxl_devices_state, + &error_fatal); + +if (s->cxl_devices_state.is_enabled) { +cxl_fmws_link_targets (&error_fatal); + }hwaddr firmware_end_addr;vaddr kernel_start_addr;const char *firmware_name =riscv_default_firmware_name (&s->soc[0 ]);
@@ -1451,6 +1484,8 @@ static void virt_machine_init(MachineState *machine)ROUND_UP (virt_high_pcie_memmap.base, virt_high_pcie_memmap.size); } +cxl_host_state_init (s); +/* register system main memory (actual RAM) */ memory_region_add_subregion (system_memory, s->memmap[VIRT_DRAM ].base, machine->ram);
@@ -1558,6 +1593,8 @@ static void virt_machine_instance_init(Object *obj) s->acpi =ON_OFF_AUTO_AUTO ; s->iommu_sys =ON_OFF_AUTO_AUTO ; s->num_sources =VIRT_IRQCHIP_NUM_SOURCES ; + +cxl_machine_init (obj, &s->cxl_devices_state); }static char *virt_get_aia_guests (Object *obj,Error **errp)
include/hw/riscv/virt.h +3 -0
@@ -24,6 +24,7 @@#include "hw/core/sysbus.h" #include "hw/block/flash.h" #include "hw/intc/riscv_imsic.h" +#include "hw/cxl/cxl.h" #define VIRT_CPUS_MAX_BITS 9#define VIRT_CPUS_MAX (1 << VIRT_CPUS_MAX_BITS)
@@ -65,6 +66,8 @@ struct RISCVVirtState {OnOffAuto iommu_sys;uint16_t pci_iommu_bdf;uint16_t num_sources; +CXLState cxl_devices_state; +PCIBus *pci_bus; };enum {
Signature · 1 line
2.50.1
PATCH 2/4 hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge dependency
In reply to Chen PeiOn RISC-V QEMU virt platform with CXL enabled, the probe ordering of acpi_pci_root (ACPI0016) and cxl_acpi (ACPI0017) is not guaranteed. If cxl_acpi probes before acpi_pci_root has attached the CXL host bridges, the CXL port topology will be incomplete because to_cxl_host_bridge() silently skips devices whose PCI root is not yet ready.
Add a _DEP object to the ACPI0017 device in the DSDT, declaring its dependency on the ACPI0016 CXL host bridge devices. This tells the OS to defer ACPI0017 enumeration until all ACPI0016 devices have been attached by acpi_pci_root.
This requires a corresponding kernel change to call acpi_dev_clear_dependencies() in acpi_pci_root_add().
Note on kernel compatibility: this is a "new QEMU feature requires a sufficiently new kernel" situation. Kernels that process _DEP but lack the acpi_dev_clear_dependencies() call will leave the CXLM device deferred, so CXL memory is unusable there; kernels old enough to ignore _DEP entirely behave exactly as before this change. In neither case is existing (non-CXL) PCIe affected.
The resulting DSDT fragment (iasl -d output) for a single CXL host bridge at bus 0x01 looks like:
Device (CXLM)
{
Name (_HID, "ACPI0017")
Name (_DEP, Package (0x01)
{
\_SB.PC01
})
Method (_STA, 0, NotSerialized)
{
Return (0x0B)
}
...
}--- hw/riscv/virt-acpi-build.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
hw/riscv/virt-acpi-build.c +32 -0
@@ -521,6 +521,38 @@ static void build_dsdt(GArray *table_data,if (s->cxl_devices_state.is_enabled) {Aml *cxl_dev =aml_device ("CXLM" );aml_append (cxl_dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + +/* +* Declare a _DEP on every ACPI0016 CXL host bridge so the OS +* defers ACPI0017 enumeration until acpi_pci_root has attached +* the CXL host bridges. Without this, cxl_acpi may probe before +* to_cxl_host_bridge() can resolve the PCI root and the CXL +* port topology comes up empty. +*/ +if (s->pci_bus) { +PCIBus *bus; +uint32_t num_cxl_hbs =0 ; + + QLIST_FOREACH(bus, &s->pci_bus->child, sibling) { +if (pci_bus_is_root (bus) &&pci_bus_is_cxl (bus)) { + num_cxl_hbs++; + } + } + +if (num_cxl_hbs >0 ) { +Aml *dep_pkg =aml_package (num_cxl_hbs); + + QLIST_FOREACH(bus, &s->pci_bus->child, sibling) { +if (pci_bus_is_root (bus) &&pci_bus_is_cxl (bus)) { +aml_append (dep_pkg, +aml_name ("\\_SB.PC%.02X" , +pci_bus_num (bus))); + } + } +aml_append (cxl_dev,aml_name_decl ("_DEP" , dep_pkg)); + } + } +Aml *method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED );aml_append (method,aml_return (aml_int (0x0B )));aml_append (cxl_dev, method);
Signature · 1 line
2.50.1
PATCH 3/4 hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges
In reply to Chen PeiCXL component register BAR (BAR0 on CXL Root Port and Type3 device) and the CXL device register BAR (BAR2 on Type3 device) are declared as 64-bit non-prefetchable memory. Per the PCI-to-PCI Bridge Architecture Specification Rev 1.2 (PCI-SIG, 2003):
- §3.2.5.8 (Memory Base/Limit): the non-prefetchable window covers only 32-bit addresses (AD[31:20]); the Type 1 header defines no upper-32-bit extension for it.
- §3.2.5.9 (Prefetchable Memory Base/Limit): the bottom 4 bits encode 64-bit support (01h), but this applies exclusively to the *prefetchable* window.
- §3.2.5.10 (Prefetchable Base/Limit Upper 32 Bits): optional registers for AD[63:32] of the prefetchable range only.
The architecture therefore allows a 64-bit window only when it is also prefetchable; there is no 64-bit non-prefetchable form. PCIe inherits this Type 1 header layout unchanged. Linux thus places 64-bit non-prefetchable BARs in the 32-bit non-prefetchable bridge window, which requires the bridge to own enough address space below 4 GiB.
On RISC-V virt the 32-bit PCIe MMIO range (1 GiB at 0x40000000) is currently consumed entirely by PCI0, so CXL host bridges (ACPI0016) have no non-prefetchable window and Linux fails to assign these BARs.
Reserve the top 256 MiB of the 32-bit MMIO window exclusively for CXL host bridges, keeping all of the logic in the riscv code:
- Reduce the FDT 'ranges' for PCI0 by 256 MiB so that device-tree driven firmware does not hand that range to PCI0.
- The CXL host bridge _CRS is produced by the generic build_crs() path. EDK2's PciBusDxe only recurses into PCI-to-PCI bridges, while the pxb-cxl expander bridge presents as a class 0x0600 host bridge with a type-0 header, so firmware never enumerates behind it and leaves the CXL root port's window and bus-number registers unset; build_crs() would therefore return an empty _CRS. Simulate the firmware PCI initialization in virt_cxl_init_bridge_windows(): assign the reserved window to the root port's memory window, and allocate primary/secondary/subordinate bus numbers depth-first. build_crs() then emits both the memory range and a correct bus-range resource (without a bus range Linux only claims the host-bridge bus and cannot enumerate the devices behind the root port), and the range is excluded from PCI0's _CRS. A reset handler re-applies the state, as a PCI reset clears those registers.
This targets a single pxb-cxl (one CXL host bridge), which matches the bios-tables test added later in the series. With more than one pxb-cxl the single reserved window would be claimed by every ACPI0016 bridge, so it would need to be partitioned per bridge; that is left as future work (see the TODO in virt_cxl_init_bridge_windows()).
Verified by booting an RVA22 kernel: the CXL root port and Type3 device are enumerated (0000:0c:00.0 / 0000:0d:00.0) and 'cxl list' reports the memdev and the CFMWS root decoder.
--- hw/riscv/virt.c | 154 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 153 insertions(+), 1 deletion(-)
hw/riscv/virt.c +153 -1
@@ -48,11 +48,13 @@#include "chardev/char.h" #include "system/device_tree.h" #include "system/system.h" +#include "system/reset.h" #include "system/tcg.h" #include "system/kvm.h" #include "system/tpm.h" #include "system/qtest.h" #include "hw/pci/pci.h" +#include "hw/pci/pci_bridge.h" #include "hw/pci-host/gpex.h" #include "hw/display/ramfb.h" #include "hw/cxl/cxl.h"
@@ -115,6 +117,9 @@ static const MemMapEntry virt_memmap[] = {/* PCIe high mmio for RV64, size is fixed but base depends on top of RAM */ #define VIRT64_HIGH_PCIE_MMIO_SIZE (16 * GiB) +/* 32-bit MMIO range carved out of VIRT_PCIE_MMIO for CXL host bridges */ +#define VIRT_CXL_MMIO32_SIZE (256 * MiB) +static MemMapEntry virt_high_pcie_memmap;#define VIRT_FLASH_SECTOR_SIZE (256 * KiB)
@@ -727,6 +732,18 @@ static void create_fdt_pcie(RISCVVirtState *s, { g_autofree char *name = NULL;MachineState *ms =MACHINE (s); +/* +* When CXL is enabled, reserve the last 256 MiB of the 32-bit MMIO +* window for CXL host bridges and exclude it from the main PCIe host +* bridge's FDT 'ranges' so UEFI's PciHostBridgeDxe does not allocate +* that range to PCI0. The CXL host bridge _CRS declares this range +* independently. +*/ +hwaddr mmio32_size = s->memmap[VIRT_PCIE_MMIO ].size; + +if (s->cxl_devices_state.is_enabled) { + mmio32_size -=VIRT_CXL_MMIO32_SIZE ; + } name =g_strdup_printf ("/soc/pci@%" HWADDR_PRIx, s->memmap[VIRT_PCIE_ECAM ].base);
@@ -752,7 +769,7 @@ static void create_fdt_pcie(RISCVVirtState *s,2 , s->memmap[VIRT_PCIE_PIO ].base,2 , s->memmap[VIRT_PCIE_PIO ].size,1 ,FDT_PCI_RANGE_MMIO ,2 , s->memmap[VIRT_PCIE_MMIO ].base, -2 , s->memmap[VIRT_PCIE_MMIO ].base,2 , s->memmap[VIRT_PCIE_MMIO ].size, +2 , s->memmap[VIRT_PCIE_MMIO ].base,2 , mmio32_size,1 ,FDT_PCI_RANGE_MMIO_64BIT ,2 , virt_high_pcie_memmap.base, 2, virt_high_pcie_memmap.base, 2, virt_high_pcie_memmap.size);
@@ -1130,6 +1147,132 @@ static void cxl_host_state_init(RISCVVirtState *s)cxl_fmws_update_mmio (); } +/* +* Assign PCI bus numbers to the bridges under @bus in depth-first order, +* the way firmware does during enumeration. QEMU leaves the secondary and +* subordinate bus registers at 0 until firmware programs them, so without +* this build_crs() would see an empty bus range for the CXL host bridge. +* @bus itself is numbered @bus_num. Returns the highest bus number used. +*/ +static int virt_cxl_assign_bus_numbers (PCIBus *bus,int bus_num) +{ +int max_bus = bus_num; +int next_bus = bus_num +1 ; +int devfn; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *dev = bus->devices[devfn]; +PCIBus *sec_bus; +int subordinate; + +if (!dev) { +continue ; + } +if ((dev->config[PCI_HEADER_TYPE ] & + ~PCI_HEADER_TYPE_MULTI_FUNCTION ) !=PCI_HEADER_TYPE_BRIDGE ) { +continue ; + } + + sec_bus =pci_bridge_get_sec_bus (PCI_BRIDGE (dev)); + subordinate =virt_cxl_assign_bus_numbers (sec_bus, next_bus); + +pci_set_byte (dev->config +PCI_PRIMARY_BUS , bus_num); +pci_set_byte (dev->config +PCI_SECONDARY_BUS , next_bus); +pci_set_byte (dev->config +PCI_SUBORDINATE_BUS , subordinate); + +if (subordinate > max_bus) { + max_bus = subordinate; + } + next_bus = subordinate +1 ; + } + +return max_bus; +} + +/* +* Simulate the PCI resource initialization that firmware (UEFI) would +* normally perform for the CXL host bridge. +* +* EDK2's PciBusDxe only recurses into devices that look like PCI-to-PCI +* bridges, while the pxb-cxl expander bridge presents as a class 0x0600 +* host bridge with a standard (type 0) header, so the firmware neither +* enumerates behind it nor assigns it a memory window or bus numbers. As a +* result the generic build_crs() path would produce an empty _CRS for the +* CXL host bridge (ACPI0016). +* +* Program each CXL root port the way firmware would: +* - assign the reserved 32-bit MMIO range (see create_fdt_pcie()) to its +* memory window, so build_crs() emits that range in the host bridge _CRS +* and excludes it from PCI0's _CRS; +* - set the primary/secondary/subordinate bus numbers, so build_crs() +* emits a correct bus-range resource. Without a bus range Linux only +* claims the single host-bridge bus and cannot enumerate the devices +* behind the root port. +* +* TODO: this handles a single pxb-cxl with a single root port, which is +* the supported topology for now. With multiple CXL host bridges or root +* ports the reserved window would need to be partitioned per bridge. +*/ +static void virt_cxl_init_bridge_windows (RISCVVirtState *s) +{ +hwaddr cxl_mmio_base, cxl_mmio_limit; +PCIBus *bus; + +if (!s->cxl_devices_state.is_enabled) { +return ; + } + + cxl_mmio_base = s->memmap[VIRT_PCIE_MMIO ].base + + s->memmap[VIRT_PCIE_MMIO ].size -VIRT_CXL_MMIO32_SIZE ; + cxl_mmio_limit = cxl_mmio_base +VIRT_CXL_MMIO32_SIZE -1 ; + + QLIST_FOREACH(bus, &s->pci_bus->child, sibling) { +int devfn; + +if (!pci_bus_is_root (bus) || !pci_bus_is_cxl (bus)) { +continue ; + } + +/* Assign bus numbers for the whole CXL subtree (firmware would). */ +virt_cxl_assign_bus_numbers (bus,pci_bus_num (bus)); + +/* Assign the reserved MMIO window to the root-port bridge. */ +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *dev = bus->devices[devfn]; +PCIBridge *br; + +if (!dev) { +continue ; + } +if ((dev->config[PCI_HEADER_TYPE ] & + ~PCI_HEADER_TYPE_MULTI_FUNCTION ) !=PCI_HEADER_TYPE_BRIDGE ) { +continue ; + } + + br =PCI_BRIDGE (dev); + +pci_set_word (dev->config +PCI_MEMORY_BASE , + (cxl_mmio_base >>16 ) &PCI_MEMORY_RANGE_MASK ); +pci_set_word (dev->config +PCI_MEMORY_LIMIT , + (cxl_mmio_limit >>16 ) &PCI_MEMORY_RANGE_MASK ); +pci_word_test_and_set_mask (dev->config +PCI_COMMAND , +PCI_COMMAND_MEMORY ); +pci_bridge_update_mappings (br); +break ; + } + } +} + +/* +* A PCI reset clears the bridge window and bus-number registers programmed +* above, so re-apply them after every reset, the way firmware would during +* its PCI enumeration. +*/ +static void virt_cxl_reset_bridge_windows (void *opaque) +{ +virt_cxl_init_bridge_windows (RISCV_VIRT_MACHINE (opaque)); +} +static FWCfgState *create_fw_cfg (const MachineState *ms,hwaddr base) {FWCfgState *fw_cfg;
@@ -1238,6 +1381,15 @@ static void virt_machine_done(Notifier *notifier, void *data)if (s->cxl_devices_state.is_enabled) {cxl_fmws_link_targets (&error_fatal); } + +/* +* Assign the CXL bridge memory window (simulating firmware) so the +* ACPI _CRS built for the CXL host bridge is populated. Do it here for +* the initial build, and register a reset handler so the window is +* re-applied after any PCI reset (which clears the bridge registers). +*/ +virt_cxl_init_bridge_windows (s); +qemu_register_reset (virt_cxl_reset_bridge_windows, s);hwaddr firmware_end_addr;vaddr kernel_start_addr;const char *firmware_name =riscv_default_firmware_name (&s->soc[0 ]);
Signature · 1 line
2.50.1
Re: [PATCH 3/4] hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges
Hi Chen Pei,
Quoted 6 lines · Chen Pei
- The CXL host bridge
_CRSis produced by the genericbuild_crs()path. EDK2's PciBusDxe only recurses into PCI-to-PCI bridges, while the pxb-cxl expander bridge presents as a class0x0600host bridge with a type-0 header, so firmware never enumerates behind it and leaves the CXL root port's window and bus-number registers unset;build_crs()would therefore return an empty_CRS. Simulate the
I see the same empty _CRS you traced for Igor on v2, but the cause sits one level up: riscv virt never tells the firmware that the expander bus is there.
etc/extra-pci-roots is how the other machines advertise expander root buses. hw/arm/virt.c, hw/i386/pc.c and hw/hppa/machine.c all call pci_bus_add_fw_cfg_extra_pci_roots(); arm got it in 09fad16744 as part of the pxb enablement there. hw/riscv/virt.c has no such call, and since PXB is default y only for Q35 and ARM_VIRT, this series is what first brings pxb to riscv at all -- so nothing has needed that fw_cfg entry until now.
Same binary, in-tree edk2-riscv, only difference on the command line being -fw_cfg name=etc/extra-pci-roots,file=<le64 1>:
PciHostBridgeUtilityGetRootBridgesBusScan: 1 extra root buses ... PciHostBridgeUtilityInitRootBridge: populated root bus 12, ...
PciBus: Resource Map for Bridge [0C|00|00]
Type = Mem32; Base = 0x40400000; Length = 0x200000; ...
Base = 0x40400000; Length = 0x10000; ... Owner = PCI [0D|00|00:10]
so PciBusDxe does go behind the pxb-cxl once it knows the root bus exists, and it assigns the root port window and the Type3 BARs itself.
With this patch dropped and pci_bus_add_fw_cfg_extra_pci_roots() called from virt_machine_done() instead, next to cxl_fmws_link_targets(), the regenerated DSDT has PC0C as
DWordMemory (ResourceProducer, ...)
0x40400000, // Range Minimum
0x405FFFFF, // Range Maximum
QWordMemory (ResourceProducer, ...)
0x0000000400100000, // Range Minimum
0x000000040010FFFF, // Range Maximum
WordBusNumber (...)
0x000C, // Range Minimum
0x000D, // Range Maximumwith PCI0 keeping the rest of the 32-bit aperture (0x40000000-0x403FFFFF and 0x40600000-0x7FFFFFFF). Firmware partitions the aperture itself, sizes the window to what is actually behind the bridge, and the 64-bit window comes along too. With a second pxb-cxl added (bus_nr 12 and 200), the two bridges get 0x40400000-0x405FFFFF and 0x40600000-0x407FFFFF, so the single-bridge limitation in the TODO would not arise either.
I have only looked at firmware enumeration and the generated tables here, not at a booting CXL guest, so your RVA22 run may well turn up something this misses. But if it holds, it drops the 256 MiB carve-out, virt_cxl_init_bridge_windows() and the reset handler, and it is the "fix UEFI to perform required initialization" option Igor raised on v2, with the UEFI side already in place.
Many thanks,
Junjie
Re: [PATCH 3/4] hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges
Hi Junjie,
Thanks a lot for the careful review, and especially for taking the time to actually run this and trace the root cause rather than just pointing at the symptom.
You are right, and I should correct my earlier judgement: I had convinced myself the firmware had no way to learn about the expander bus, so I went down the simulate-in-QEMU route. The real root cause sits one level up -- riscv virt simply never advertises the expander root bus to the firmware.
Thanks for the new direction and for the groundwork you have already done along this line. That is clearly the better approach, and it matches the "fix UEFI to perform required initialization" option Igor raised on v2.
I will rework 3/4 following your guidance. Before respinning, I will also do the end-to-end verification you flagged as missing -- on a booting CXL guest -- and I will sync back with you once I have the result.
Thanks again,
Pei
Re: [PATCH 3/4] hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges
Hi Junjie,
Following up on my earlier note -- I have done the rework and the end-to-end verification you flagged.
Reworked along the extra-pci-roots line: riscv virt now calls pci_bus_add_fw_cfg_extra_pci_roots() to advertise the pxb-cxl expander root bus, and EDK2's PciBusDxe enumerates behind it and assigns the root-port window and bus numbers. build_crs() then emits a correct _CRS for the ACPI0016 host bridge. The regenerated DSDT shows the firmware-assigned window (DWordMemory 0x40400000-0x405FFFFF, a QWordMemory, and WordBusNumber [0x0C-0x0D]), and as you noted it scales to multiple pxb-cxl (each gets its own window from the shared aperture).
Verified end-to-end on a booting RVA22 guest:
# cxl list -> mem0 (4 GiB), port1, root0 # cxl enable-memdev mem0 -> enabled 1 mem # cxl create-region -d decoder0.0 -w 1 mem0 -> region0 committed # daxctl online-memory dax0.0 -> onlined memory for 1 device # free -h -> 3.8Gi -> 7.8Gi (+4 GiB)
I will send the respun series shortly.
Thanks again,
Pei
PATCH 4/4 tests/qtest: Add RISC-V ACPI bios tables test for CXL
In reply to Chen PeiAdd test_acpi_riscv64_virt_tcg_cxl() to verify that enabling CXL on the RISC-V virt machine produces correct ACPI tables, including the ACPI0017 CXLM device with _DEP in the DSDT and the CEDT table.
The test boots with cxl=on, one pxb-cxl bus (bus_nr=12), a CXL root port, a cxl-type3 persistent memory device, and a fixed memory window, mirroring the existing x86 q35 CXL test pattern.
Since pxb-cxl is a root bus, using -cdrom causes QEMU to auto-plug the cdrom drive into pxb-cxl, triggering "Only PCI/PCIe bridges can be plugged into pxb-cxl". The ISO is instead attached explicitly via a virtio-scsi-pci controller on pcie.0, following the same approach as test_acpi_aarch64_virt_tcg_pxb().
--- tests/data/acpi/riscv64/virt/CEDT.cxl | Bin 0 -> 108 bytes tests/data/acpi/riscv64/virt/DSDT.cxl | Bin 0 -> 6212 bytes tests/qtest/bios-tables-test.c | 54 ++++++++++++++++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 tests/data/acpi/riscv64/virt/CEDT.cxl create mode 100644 tests/data/acpi/riscv64/virt/DSDT.cxl
tests/data/acpi/riscv64/virt/CEDT.cxl new file binary +0 -0
tests/data/acpi/riscv64/virt/DSDT.cxl new file binary +0 -0
tests/qtest/bios-tables-test.c +54 -0
66 lines not on this page · raw message
Signature · 1 line
2.50.1
Re: [PATCH 4/4] tests/qtest: Add RISC-V ACPI bios tables test for CXL
Hi Chen Pei,
Quoted 3 lines · Chen Pei
tests/data/acpi/riscv64/virt/CEDT.cxl | Bin 0 -> 108 bytes tests/data/acpi/riscv64/virt/DSDT.cxl | Bin 0 -> 6212 bytes tests/qtest/bios-tables-test.c | 54 ++++++++++++++++++++++++++
checkpatch errors on this one: expected files added together with tests. The flow at the top of bios-tables-test.c wants the new files added empty and listed in bios-tables-test-allowed-diff.h first, then the test, then the AML and the emptied allowed-diff in a final patch. The riscv64 goldens went in that way originally (cc3ba2422554). Alistair's Acked-by would need re-applying to whichever patch ends up holding the test code.
Worth settling 3/4 first -- if the _CRS ends up coming from firmware enumeration, the DSDT here changes anyway.
Many thanks,
Junjie
Re: [PATCH 4/4] tests/qtest: Add RISC-V ACPI bios tables test for CXL
Hi Junjie,
Thanks for the reminder.
I will first work on 3/4 along the extra-pci-roots line (and the end-to-end verification on a booting CXL guest), then regenerate the golden tables and respin this patch accordingly.
Thanks,
Pei
Hi all,
Apologies, I forgot to add the version tag in the subject prefix. This is actually v3 of the series; the changelog against v2 and v1 is in the cover letter. Please ignore the missing "v3" tag for now -- it will be tagged correctly in the next revision.
Sorry for the noise.
Thanks,
Pei
Hi Chen Pei,
Quoted 7 lines · Chen Pei
- hw/riscv/virt: The MMIO-window patch no longer touches the common gpex code (Igor). The gpex_cfg.cxl_mmio32 field and the
is_cxlstatic_CRSbranch are dropped; the ACPI0016_CRSnow comes from the genericbuild_crs()path. Since EDK2 does not enumerate the pxb-cxl expander bridge, riscv simulates the firmware PCI initialization (reserved window + depth-first bridge bus numbers) and re-applies it via a reset handler.
Ran this on master eea8fe61b8 (it still applies clean on bde2492aac). riscv64 bios-tables-test passes. In the generated DSDT the PC0C _CRS now comes out of build_crs() with the reserved window 0x70000000-0x7FFFFFFF and a bus range [0x0C,0x0D] taken from the real subordinate, PCI0 shrunk to 0x40000000-0x6FFFFFFF, and the CEDT CHBS and CFMWS matching the machine memmap. The machine-global window synthesis and the +15 bus range from v2 are gone.
system_reset from the monitor leaves the root port with the same window and secondary/subordinate numbers, so the reset handler does what it says.
One larger question on 3/4 about where that enumeration should come from, and a note on 4/4.
Many thanks,
Junjie