QEMU: CXL Type-2 device passthrough via vfio-pci
cover QEMU: CXL Type-2 device passthrough via vfio-pci
From: Manish Honap <[email protected]>
This series adds QEMU-side support for passing CXL Type-2 devices (GPUs and accelerators with host-managed device memory) to VMs via vfio-pci.
It pairs with the kernel series "vfio/pci: CXL Type-2 passthrough"[1] posted to the vfio mailing list. Patches 3-7 need that kernel series present to do anything useful. I am new to QEMU development, so please forgive and point me in the right direction for correct infrastructure decisions.
Background
----------
CXL Type-2 devices expose device memory (CXL.mem) through HDM decoders. The kernel vfio-pci driver shadows the HDM Decoder Capability registers so userspace can observe and control decoder commits without touching the hardware register page directly.
Without this series, the guest never sees the device memory range and the HDM decoder goes unconfigured. The device shows up but its memory is unreachable.
Design decisions
----------------
CXL.mem is exposed to the guest as a dedicated GPA window declared in ACPI (CEDT/CFMWS) rather than a PCI BAR. The HDM decoder BASE must match the CFMWS base and remain stable; BAR assignment is not stable. A separate VIRT_HIGH_CXL_MMIO window in the ARM virt memory map carries this GPA range, independent of the existing PCIe MMIO slots.
The Component Register BAR contains two distinct ranges. Accelerator register windows are passed through as direct hardware mmaps via VFIO_REGION_INFO_CAP_SPARSE_MMAP. The HDM Decoder Capability block is excluded from that sparse list by the kernel and must be intercepted by QEMU to track decoder state. A single priority-1 COMP_REGS overlay placed at hdm_regs_offset inside the BAR container wins over any hardware-backed alias at the same offset, with no per-window aliasing required.
The guest has no mechanism to remap host physical mappings. QEMU programs decoder 0 with the CFMWS base through the kernel's COMP_REGS shadow at machine_done time, after all devices are realized and before the guest starts. The notifier is registered only for devices the kernel reports as firmware-committed (VFIO_CXL_CAP_FIRMWARE_COMMITTED).
The CXL.mem MemoryRegion is a mmap-backed RAM-device region backed by a VM_IO|VM_PFNMAP VMA. The VFIO MemoryListener would attempt an IOMMU DMA mapping for it when it is added to system_memory, which always
fails: pin_user_pages() refuses VM_IO pages. No IOMMU mapping is needed for these regions - CPU access goes via KVM Stage-2 page faults and device DMA to RAM uses separate per-RAM-section IOMMU entries. The listener is extended to skip the mapping attempt for VFIO-owned RAM-device regions.
pxb-cxl bridges had no _DSM method. Without _DSM function 5 the OS defaults to treating PCI configuration as reassignable.
On machines with firmware-committed HDM decoders that reassignment breaks the CXL.mem mapping, so the _DSM is added with preserve_config=true for ARM and false for x86.
Known issues:
- The bios-tables test will fail due to the _DSM addition.
A fix will be provided in a follow-up round.
- VFIO_CXL_CAP_CACHE_CAPABLE will require additional handling.
- Devices with multiple firmware-committed HDM decoders are not fully supported.
- Non-firmware-committed devices are not supported.
- linux-headers sync is manual and temporary; once the kernel series is merged, this patch will be replaced with script generated update.
[1] https://lore.kernel.org/linux-cxl/[email protected]
Manish Honap (9):
hw/arm/virt: Add CXL FMWS PA window for device memory
cxl: Add preserve_config to pxb-cxl OSC method
linux-headers: Update vfio.h for CXL Type-2 device passthrough hw/vfio/region: Add vfio_region_setup_with_ops() for custom region ops hw/vfio/pci: Add CXL Type-2 device detection and region setup hw/vfio/pci: Wire CXL component-register BAR with COMP_REGS overlay hw/vfio+cxl: Program HDM decoder 0 at machine_done for
firmware-committed devices
hw/arm/smmu-common: Allow pxb-cxl as SMMUv3 primary bus vfio/listener: Skip DMA mapping for VFIO-owned RAM-device regions
hw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +- hw/arm/smmu-common.c | 17 +- hw/arm/virt-acpi-build.c | 5 + hw/arm/virt.c | 7 + hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 8 + hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 43 +++- hw/vfio/listener.c | 14 ++ hw/vfio/pci.c | 411 +++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 15 ++ hw/vfio/region.c | 15 +- hw/vfio/trace-events | 6 + hw/vfio/vfio-region.h | 3 + include/hw/acpi/cxl.h | 2 +- include/hw/arm/virt.h | 2 + include/hw/cxl/cxl_host.h | 10 + include/hw/pci-host/gpex.h | 2 + linux-headers/linux/vfio.h | 18 ++ 20 files changed, 570 insertions(+), 18 deletions(-)
--
2.25.1
RFC 1/9 hw/arm/virt: Add CXL FMWS PA window for device memory
From: Manish Honap <[email protected]>
CXL VFIO passthrough needs a stable guest physical address range for device memory (DPA) that falls inside a CFMWS entry the guest discovers from ACPI CEDT. Without a dedicated range in the address map, the HDM decoder has nowhere to point.
Add VIRT_HIGH_CXL_MMIO immediately after the second PCIe MMIO window. It gets its own highmem_cxl_mmio flag in VirtMachineState rather than sharing highmem_cxl, so the two slots are independently controllable even though both are currently tied to CXL bridge presence.
The base and size flow through GPEXConfig.cxl_mmio to acpi_dsdt_add_gpex(), which carves out a QWord memory descriptor in the first CXL root bridge's _CRS. The CFMWS window is system-wide, so only the first CXL bridge gets the descriptor - subsequent ones would produce duplicate resource claims for the same range.
build_crs() already emits the bridge's own 64-bit ranges into crs. The CFMWS window is a separate system-wide range, so only that window is appended as a new QWord descriptor; the bridge ranges are not re-emitted. A warn_report() fires if the CFMWS window overlaps any existing bridge 64-bit range, since that would indicate an address layout conflict.
--- hw/arm/virt-acpi-build.c | 5 +++++ hw/arm/virt.c | 9 +++++++++ hw/pci-host/gpex-acpi.c | 40 ++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 2 ++ include/hw/pci-host/gpex.h | 1 + 5 files changed, 57 insertions(+)
hw/arm/virt-acpi-build.c +5 -0
@@ -176,6 +176,11 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; } +if (vms->highmem_cxl) { + cfg.cxl_mmio.base = memmap[VIRT_HIGH_CXL_MMIO ].base; + cfg.cxl_mmio.size = memmap[VIRT_HIGH_CXL_MMIO ].size; + } +acpi_dsdt_add_gpex (scope, &cfg); QLIST_FOREACH(bus, &vms->bus->child, sibling) {if (pci_bus_is_cxl (bus)) {
hw/arm/virt.c +9 -0
@@ -211,6 +211,8 @@ static const MemMapEntry base_memmap[] = {#define DEFAULT_HIGH_PCIE_MMIO_SIZE_GB 512#define DEFAULT_HIGH_PCIE_MMIO_SIZE (DEFAULT_HIGH_PCIE_MMIO_SIZE_GB * GiB) +#define DEFAULT_HIGH_CXL_MMIO_SIZE DEFAULT_HIGH_PCIE_MMIO_SIZE +/* * Highmem IO Regions: This memory map is floating, located after the RAM. * Each MemMapEntry base (GPA) will be dynamically computed, depending on the
@@ -237,6 +239,11 @@ static MemMapEntry extended_memmap[] = { [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, /* Second PCIe window */ [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,DEFAULT_HIGH_PCIE_MMIO_SIZE }, +/* +* CXL FMWS guest PA window - separate from PCIe MMIO so the two are +* independently sizeable. Same default size for now. + */ + [VIRT_HIGH_CXL_MMIO] = { 0x0, DEFAULT_HIGH_CXL_MMIO_SIZE }, /* Any CXL Fixed memory windows come here */ };
@@ -1724,6 +1731,7 @@ static void create_cxl_host_reg_region(VirtMachineState *vms) vms->memmap[VIRT_CXL_HOST].size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_CXL_HOST ].base, mr); vms->highmem_cxl = true; + vms->highmem_cxl_mmio = true; }static void create_platform_bus (VirtMachineState *vms)
@@ -1897,6 +1905,7 @@ static inline bool *virt_get_high_memmap_enabled(VirtMachineState *vms, &vms->highmem_cxl, &vms->highmem_ecam, &vms->highmem_mmio, + &vms->highmem_cxl_mmio, }; assert(ARRAY_SIZE(extended_memmap) - VIRT_LOWMEMMAP_LAST ==
hw/pci-host/gpex-acpi.c +40 -0
@@ -7,6 +7,7 @@#include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" #include "hw/acpi/cxl.h" +#include "qemu/error-report.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq,Aml *scope,uint8_t bus_num)
@@ -108,6 +109,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)CrsRangeSet crs_range_set;CrsRangeEntry *entry;int i; +bool first_cxl = true;/* start to construct the tables for pxb */ crs_range_set_init (&crs_range_set);
@@ -161,6 +163,44 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)*/ crs =build_crs (PCI_HOST_BRIDGE (BUS (bus)->parent), &crs_range_set, cfg->pio.base,0 ,0 ,0 ); +if (is_cxl && first_cxl && cfg->cxl_mmio.size) { +uint64_t cfmws_end = cfg->cxl_mmio.base + + cfg->cxl_mmio.size -1 ; + +/* +* The CXL Fixed Memory Window (CFMWS) is a system-wide GPA +* range. Only the first CXL root bridge emits the QWord +* descriptor; adding it to every bridge would give the OS +* duplicate resource claims for the same range. +* +* build_crs() has already appended the bridge's own 64-bit +* ranges into crs. Do not copy them again here; only append +* the CFMWS window itself as a new QWord descriptor. +* +* Warn if the CFMWS window overlaps any range already claimed +* by the bridge; in the current address layout they should be +* disjoint, but catch it early if the layout ever changes. +*/ +for (i =0 ; i < crs_range_set.mem_64bit_ranges->len; i++) { + entry =g_ptr_array_index (crs_range_set.mem_64bit_ranges, + i); +if (entry->base <= cfmws_end && + entry->limit >= cfg->cxl_mmio.base) { +warn_report ("CXL CFMWS [0x%" PRIx64"-0x%" PRIx64"] " +"overlaps CXL root bridge 64-bit range " +"[0x%" PRIx64"-0x%" PRIx64"]" , + cfg->cxl_mmio.base, cfmws_end, + entry->base, entry->limit); + } + } +aml_append (crs, +aml_qword_memory (AML_POS_DECODE ,AML_MIN_FIXED , +AML_MAX_FIXED ,AML_NON_CACHEABLE ,AML_READ_WRITE , +0x0000 , cfg->cxl_mmio.base, cfmws_end,0x0000 , + cfg->cxl_mmio.size)); + first_cxl = false; + } +aml_append (dev,aml_name_decl ("_CRS" , crs));if (is_cxl) {
include/hw/arm/virt.h +2 -0
@@ -91,6 +91,7 @@ enum {VIRT_CXL_HOST ,VIRT_HIGH_PCIE_ECAM ,VIRT_HIGH_PCIE_MMIO , + VIRT_HIGH_CXL_MMIO, }; typedef enum VirtIOMMUType {
@@ -147,6 +148,7 @@ struct VirtMachineState {bool highmem;bool highmem_compact;bool highmem_cxl; +bool highmem_cxl_mmio;/* VIRT_HIGH_CXL_MMIO window; follows highmem_cxl */ bool highmem_ecam;bool highmem_mmio;bool highmem_redists;
include/hw/pci-host/gpex.h +1 -0
@@ -43,6 +43,7 @@ struct GPEXConfig {MemMapEntry mmio32;MemMapEntry mmio64;MemMapEntry pio; +MemMapEntry cxl_mmio;int irq;PCIBus *bus;bool pci_native_hotplug;
Signature · 1 line
2.25.1
Re: [RFC 1/9] hw/arm/virt: Add CXL FMWS PA window for device memory
Quoted 11 lines · On Mon, 27 Apr 2026 23:42:27 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
CXL VFIO passthrough needs a stable guest physical address range for device memory (DPA) that falls inside a CFMWS entry the guest discovers from ACPI CEDT. Without a dedicated range in the address map, the HDM decoder has nowhere to point.
Add
VIRT_HIGH_CXL_MMIOimmediately after the second PCIe MMIO window. It gets its ownhighmem_cxl_mmioflag in VirtMachineState rather than sharinghighmem_cxl, so the two slots are independently controllable even though both are currently tied to CXL bridge presence.
Can we clearly state here what they are. highmem_cxl is the root bridge registers. I'm not entirely sure I yet understand what highmem_cxl_mmio
Quoted 6 lines
The base and size flow through GPEXConfig.cxl_mmio to
acpi_dsdt_add_gpex(), which carves out a QWord memory descriptor in the first CXL root bridge's_CRS. The CFMWS window is system-wide, so only the first CXL bridge gets the descriptor - subsequent ones would produce duplicate resource claims for the same range.
I don't follow. CFMWS are not system wide. They refer to one or more of the root bridges. I suppose you could argue that the 'or more' bit makes them kind of system wide. Right now I'm not even sure why they are in CRS at all as opposed to being obtained from CEDT.
Maybe a spec reference for why a bridge should be claiming these regions?
I'm not understanding yet why we can't use the existing CFMWS support so please add information on that tot his commit message.
Quoted 48 lines
build_crs()already emits the bridge's own 64-bit ranges into crs. The CFMWS window is a separate system-wide range, so only that window is appended as a new QWord descriptor; the bridge ranges are not re-emitted. Awarn_report()fires if the CFMWS window overlaps any existing bridge 64-bit range, since that would indicate an address layout conflict.Signed-off-by Zhi Wang <[email protected]>Signed-off-by Manish Honap <[email protected]>--- hw/arm/virt-acpi-build.c | 5 +++++ hw/arm/virt.c | 9 +++++++++ hw/pci-host/gpex-acpi.c | 40 ++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 2 ++ include/hw/pci-host/gpex.h | 1 + 5 files changed, 57 insertions(+)hw/arm/virt-acpi-build.c +5
-0@@ -176,6 +176,11 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; } +if (vms->highmem_cxl) { + cfg.cxl_mmio.base = memmap[VIRT_HIGH_CXL_MMIO ].base; + cfg.cxl_mmio.size = memmap[VIRT_HIGH_CXL_MMIO ].size; + } +acpi_dsdt_add_gpex (scope, &cfg); QLIST_FOREACH(bus, &vms->bus->child, sibling) {if (pci_bus_is_cxl (bus)) {hw/arm/virt.c +2
-0@@ -211,6 +211,8 @@ static const MemMapEntry base_memmap[] = {#define DEFAULT_HIGH_PCIE_MMIO_SIZE_GB 512#define DEFAULT_HIGH_PCIE_MMIO_SIZE (DEFAULT_HIGH_PCIE_MMIO_SIZE_GB * GiB) +#define DEFAULT_HIGH_CXL_MMIO_SIZE DEFAULT_HIGH_PCIE_MMIO_SIZE +/* * Highmem IO Regions: This memory map is floating, located after the RAM. * Each MemMapEntry base (GPA) will be dynamically computed, depending on the
hw/arm/virt.c static MemMapEntry extended_memmap[] = { · line 247
Quoted 9 lines
hw/arm/virt.c +5
-0@@ -237,6 +239,11 @@ static MemMapEntry extended_memmap[] = { [VIRT_HIGH_PCIE_ECAM] = { 0x0, 256 * MiB }, /* Second PCIe window */ [VIRT_HIGH_PCIE_MMIO] = { 0x0, DEFAULT_HIGH_PCIE_MMIO_SIZE }, +/* +* CXL FMWS guest PA window - separate from PCIe MMIO so the two are +* independently sizeable. Same default size for now. + */ + [VIRT_HIGH_CXL_MMIO] = { 0x0, DEFAULT_HIGH_CXL_MMIO_SIZE },/* Any CXL Fixed memory windows come here */
Why not use the infra this comment is talking about?
Quoted 115 lines
hw/arm/virt.c +2
-0};@@ -1724,6 +1731,7 @@ static void create_cxl_host_reg_region(VirtMachineState *vms) vms->memmap[VIRT_CXL_HOST].size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_CXL_HOST ].base, mr); vms->highmem_cxl = true; + vms->highmem_cxl_mmio = true; }static void create_platform_bus (VirtMachineState *vms)@@ -1897,6 +1905,7 @@ static inline bool *virt_get_high_memmap_enabled(VirtMachineState *vms, &vms->highmem_cxl, &vms->highmem_ecam, &vms->highmem_mmio, + &vms->highmem_cxl_mmio, }; assert(ARRAY_SIZE(extended_memmap) - VIRT_LOWMEMMAP_LAST ==hw/pci-host/gpex-acpi.c +40
-0@@ -7,6 +7,7 @@#include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" #include "hw/acpi/cxl.h" +#include "qemu/error-report.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq,Aml *scope,uint8_t bus_num)@@ -108,6 +109,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)CrsRangeSet crs_range_set;CrsRangeEntry *entry;int i; +bool first_cxl = true;/* start to construct the tables for pxb */ crs_range_set_init (&crs_range_set);@@ -161,6 +163,44 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)*/ crs =build_crs (PCI_HOST_BRIDGE (BUS (bus)->parent), &crs_range_set, cfg->pio.base,0 ,0 ,0 ); +if (is_cxl && first_cxl && cfg->cxl_mmio.size) { +uint64_t cfmws_end = cfg->cxl_mmio.base + + cfg->cxl_mmio.size -1 ; + +/* +* The CXL Fixed Memory Window (CFMWS) is a system-wide GPA +* range. Only the first CXL root bridge emits the QWord +* descriptor; adding it to every bridge would give the OS +* duplicate resource claims for the same range. +* +* build_crs() has already appended the bridge's own 64-bit +* ranges into crs. Do not copy them again here; only append +* the CFMWS window itself as a new QWord descriptor. +* +* Warn if the CFMWS window overlaps any range already claimed +* by the bridge; in the current address layout they should be +* disjoint, but catch it early if the layout ever changes. +*/ +for (i =0 ; i < crs_range_set.mem_64bit_ranges->len; i++) { + entry =g_ptr_array_index (crs_range_set.mem_64bit_ranges, + i); +if (entry->base <= cfmws_end && + entry->limit >= cfg->cxl_mmio.base) { +warn_report ("CXL CFMWS [0x%" PRIx64"-0x%" PRIx64"] " +"overlaps CXL root bridge 64-bit range " +"[0x%" PRIx64"-0x%" PRIx64"]" , + cfg->cxl_mmio.base, cfmws_end, + entry->base, entry->limit); + } + } +aml_append (crs, +aml_qword_memory (AML_POS_DECODE ,AML_MIN_FIXED , +AML_MAX_FIXED ,AML_NON_CACHEABLE ,AML_READ_WRITE , +0x0000 , cfg->cxl_mmio.base, cfmws_end,0x0000 , + cfg->cxl_mmio.size)); + first_cxl = false; + } +aml_append (dev,aml_name_decl ("_CRS" , crs));if (is_cxl) {include/hw/arm/virt.h +2
-0@@ -91,6 +91,7 @@ enum {VIRT_CXL_HOST ,VIRT_HIGH_PCIE_ECAM ,VIRT_HIGH_PCIE_MMIO , + VIRT_HIGH_CXL_MMIO, }; typedef enum VirtIOMMUType {@@ -147,6 +148,7 @@ struct VirtMachineState {bool highmem;bool highmem_compact;bool highmem_cxl; +bool highmem_cxl_mmio;/* VIRT_HIGH_CXL_MMIO window; follows highmem_cxl */ bool highmem_ecam;bool highmem_mmio;bool highmem_redists;include/hw/pci-host/gpex.h +1
-0@@ -43,6 +43,7 @@ struct GPEXConfig {MemMapEntry mmio32;MemMapEntry mmio64;MemMapEntry pio; +MemMapEntry cxl_mmio;int irq;PCIBus *bus;bool pci_native_hotplug;
RE: [RFC 1/9] hw/arm/virt: Add CXL FMWS PA window for device memory
Quoted 119 lines
-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 19 May 2026 18:10
To: Manish Honap <[email protected]>
Cc: Alex Williamson <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Ankit Agrawal <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; alejandro.lucero- [email protected]; Vikram Sethi <[email protected]>; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]; qemu- [email protected]
Subject: Re: [RFC 1/9] hw/arm/virt: Add CXL FMWS PA window for device memoryExternal email: Use caution opening links or attachments
Quoted 11 lines · On Mon, 27 Apr 2026 23:42:27 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
CXL VFIO passthrough needs a stable guest physical address range for device memory (DPA) that falls inside a CFMWS entry the guest discovers from ACPI CEDT. Without a dedicated range in the address map, the HDM decoder has nowhere to point.
Add
VIRT_HIGH_CXL_MMIOimmediately after the second PCIe MMIO window. It gets its ownhighmem_cxl_mmioflag in VirtMachineState rather than sharinghighmem_cxl, so the two slots are independently controllable even though both are currently tied to CXL bridge presence.Can we clearly state here what they are.
highmem_cxlis the root bridge registers. I'm not entirely sure I yet understand whathighmem_cxl_mmioQuoted 6 lines
The base and size flow through GPEXConfig.cxl_mmio to
acpi_dsdt_add_gpex(), which carves out a QWord memory descriptor in the first CXL root bridge's_CRS. The CFMWS window is system-wide, so only the first CXL bridge gets the descriptor - subsequent ones would produce duplicate resource claims for the same range.I don't follow. CFMWS are not system wide. They refer to one or more of the root bridges. I suppose you could argue that the 'or more' bit makes them kind of system wide. Right now I'm not even sure why they are in CRS at all as opposed to being obtained from CEDT.
Maybe a spec reference for why a bridge should be claiming these regions?
I'm not understanding yet why we can't use the existing CFMWS support so please add information on that tot his commit message.
Quoted 8 lines
build_crs()already emits the bridge's own 64-bit ranges into crs. The CFMWS window is a separate system-wide range, so only that window is appended as a new QWord descriptor; the bridge ranges are not re-emitted. Awarn_report()fires if the CFMWS window overlaps any existing bridge 64-bit range, since that would indicate an address layout conflict.Quoted 6 lines
Signed-off-by Zhi Wang <[email protected]>Signed-off-by Manish Honap <[email protected]>--- hw/arm/virt-acpi-build.c | 5 +++++ hw/arm/virt.c | 9 +++++++++ hw/pci-host/gpex-acpi.c | 40++++++++++++++++++++++++++++++++++++++
hw/arm/virt-acpi-build.c static void acpi_dsdt_add_pci(Aml *scope, const
Quoted 9 lines
include/hw/arm/virt.h | 2 ++ include/hw/pci-host/gpex.h | 1 + 5 files changed, 57 insertions(+)hw/arm/virt-acpi-build.c index renamed from hw/arm/virt-acpi-build.c +0
-0
591cfc993c..863e0680fb100644hw/arm/virt-acpi-build.c +0
-0@@ -176,6 +176,11 @@ static void acpi_dsdt_add_pci(Aml *scope, constMemMapEntry *memmap,
Quoted 16 lines
hw/arm/virt-acpi-build.c +5
-0cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; } +if (vms->highmem_cxl) { + cfg.cxl_mmio.base = memmap[VIRT_HIGH_CXL_MMIO ].base; + cfg.cxl_mmio.size = memmap[VIRT_HIGH_CXL_MMIO ].size; + } +acpi_dsdt_add_gpex (scope, &cfg); QLIST_FOREACH(bus, &vms->bus->child, sibling) {if (pci_bus_is_cxl (bus)) {hw/arm/virt.c index renamed from hw/arm/virt.c +0
-0
ec0d8475ca..fa07819401100644hw/arm/virt.c static const MemMapEntry base_memmap[] = { · line 217
Quoted 7 lines
hw/arm/virt.c +2
-0@@ -211,6 +211,8 @@ static const MemMapEntry base_memmap[] = {#define DEFAULT_HIGH_PCIE_MMIO_SIZE_GB 512 #define DEFAULT_HIGH_PCIE_MMIO_SIZE (DEFAULT_HIGH_PCIE_MMIO_SIZE_GB * GiB) +#define DEFAULT_HIGH_CXL_MMIO_SIZE DEFAULT_HIGH_PCIE_MMIO_SIZE +/* * Highmem IO Regions: This memory map is floating, located after the RAM.
Quoted 2 lines
hw/arm/virt.c +0
-0* Each MemMapEntry base (GPA) will be dynamically computed, depending on the @@ -237,6 +239,11 @@ static MemMapEntry
extended_memmap[] = {
hw/arm/virt.c
Quoted 5 lines
[VIRT_HIGH_PCIE_ECAM] = { 0x0, 256 * MiB }, /* Second PCIe window */ [VIRT_HIGH_PCIE_MMIO] = { 0x0, DEFAULT_HIGH_PCIE_MMIO_SIZE },hw/arm/virt.c +2
-0+/* +* CXL FMWS guest PA window - separate from PCIe MMIO so the two
Quoted 7 lines
are
hw/arm/virt.c static MemMapEntry extended_memmap[] = { · line 247
Quoted 4 lines
hw/arm/virt.c +3
-0+* independently sizeable. Same default size for now. + */ + [VIRT_HIGH_CXL_MMIO] = { 0x0, DEFAULT_HIGH_CXL_MMIO_SIZE },/* Any CXL Fixed memory windows come here */ Why not use the infra this comment is talking about?
Hello Jonathan,
You're right that the new VIRT_HIGH_CXL_MMIO slot is redundant. I will use the existing CFMWS support in V2.
Quoted 137 lines
hw/arm/virt.c static void
Quoted 4 lines
};
hw/arm/virt.c +0
-0@@ -1724,6 +1731,7 @@ static void
create_cxl_host_reg_region(VirtMachineState *vms)Quoted 9 lines
hw/arm/virt.c +1
-0vms->memmap[VIRT_CXL_HOST].size);memory_region_add_subregion (sysmem, vms- memmap[VIRT_CXL_HOST ].base, mr); vms->highmem_cxl = true; + vms->highmem_cxl_mmio = true; }static void
create_platform_bus(VirtMachineState *vms)@@ -1897,6
+1905,7 @@ static inline bool*virt_get_high_memmap_enabled(VirtMachineState *vms,
Quoted 19 lines
&vms->highmem_cxl, &vms->highmem_ecam, &vms->highmem_mmio, + &vms->highmem_cxl_mmio, }; assert(ARRAY_SIZE(extended_memmap) - VIRT_LOWMEMMAP_LAST == diff--git
a/hw/pci-host/gpex-acpi.cb/hw/pci-host/gpex-acpi.cindexd9820f9b41..7de57bbc46100644hw/pci-host/gpex-acpi.c +1
-0@@ -7,6 +7,7 @@#include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" #include "hw/acpi/cxl.h" +#include "qemu/error-report.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq, Aml *scope, uint8_tQuoted 1 line
bus_num) @@ -108,6 +109,7 @@ void acpi_dsdt_add_gpex(Aml *scope,struct GPEXConfig *cfg)
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct
Quoted 8 lines
CrsRangeSet crs_range_set; CrsRangeEntry *entry; int i;hw/pci-host/gpex-acpi.c +1
-0+bool first_cxl = true;/* start to construct the tables for pxb */ crs_range_set_init (&crs_range_set);@@ -161,6 +163,44 @@ void acpi_dsdt_add_gpex(Aml *scope, structGPEXConfig *cfg)
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 164
Quoted 2 lines
hw/pci-host/gpex-acpi.c +0
-0*/ crs = build_crs(PCI_HOST_BRIDGE(BUS(bus)->parent),
&crs_range_set,hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
Quoted 7 lines
hw/pci-host/gpex-acpi.c +6
-0cfg->pio.base, 0, 0, 0); +if (is_cxl && first_cxl && cfg->cxl_mmio.size) { +uint64_t cfmws_end = cfg->cxl_mmio.base + + cfg->cxl_mmio.size -1 ; + +/* +* The CXL Fixed Memory Window (CFMWS) is a system- wide GPA
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 172
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+* range. Only the first CXL root bridge emits the QWord
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 173
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+* descriptor; adding it to every bridge would give the OS
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
Quoted 3 lines
hw/pci-host/gpex-acpi.c +3
-0+* duplicate resource claims for the same range. +* +* build_crs() has already appended the bridge's own 64-bit
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 177
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+* ranges into crs. Do not copy them again here; only append
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
Quoted 3 lines
hw/pci-host/gpex-acpi.c +3
-0+* the CFMWS window itself as a new QWord descriptor. +* +* Warn if the CFMWS window overlaps any range already claimed
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 181
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+* by the bridge; in the current address layout they should be
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 182
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+* disjoint, but catch it early if the layout ever changes.
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 184
Quoted 2 lines
hw/pci-host/gpex-acpi.c +2
-0+*/ + for (i = 0; i < crs_range_set.mem_64bit_ranges->len;i++) {
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 185
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+ entry =g_ptr_array_index(crs_range_set.mem_64bit_ranges,
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
Quoted 4 lines
hw/pci-host/gpex-acpi.c +4
-0+ i); + if (entry->base <= cfmws_end && + entry->limit >= cfg->cxl_mmio.base) { + warn_report("CXL CFMWS [0x%"PRIx64"-0x%"PRIx64"] "
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 190
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+ "overlaps CXL root bridge 64-bitrange "
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
Quoted 8 lines
hw/pci-host/gpex-acpi.c +8
-0+"[0x%" PRIx64"-0x%" PRIx64"]" , + cfg->cxl_mmio.base, cfmws_end, + entry->base, entry->limit); + } + } + aml_append(crs, + aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, + AML_MAX_FIXED, AML_NON_CACHEABLE,
AML_READ_WRITE,hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct · line 199
Quoted 1 line
hw/pci-host/gpex-acpi.c +1
-0+ 0x0000, cfg->cxl_mmio.base, cfmws_end,
0x0000,Quoted 40 lines
hw/pci-host/gpex-acpi.c +4
-0+ cfg->cxl_mmio.size)); + first_cxl = false; + } +aml_append (dev,aml_name_decl ("_CRS" , crs));if (is_cxl) {include/hw/arm/virt.h index renamed from include/hw/arm/virt.h +0
-0
5fcbd1c76f..88bb3c0bdf100644include/hw/arm/virt.h +3
-0@@ -91,6 +91,7 @@ enum {VIRT_CXL_HOST ,VIRT_HIGH_PCIE_ECAM ,VIRT_HIGH_PCIE_MMIO , + VIRT_HIGH_CXL_MMIO, }; typedef enum VirtIOMMUType {@@ -147,6 +148,7 @@ struct VirtMachineState {bool highmem;bool highmem_compact;bool highmem_cxl; +bool highmem_cxl_mmio;/* VIRT_HIGH_CXL_MMIO window; follows +highmem_cxl */ bool highmem_ecam;bool highmem_mmio;bool highmem_redists;include/hw/pci-host/gpex.h +1
-0@@ -43,6 +43,7 @@ struct GPEXConfig {MemMapEntry mmio32;MemMapEntry mmio64;MemMapEntry pio; +MemMapEntry cxl_mmio;int irq;PCIBus *bus;bool pci_native_hotplug;
RFC 2/9 cxl: Add preserve_config to pxb-cxl OSC method
In reply to [email protected]From: Manish Honap <[email protected]>
Before this patch, pxb-cxl bridges had no _DSM method at all. When the OS called _DSM on a CXL host bridge, ACPI returned an error and the OS defaulted to reassigning resources across suspend/resume. On machines where firmware pre-commits the HDM decoder, that reassignment breaks the DPA mapping.
Wire preserve_config through GPEXConfig into build_cxl_osc_method() so pxb-cxl host bridges get a _DSM method that signals the OS to keep resource assignments stable when needed. The _DSM function 5 (preserve firmware PCI configuration) is the mechanism used to convey this.
build_pci_host_bridge_dsm_method() is promoted from static to exported so cxl.c can call it without duplicating the AML.
The x86 build_cxl_osc_method() call site passes false since x86 does not use firmware-committed HDM decoders.
build_cxl_osc_method is renamed to acpi_dsdt_add_cxl_host_bridge_methods The function now appends both the CXL _OSC method and the _DSM method, so its old name is misleading. Renamed it to match the pxb-pcie analogue acpi_dsdt_add_host_bridge_methods(), making the two root bridge code paths symmetric. No AML change.
--- hw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +++- hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 5 +++-- include/hw/acpi/cxl.h | 2 +- include/hw/pci-host/gpex.h | 1 + 6 files changed, 10 insertions(+), 6 deletions(-)
hw/acpi/cxl-stub.c +1 -1
@@ -6,7 +6,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/cxl.h" -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config) {g_assert_not_reached (); }
hw/acpi/cxl.c +3 -1
@@ -23,6 +23,7 @@#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" #include "hw/cxl/cxl_host.h" +#include "hw/pci-host/gpex.h" #include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h"
@@ -320,11 +321,12 @@ static Aml *__build_cxl_osc_method(void)return method; } -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config) {aml_append (dev,aml_name_decl ("SUPP" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRL" ,aml_int (0 )));aml_append (dev,aml_name_decl ("SUPC" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRC" ,aml_int (0 )));aml_append (dev,__build_cxl_osc_method ()); +aml_append (dev,build_pci_host_bridge_dsm_method (preserve_config)); }
hw/i386/acpi-build.c +1 -1
@@ -1013,7 +1013,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (aml_pkg,aml_eisaid ("PNP0A08" ));aml_append (aml_pkg,aml_eisaid ("PNP0A03" ));aml_append (dev,aml_name_decl ("_CID" , aml_pkg)); -build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, false); }else if (pci_bus_is_express (bus)) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));
hw/pci-host/gpex-acpi.c +3 -2
@@ -52,7 +52,7 @@ static void acpi_dsdt_add_pci_route_table(Aml *dev, uint32_t irq, } } -static Aml *build_pci_host_bridge_dsm_method (bool preserve_config) +Aml *build_pci_host_bridge_dsm_method (bool preserve_config) {Aml *method =aml_method ("_DSM" ,4 ,AML_NOTSERIALIZED );Aml *UUID , *ifctx, *ifctx1, *buf;
@@ -204,7 +204,8 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)aml_append (dev,aml_name_decl ("_CRS" , crs));if (is_cxl) { -build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, + cfg->preserve_config); }else {/* pxb bridges do not have ACPI PCI Hot-plug enabled */ acpi_dsdt_add_host_bridge_methods(dev, true,
include/hw/acpi/cxl.h +1 -1
@@ -24,7 +24,7 @@void cxl_build_cedt (GArray *table_offsets,GArray *table_data,BIOSLinker *linker,const char *oem_id,const char *oem_table_id,CXLState *cxl_state); -void build_cxl_osc_method (Aml *dev); +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config);void build_cxl_dsm_method (Aml *dev);#endif
include/hw/pci-host/gpex.h +1 -0
@@ -73,6 +73,7 @@ struct GPEXHost {int gpex_set_irq_num (GPEXHost *s,int index,int gsi);void acpi_dsdt_add_gpex (Aml *scope,struct GPEXConfig *cfg); +Aml *build_pci_host_bridge_dsm_method (bool preserve_config);void acpi_dsdt_add_gpex_host (Aml *scope,uint32_t irq);#define PCI_HOST_PIO_BASE "x-pio-base"
Signature · 1 line
2.25.1
Re: [RFC 2/9] cxl: Add preserve_config to pxb-cxl OSC method
Quoted 7 lines · On Mon, 27 Apr 2026 23:42:28 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
Before this patch, pxb-cxl bridges had no
_DSMmethod at all. When the OS called_DSMon a CXL host bridge, ACPI returned an error and the OS defaulted to reassigning resources across suspend/resume. On machines where firmware pre-commits the HDM decoder, that reassignment breaks the DPA mapping.
Ah. Ok. So it's suspend / resume that is causing trouble. I wasn't aware PCI reenumerated on that as opposed to boot. I surprised more doesn't break. Make sure to call that out in the cover letter as the impression I got was the problem was initial enumeration.
Mind you I'm still confused. Why is CXL.mem path affected? That's nothing to do with PCI enumeration.
Jonathan
Quoted 129 lines
Wire
preserve_configthrough GPEXConfig intobuild_cxl_osc_method()so pxb-cxl host bridges get a_DSMmethod that signals the OS to keep resource assignments stable when needed. The_DSMfunction 5 (preserve firmware PCI configuration) is the mechanism used to convey this.
build_pci_host_bridge_dsm_method()is promoted from static to exported socxl.ccan call it without duplicating the AML.The x86
build_cxl_osc_method()call site passes false since x86 does not use firmware-committed HDM decoders.
build_cxl_osc_methodis renamed toacpi_dsdt_add_cxl_host_bridge_methodsThe function now appends both the CXL_OSCmethod and the_DSMmethod, so its old name is misleading. Renamed it to match the pxb-pcie analogueacpi_dsdt_add_host_bridge_methods(), making the two root bridge code paths symmetric. No AML change.Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Manish Honap <[email protected]>--- hw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +++- hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 5 +++-- include/hw/acpi/cxl.h | 2 +- include/hw/pci-host/gpex.h | 1 + 6 files changed, 10 insertions(+), 6 deletions(-)hw/acpi/cxl-stub.c +1
-1@@ -6,7 +6,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/cxl.h" -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config) {g_assert_not_reached (); }hw/acpi/cxl.c +3
-1@@ -23,6 +23,7 @@#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" #include "hw/cxl/cxl_host.h" +#include "hw/pci-host/gpex.h" #include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h" @@ -320,11 +321,12 @@ static Aml *__build_cxl_osc_method(void)return method; } -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config) {aml_append (dev,aml_name_decl ("SUPP" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRL" ,aml_int (0 )));aml_append (dev,aml_name_decl ("SUPC" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRC" ,aml_int (0 )));aml_append (dev,__build_cxl_osc_method ()); +aml_append (dev,build_pci_host_bridge_dsm_method (preserve_config)); }hw/i386/acpi-build.c +1
-1@@ -1013,7 +1013,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (aml_pkg,aml_eisaid ("PNP0A08" ));aml_append (aml_pkg,aml_eisaid ("PNP0A03" ));aml_append (dev,aml_name_decl ("_CID" , aml_pkg)); -build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, false); }else if (pci_bus_is_express (bus)) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));hw/pci-host/gpex-acpi.c +3
-2@@ -52,7 +52,7 @@ static void acpi_dsdt_add_pci_route_table(Aml *dev, uint32_t irq, } } -static Aml *build_pci_host_bridge_dsm_method (bool preserve_config) +Aml *build_pci_host_bridge_dsm_method (bool preserve_config) {Aml *method =aml_method ("_DSM" ,4 ,AML_NOTSERIALIZED );Aml *UUID , *ifctx, *ifctx1, *buf;@@ -204,7 +204,8 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)aml_append (dev,aml_name_decl ("_CRS" , crs));if (is_cxl) { -build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, + cfg->preserve_config); }else {/* pxb bridges do not have ACPI PCI Hot-plug enabled */ acpi_dsdt_add_host_bridge_methods(dev, true,include/hw/acpi/cxl.h +1
-1@@ -24,7 +24,7 @@void cxl_build_cedt (GArray *table_offsets,GArray *table_data,BIOSLinker *linker,const char *oem_id,const char *oem_table_id,CXLState *cxl_state); -void build_cxl_osc_method (Aml *dev); +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool preserve_config);void build_cxl_dsm_method (Aml *dev);#endif include/hw/pci-host/gpex.h +1
-0@@ -73,6 +73,7 @@ struct GPEXHost {int gpex_set_irq_num (GPEXHost *s,int index,int gsi);void acpi_dsdt_add_gpex (Aml *scope,struct GPEXConfig *cfg); +Aml *build_pci_host_bridge_dsm_method (bool preserve_config);void acpi_dsdt_add_gpex_host (Aml *scope,uint32_t irq);#define PCI_HOST_PIO_BASE "x-pio-base"
RE: [RFC 2/9] cxl: Add preserve_config to pxb-cxl OSC method
Quoted 34 lines
-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 19 May 2026 18:16
To: Manish Honap <[email protected]>
Cc: Alex Williamson <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Ankit Agrawal <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vikram Sethi <[email protected]>; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]; qemu- [email protected]
Subject: Re: [RFC 2/9] cxl: Addpreserve_configto pxb-cxl OSC methodExternal email: Use caution opening links or attachments
Quoted 7 lines · On Mon, 27 Apr 2026 23:42:28 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
Before this patch, pxb-cxl bridges had no
_DSMmethod at all. When the OS called_DSMon a CXL host bridge, ACPI returned an error and the OS defaulted to reassigning resources across suspend/resume. On machines where firmware pre-commits the HDM decoder, that reassignment breaks the DPA mapping.
Quoted 7 lines
Ah. Ok. So it's suspend / resume that is causing trouble. I wasn't aware PCI reenumerated on that as opposed to boot. I surprised more doesn't break. Make sure to call that out in the cover letter as the impression I got was the problem was initial enumeration.
Mind you I'm still confused. Why is CXL.mem path affected? That's nothing to do with PCI enumeration.
Yes, agreed, the FMWS region is described by CEDT already, no CRS entry needed. I'll drop the CRS insertion in v2.
Quoted 140 lines
Jonathan
Quoted 77 lines
Wire
preserve_configthrough GPEXConfig intobuild_cxl_osc_method()so pxb-cxl host bridges get a_DSMmethod that signals the OS to keep resource assignments stable when needed. The_DSMfunction 5 (preserve firmware PCI configuration) is the mechanism used to convey this.
build_pci_host_bridge_dsm_method()is promoted from static to exported socxl.ccan call it without duplicating the AML.The x86
build_cxl_osc_method()call site passes false since x86 does not use firmware-committed HDM decoders.
build_cxl_osc_methodis renamed toacpi_dsdt_add_cxl_host_bridge_methods
The function now appends both the CXL_OSCmethod and the_DSMmethod, so its old name is misleading. Renamed it to match the pxb-pcie analogueacpi_dsdt_add_host_bridge_methods(), making the two root bridge code paths symmetric. No AML change.Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Manish Honap <[email protected]>--- hw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +++- hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 5 +++-- include/hw/acpi/cxl.h | 2 +- include/hw/pci-host/gpex.h | 1 + 6 files changed, 10 insertions(+), 6 deletions(-)hw/acpi/cxl-stub.c index renamed from hw/acpi/cxl-stub.c +0
-0
15bc21076b..d7c6731975100644hw/acpi/cxl-stub.c +2
-1@@ -6,7 +6,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/cxl.h" -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool +preserve_config) {g_assert_not_reached (); }hw/acpi/cxl.c index renamed from hw/acpi/cxl.c +0
-0
f92f7fa3d5..b32740a3e3100644hw/acpi/cxl.c +5
-1@@ -23,6 +23,7 @@#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" #include "hw/cxl/cxl_host.h" +#include "hw/pci-host/gpex.h" #include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h" @@ -320,11 +321,12 @@ static Aml *__build_cxl_osc_method(void)return method; } -void build_cxl_osc_method (Aml *dev) +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool +preserve_config) {aml_append (dev,aml_name_decl ("SUPP" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRL" ,aml_int (0 )));aml_append (dev,aml_name_decl ("SUPC" ,aml_int (0 )));aml_append (dev,aml_name_decl ("CTRC" ,aml_int (0 )));aml_append (dev,__build_cxl_osc_method ()); +aml_append (dev, +build_pci_host_bridge_dsm_method (preserve_config)); }hw/i386/acpi-build.c index renamed from hw/i386/acpi-build.c +0
-0
f622b91b76..f66ec8ed24100644hw/i386/acpi-build.c build_dsdt(GArray *table_data, BIOSLinker *linker, · line 1018
Quoted 7 lines
hw/i386/acpi-build.c +1
-1@@ -1013,7 +1013,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (aml_pkg,aml_eisaid ("PNP0A08" ));aml_append (aml_pkg,aml_eisaid ("PNP0A03" ));aml_append (dev,aml_name_decl ("_CID" , aml_pkg)); -build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, false); }else if (pci_bus_is_express (bus)) { aml_append(dev, aml_name_decl("_HID",
aml_eisaid("PNP0A08")));hw/pci-host/gpex-acpi.c static void acpi_dsdt_add_pci_route_table(Aml *dev,
Quoted 6 lines
hw/i386/acpi-build.c +0
-0aml_append(dev, aml_name_decl("_CID",
aml_eisaid("PNP0A03"))); diff --gita/hw/pci-host/gpex-acpi.cb/hw/pci-host/gpex-acpi.cindex7de57bbc46..247bd78152100644hw/pci-host/gpex-acpi.c +0
-0@@ -52,7 +52,7 @@ static void acpi_dsdt_add_pci_route_table(Aml *dev,
uint32_tirq,Quoted 45 lines
hw/pci-host/gpex-acpi.c +1
-1} } -static Aml *build_pci_host_bridge_dsm_method (bool preserve_config) +Aml *build_pci_host_bridge_dsm_method (bool preserve_config) {Aml *method =aml_method ("_DSM" ,4 ,AML_NOTSERIALIZED ); Aml *UUID, *ifctx, *ifctx1, *buf; @@ -204,7 +204,8 @@ void
acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)aml_append(dev, aml_name_decl("_CRS", crs)); if (is_cxl) {hw/pci-host/gpex-acpi.c +3
-2-build_cxl_osc_method (dev); +acpi_dsdt_add_cxl_host_bridge_methods (dev, + + cfg->preserve_config); }else {/* pxb bridges do not have ACPI PCI Hot-plug enabled */ acpi_dsdt_add_host_bridge_methods(dev, true, diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h index
8f22c71530..6fe6c9c58d100644include/hw/acpi/cxl.h +2
-1@@ -24,7 +24,7 @@void cxl_build_cedt (GArray *table_offsets,GArray *table_data,BIOSLinker *linker,const char *oem_id,const char *oem_table_id,CXLState *cxl_state); -void build_cxl_osc_method (Aml *dev); +void acpi_dsdt_add_cxl_host_bridge_methods (Aml *dev,bool +preserve_config);void build_cxl_dsm_method (Aml *dev);#endif include/hw/pci-host/gpex.h +1
-0@@ -73,6 +73,7 @@ struct GPEXHost {int gpex_set_irq_num (GPEXHost *s,int index,int gsi);void acpi_dsdt_add_gpex (Aml *scope,struct GPEXConfig *cfg); +Aml *build_pci_host_bridge_dsm_method (bool preserve_config);void acpi_dsdt_add_gpex_host (Aml *scope,uint32_t irq);#define PCI_HOST_PIO_BASE "x-pio-base"
RFC 3/9 linux-headers: Update vfio.h for CXL Type-2 device passthrough
In reply to [email protected]From: Manish Honap <[email protected]>
Sync the VFIO UAPI additions from the kernel CXL Type-2 passthrough series.
VFIO_DEVICE_FLAGS_CXL (bit 9) marks a device as CXL Type-2 and guarantees the capability chain includes a vfio_device_info_cap_cxl entry (cap id 6). That capability carries the BAR index holding the CXL component registers, flags for firmware-committed and cache-capable devices, the byte offset to the HDM Decoder Capability block within that BAR, and region indices for both the DPA memory region and the Component Register shadow.
Two new region subtypes:
VFIO_REGION_SUBTYPE_CXL (1): mmappable DPA memory VFIO_REGION_SUBTYPE_CXL_COMP_REGS (2): HDM decoder shadow, r/w only
Note: UAPI headers are normally kept in sync via scripts/update-linux-headers.sh once upstream kernel changes merge. This patch manually adds the CXL Type-2 additions as a temporary measure to unblock QEMU development. It should be dropped and replaced with a proper header sync once the kernel series is accepted.
--- linux-headers/linux/vfio.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
linux-headers/linux/vfio.h +18 -0
@@ -215,6 +215,7 @@ struct vfio_device_info {#define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6)/* vfio-fsl-mc device */ #define VFIO_DEVICE_FLAGS_CAPS (1 << 7)/* Info supports caps */ #define VFIO_DEVICE_FLAGS_CDX (1 << 8)/* vfio-cdx device */ +#define VFIO_DEVICE_FLAGS_CXL (1 << 9)/* vfio-cxl device */ __u32 num_regions;/* Max region index + 1 */ __u32 num_irqs;/* Max IRQ index + 1 */ __u32 cap_offset;/* Offset within info struct of first cap */
@@ -257,6 +258,19 @@ struct vfio_device_info_cap_pci_atomic_comp {__u32 reserved; }; +#define VFIO_DEVICE_INFO_CAP_CXL 6 +struct vfio_device_info_cap_cxl { +struct vfio_info_cap_header header;/* id=6, version=1 */ +__u8 hdm_regs_bar_index;/* PCI BAR containing CXL component registers */ +__u8 reserved[3 ]; +__u32 flags;/* VFIO_CXL_CAP_* flags */ +#define VFIO_CXL_CAP_FIRMWARE_COMMITTED (1 << 0) +#define VFIO_CXL_CAP_CACHE_CAPABLE (1 << 1) +__u64 hdm_regs_offset;/* byte offset within BAR to CXL.mem register area */ +__u32 dpa_region_index;/* VFIO region index for DPA memory */ +__u32 comp_regs_region_index;/* VFIO region index for COMP_REGS */ +}; +/** * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, * struct vfio_region_info)
@@ -373,6 +387,10 @@ struct vfio_region_info_cap_type {/* sub-types for VFIO_REGION_TYPE_GFX */ #define VFIO_REGION_SUBTYPE_GFX_EDID (1) +/* sub-types for VFIO CXL regions */ +#define VFIO_REGION_SUBTYPE_CXL (1)/* DPA memory region */ +#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS (2)/* HDM register shadow */ +/** * struct vfio_region_gfx_edid - EDID region layout. *
Signature · 1 line
2.25.1
RFC 4/9 hw/vfio/region: Add vfio_region_setup_with_ops() for custom region ops
In reply to [email protected]From: Manish Honap <[email protected]>
vfio_region_setup() always initializes the region MemoryRegion with vfio_region_ops. CXL needs custom pread/pwrite ops for the Component Register shadow region.
Add vfio_region_setup_with_ops() which accepts a const MemoryRegionOps * parameter. When non-NULL it is passed to memory_region_init_io(); when NULL the existing vfio_region_ops is used. vfio_region_setup() is retained unchanged as a thin wrapper for all existing callers.
--- hw/vfio/region.c | 15 ++++++++++++--- hw/vfio/vfio-region.h | 3 +++ 2 files changed, 15 insertions(+), 3 deletions(-)
hw/vfio/region.c +12 -3
@@ -228,8 +228,9 @@ static int vfio_setup_region_sparse_mmaps(VFIORegion *region,return 0 ; } -int vfio_region_setup (Object *obj,VFIODevice *vbasedev,VFIORegion *region, -int index,const char *name,Error **errp) +int vfio_region_setup_with_ops (Object *obj,VFIODevice *vbasedev, +VFIORegion *region,int index,const char *name, +Error **errp,const MemoryRegionOps *ops) {struct vfio_region_info *info =NULL ;int ret;
@@ -249,7 +250,8 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,if (region->size) { region->mem =g_new0 (MemoryRegion,1 ); -memory_region_init_io (region->mem, obj, &vfio_region_ops, +memory_region_init_io (region->mem, obj, + ops ? ops : &vfio_region_ops, region, name, region->size); if (!vbasedev->no_mmap &&
@@ -273,6 +275,13 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,return 0 ; } +int vfio_region_setup (Object *obj,VFIODevice *vbasedev,VFIORegion *region, +int index,const char *name,Error **errp) +{ +return vfio_region_setup_with_ops (obj, vbasedev, region, index, + name, errp,NULL ); +} +static void vfio_subregion_unmap (VFIORegion *region,int index) { trace_vfio_region_unmap(memory_region_name(®ion->mmaps[index].mem),
hw/vfio/vfio-region.h +3 -0
@@ -39,6 +39,9 @@ uint64_t vfio_region_read(void *opaque, hwaddr addr, unsigned size);int vfio_region_setup (Object *obj,VFIODevice *vbasedev,VFIORegion *region,int index,const char *name,Error **errp); +int vfio_region_setup_with_ops (Object *obj,VFIODevice *vbasedev, +VFIORegion *region,int index,const char *name, +Error **errp,const MemoryRegionOps *ops);int vfio_region_mmap (VFIORegion *region);void vfio_region_mmaps_set_enabled (VFIORegion *region,bool enabled);void vfio_region_unmap (VFIORegion *region);
Signature · 1 line
2.25.1
RFC 5/9 hw/vfio/pci: Add CXL Type-2 device detection and region setup
In reply to [email protected]From: Manish Honap <[email protected]>
When VFIO_DEVICE_FLAGS_CXL is set, the kernel has identified a CXL Type-2 device and populated the capability chain with a vfio_device_info_cap_cxl entry. Read that entry to locate the DPA and CXL Component Register shadow regions, then call vfio_region_setup() for each.
DPA covers the device's host-managed memory and is faulted in lazily by the VMM. The CXL Component Register shadow gives the VMM access to the HDM Decoder Capability block so it can intercept decoder commits without touching the hardware register page directly.
vfio_cxl_derive_hdm_info() walks the CXL Capability Array inside the Component Register shadow to find the HDM Decoder capability (ID 0x5) and extracts hdm_decoder_offset and hdm_count. All reads use le32_to_cpu() since the capability array is little-endian per the CXL spec. Dword 0 is the array header; capability entries start at dword 1, which is why the loop begins at i = 1.
CXL register constants are defined here using names that mirror <linux/cxl.h> to make cross-referencing straightforward.
Add the VFIOCXL struct embedded in VFIOPCIDevice.
--- hw/vfio/pci.c | 214 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 14 +++ hw/vfio/trace-events | 4 + 3 files changed, 232 insertions(+)
hw/vfio/pci.c +214 -0
@@ -24,6 +24,7 @@#include "hw/core/hw-error.h" #include "hw/core/iommu.h" +#include "hw/cxl/cxl_component.h" #include "hw/pci/msi.h" #include "hw/pci/msix.h" #include "hw/pci/pci_bridge.h"
@@ -2957,6 +2958,38 @@ static VFIODeviceOps vfio_pci_ops = { .vfio_load_config = vfio_pci_load_config, }; +/* +* CXL Component Register Space constants (CXL 4.0 8.2.3). +*/ + +/* CXL Capability Array Header (dword 0 of COMP_REGS) */ +#define CXL_CM_CAP_HDR_ARRAY_ID 0x0001U/* expected ID value */ +#define CXL_CM_CAP_HDR_NUM_CAPS_SHIFT 24/* bits [31:24] = num entries */ +#define CXL_CM_CAP_HDR_NUM_CAPS_MASK 0xffU +#define CXL_CM_CAP_ENTRY_ID_MASK 0xffffU/* bits [15:0] = cap ID */ +#define CXL_CM_CAP_ENTRY_PTR_SHIFT 20/* bits [31:20] = byte offset */ +#define CXL_CM_CAP_ENTRY_PTR_MASK 0xfffU +#define CXL_CM_CAP_ID_HDM 0x0005U/* HDM Decoder cap ID */ + +/* HDM Decoder Capability (HDMC) register at hdm_decoder_offset+0x00 */ +#define CXL_HDMC_DECODER_COUNT_MASK 0xfU/* bits [3:0]; 0→1, N→N*2 */ + +/* +* Per-decoder register offsets from hdm_decoder_offset (CXL 4.0 Table 8-119). +* Decoder records begin at +0x10 and are 0x20 bytes each. +*/ +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET (i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET (i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_CTRL_OFFSET (i) (0x20 * (i) + 0x20) + +/* HDM Decoder n Control register bits (CXL 4.0 Table 8-123) */ +#define CXL_HDM_CTRL_COMMIT_LOCK (1U << 8)/* decoder locked */ +#define CXL_HDM_CTRL_COMMIT (1U << 9)/* software trigger */ +#define CXL_HDM_CTRL_COMMITTED (1U << 10)/* hardware status */ + +/* HDM Decoder BASE_LO: bits [31:28] hold address bits [31:28] */ +#define CXL_HDM_BASE_LO_ADDR_MASK 0xF0000000U +bool vfio_populate_vga (VFIOPCIDevice *vdev,Error **errp) {VFIODevice *vbasedev = &vdev->vbasedev;
@@ -3102,6 +3135,25 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) {vfio_display_finalize (vdev);vfio_bars_finalize (vdev); + +/* +* The DPA region is not in bars[] and must be cleaned up here. +* Remove it from the system address space before releasing. +*/ +if (vdev->cxl.dpa_in_system_mem) { +memory_region_del_subregion (get_system_memory (), vdev->cxl.region.mem); + vdev->cxl.dpa_in_system_mem = false; +trace_vfio_cxl_put_device (vdev->vbasedev.name); + } +if (vdev->cxl.region.mem) { +vfio_region_exit (&vdev->cxl.region); +vfio_region_finalize (&vdev->cxl.region); + } +if (vdev->cxl.comp_regs_region.mem) { +vfio_region_exit (&vdev->cxl.comp_regs_region); +vfio_region_finalize (&vdev->cxl.comp_regs_region); + } +vfio_cpr_pci_unregister_device (vdev);g_free (vdev->emulated_config_bits);g_free (vdev->rom);
@@ -3254,6 +3306,164 @@ void vfio_pci_register_req_notifier(VFIOPCIDevice *vdev) } } +/* +* vfio_cxl_derive_hdm_info - read hdm_decoder_offset and hdm_count from the +* COMP_REGS region by traversing the CXL Capability Array. +* +* Dword 0: CXL Capability Array Header +* bits[31:24] = num_caps, +* bits[15:0] = 1. +* Dwords 1..N: +* bits[15:0] = cap ID; +* bits[31:20] = byte offset from region start. +* HDM Decoder cap ID = 0x5; its offset is hdm_decoder_offset. +* HDMC register at hdm_decoder_offset+0: +* bits[3:0] encode count (0→1, N→N*2). +*/ +static bool vfio_cxl_derive_hdm_info (VFIODevice *vbasedev,VFIOCXL *cxl, +Error **errp) +{ +off_t base = cxl->comp_regs_region.fd_offset; +uint32_t hdr, num_caps, i; + +if (pread (vbasedev->fd, &hdr,sizeof (hdr), base) !=sizeof (hdr)) { +error_setg (errp,"vfio-cxl: failed to read CXL Capability Header" ); +return false; + } + hdr =le32_to_cpu (hdr); + +if ((hdr &CXL_CM_CAP_ENTRY_ID_MASK ) !=CXL_CM_CAP_HDR_ARRAY_ID ) { +error_setg (errp,"vfio-cxl: unexpected CXL Capability Array ID 0x%x" , + hdr &CXL_CM_CAP_ENTRY_ID_MASK ); +return false; + } + + num_caps = (hdr >>CXL_CM_CAP_HDR_NUM_CAPS_SHIFT ) & +CXL_CM_CAP_HDR_NUM_CAPS_MASK ; + +/* +* Dword 0 is the CXL Capability Array Header; +* capability entries start at dword 1. +*/ +for (i =1 ; i <= num_caps; i++) { +uint32_t entry, cap_id; + +if (pread (vbasedev->fd, &entry,sizeof (entry), + base + i *sizeof (entry)) !=sizeof (entry)) { +error_setg (errp,"vfio-cxl: failed to read cap entry %u" , i); +return false; + } + entry =le32_to_cpu (entry); + + cap_id = entry &CXL_CM_CAP_ENTRY_ID_MASK ; +if (cap_id ==CXL_CM_CAP_ID_HDM ) { +uint32_t hdmc, field; + + cxl->hdm_decoder_offset = (entry >>CXL_CM_CAP_ENTRY_PTR_SHIFT ) & +CXL_CM_CAP_ENTRY_PTR_MASK ; + +if (pread (vbasedev->fd, &hdmc,sizeof (hdmc), + base + cxl->hdm_decoder_offset) !=sizeof (hdmc)) { +error_setg (errp,"vfio-cxl: failed to read HDMC register" ); +return false; + } + hdmc =le32_to_cpu (hdmc); + field = hdmc &CXL_HDMC_DECODER_COUNT_MASK ; + cxl->hdm_count = field ? (uint8_t )(field *2 ) :1 ; +return true; + } + } + +error_setg (errp,"vfio-cxl: HDM Decoder capability not found in COMP_REGS" ); +return false; +} + +static bool vfio_cxl_setup (VFIOPCIDevice *vdev,Error **errp) +{ +VFIODevice *vbasedev = &vdev->vbasedev; +VFIOCXL *cxl = &vdev->cxl; + g_autofree struct vfio_device_info *info = NULL; +struct vfio_info_cap_header *hdr; +struct vfio_device_info_cap_cxl *cap; + g_autofree struct vfio_region_info *region_info = NULL; + g_autofree struct vfio_region_info *comp_info = NULL; +int ret; + +if (!(vbasedev->flags &VFIO_DEVICE_FLAGS_CXL )) { +return true; + } + + info =vfio_get_device_info (vbasedev->fd); +if (!info) { +error_setg (errp,"vfio-cxl: failed to get device info" ); +return false; + } + + hdr =vfio_get_device_info_cap (info,VFIO_DEVICE_INFO_CAP_CXL ); +if (!hdr) { +error_setg (errp,"vfio-cxl: CXL capability not found in device info" ); +return false; + } + cap = (void *)hdr; + +if (cap->dpa_region_index == (uint32_t )-1 || + cap->comp_regs_region_index == (uint32_t )-1 ) { +error_setg (errp,"vfio-cxl: kernel did not provide region indices " +"(dpa=%u comp=%u)" , + cap->dpa_region_index, cap->comp_regs_region_index); +return false; + } + + cxl->hdm_regs_bar_index = cap->hdm_regs_bar_index; + cxl->hdm_regs_offset = cap->hdm_regs_offset; + +/* DPA region */ + ret =vfio_device_get_region_info (vbasedev, cap->dpa_region_index, + ®ion_info); +if (ret || !region_info) { +error_setg (errp,"vfio-cxl: failed to get DPA region info" ); +return false; + } + ret =vfio_region_setup (OBJECT (vdev), vbasedev, &cxl->region, + region_info->index,"cxl-dpa" , errp); +if (ret) { +error_setg (errp,"vfio-cxl: failed to set up DPA region" ); +return false; + } + cxl->dpa_size = region_info->size; + +if (vfio_region_mmap (&cxl->region)) { +error_setg (errp,"vfio-cxl: failed to mmap DPA region for %s" , + vbasedev->name); +return false; + } + +/* COMP_REGS region (HDM decoder shadow) */ + ret =vfio_device_get_region_info (vbasedev, cap->comp_regs_region_index, + &comp_info); +if (ret || !comp_info) { +error_setg (errp,"vfio-cxl: failed to get COMP_REGS region info" ); +return false; + } + ret =vfio_region_setup (OBJECT (vdev), vbasedev, &cxl->comp_regs_region, + comp_info->index,"cxl-comp-regs" , errp); +if (ret) { +error_setg (errp,"vfio-cxl: failed to set up COMP_REGS region" ); +return false; + } + cxl->hdm_regs_size = comp_info->size; + +if (!vfio_cxl_derive_hdm_info (vbasedev, cxl, errp)) { +return false; + } + +trace_vfio_cxl_setup_params (vbasedev->name, cxl->hdm_regs_bar_index, + cxl->hdm_regs_offset, cxl->hdm_regs_size, + cxl->dpa_size); +return true; +} + +static void vfio_unregister_req_notifier (VFIOPCIDevice *vdev) {Error *err =NULL ;
@@ -3508,6 +3718,10 @@ static void vfio_pci_realize(PCIDevice *pdev, Error **errp) goto error; } +if (!vfio_cxl_setup (vdev, errp)) { + goto error; + } +if (!vfio_pci_config_setup (vdev, errp)) { goto error; }
hw/vfio/pci.h +14 -0
@@ -122,6 +122,19 @@ typedef struct VFIOMSIXInfo { OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_DEVICE) +typedef struct VFIOCXL { +uint8_t hdm_regs_bar_index; +uint64_t hdm_regs_offset; +uint64_t hdm_regs_size; +uint64_t hdm_decoder_offset; +uint8_t hdm_count; +uint64_t dpa_size; +hwaddr fmws_base;/* GPA base programmed into HDM decoder 0 */ +bool dpa_in_system_mem; +VFIORegion region; +VFIORegion comp_regs_region; +}VFIOCXL ; +struct VFIOPCIDevice {PCIDevice parent_obj;
@@ -191,6 +204,7 @@ struct VFIOPCIDevice {VFIODisplay *dpy;Notifier irqchip_change_notifier;VFIOPCICPR cpr; +VFIOCXL cxl; };/* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
hw/vfio/trace-events +4 -0
@@ -197,3 +197,7 @@ vfio_device_get_region_info_type(const char *name, int index, uint32_t type, uin vfio_device_reset_handler(void) "" vfio_device_attach(const char *name, int group_id) " (%s) group %d" vfio_device_detach(const char *name, int group_id) " (%s) group %d" + +# pci.c CXL Type-2 passthrough +vfio_cxl_setup_params(const char *name, uint8_t bar, uint64_t hdm_off, uint64_t hdm_sz, uint64_t dpa_sz) " (%s) hdm_bar=%u hdm_regs_offset=0x%"PRIx64" hdm_regs_size=0x%"PRIx64" dpa_size=0x%"PRIx64 +vfio_cxl_put_device(const char *name) " (%s) removing DPA region from system memory"
Signature · 1 line
2.25.1
RFC 6/9 hw/vfio/pci: Wire CXL component-register BAR with COMP_REGS overlay
In reply to [email protected]From: Manish Honap <[email protected]>
The CXL Component Register BAR contains two types of ranges that need different handling:
- Accelerator register windows: passed through as direct hardware mmaps for performance. The kernel reports the real BAR size and lists mmappable windows via VFIO_REGION_INFO_CAP_SPARSE_MMAP, excluding the HDM Decoder Capability block. vfio_region_mmap() creates hardware-backed sub-regions for each sparse area.
- HDM Decoder Capability block: guest accesses must go through emulated ops so QEMU can observe and program decoder state. The kernel blocks direct mmap of this range.
vfio_bar_register(): after the normal mmap path, overlay the COMP_REGS emulation region at hdm_regs_offset with priority 1. In QEMU's MemoryRegion model, overlapping subregions are resolved by priority; the default is 0. Priority 1 ensures guest accesses to the HDM range always dispatch through the emulated COMP_REGS ops regardless of any hardware-backed sub-region at a neighbouring offset.
vfio_pci_bars_exit(): remove the COMP_REGS overlay before the normal BAR teardown path.
--- hw/vfio/pci.c | 26 ++++++++++++++++++++++++++ hw/vfio/trace-events | 1 + 2 files changed, 27 insertions(+)
hw/vfio/pci.c +26 -0
@@ -1960,6 +1960,10 @@ static void vfio_bar_register(VFIOPCIDevice *vdev, int nr)return ; } +bool cxl_comp_regs_bar = (vdev->vbasedev.flags &VFIO_DEVICE_FLAGS_CXL ) && + nr == vdev->cxl.hdm_regs_bar_index && + vdev->cxl.comp_regs_region.mem; + bar->mr =g_new0 (MemoryRegion,1 ); name =g_strdup_printf ("%s base BAR %d" , vdev->vbasedev.name, nr);memory_region_init_io (bar->mr,OBJECT (vdev),NULL ,NULL , name, bar->size);
@@ -1974,6 +1978,21 @@ static void vfio_bar_register(VFIOPCIDevice *vdev, int nr) } } +if (cxl_comp_regs_bar) { +/* +* Overlay the COMP_REGS emulation at hdm_regs_offset with priority 1. +* The kernel excludes the HDM Decoder Capability block from the +* sparse-mmap list, so vfio_region_mmap() creates hardware-backed +* sub-regions only for accelerator register windows. The emulated +* COMP_REGS region sits above those at priority 1, ensuring guest +* accesses to the HDM range always dispatch through the emulated ops. +*/ +memory_region_add_subregion_overlap (bar->mr, vdev->cxl.hdm_regs_offset, + vdev->cxl.comp_regs_region.mem,1 ); +trace_vfio_cxl_bar_subregion (vdev->vbasedev.name, nr, + vdev->cxl.hdm_regs_offset); + } +pci_register_bar (pdev, nr, bar->type, bar->mr); }
@@ -1993,9 +2012,16 @@ void vfio_pci_bars_exit(VFIOPCIDevice *vdev)for (i =0 ; i <PCI_ROM_SLOT ; i++) {VFIOBAR *bar = &vdev->bars[i]; +bool use_comp_regs = (vdev->vbasedev.flags &VFIO_DEVICE_FLAGS_CXL ) && + i == vdev->cxl.hdm_regs_bar_index && + vdev->cxl.comp_regs_region.mem;vfio_bar_quirk_exit (vdev, i);vfio_region_exit (&bar->region); +if (use_comp_regs && bar->mr) { +memory_region_del_subregion (bar->mr, + vdev->cxl.comp_regs_region.mem); + }if (bar->region.size) {memory_region_del_subregion (bar->mr, bar->region.mem); }
hw/vfio/trace-events +1 -0
@@ -201,3 +201,4 @@ vfio_device_detach(const char *name, int group_id) " (%s) group %d" # pci.c CXL Type-2 passthrough vfio_cxl_setup_params(const char *name, uint8_t bar, uint64_t hdm_off, uint64_t hdm_sz, uint64_t dpa_sz) " (%s) hdm_bar=%u hdm_regs_offset=0x%"PRIx64" hdm_regs_size=0x%"PRIx64" dpa_size=0x%"PRIx64 vfio_cxl_put_device(const char *name) " (%s) removing DPA region from system memory" +vfio_cxl_bar_subregion(const char *name, int nr, uint64_t off) " (%s) BAR%d comp_regs overlay at BAR offset 0x%"PRIx64
Signature · 1 line
2.25.1
RFC 7/9 hw/vfio+cxl: Program HDM decoder 0 at machine_done for firmware-committed devices
In reply to [email protected]From: Manish Honap <[email protected]>
setup_locked_hdm() runs as a machine_done notifier after all devices have been realized. It programs HDM decoder 0 with the CFMWS base address so the guest can fault into device memory from the first instruction.
The notifier is only registered when the kernel reports the device as firmware-committed (VFIO_CXL_CAP_FIRMWARE_COMMITTED). The host is responsible for HDM decoder programming; the guest has no mechanism to remap host physical address mappings.
The function uses cxl->fmws_base (set by the optional cxl-fmws-base device property) if non-zero; otherwise it falls back to the cxl_fmws_base global captured by cxl_fmws_set_memmap() during machine memory-map init. If neither is set, it warns and returns without programming anything.
If COMMIT_LOCK is set in decoder 0 CTRL at machine_done time (left-over from a prior FLR?), it is cleared before writing BASE so the subsequent write is not blocked. COMMIT_LOCK is re-set after programming so the hardware enforces the committed base.
read_region() return is checked; failure aborts programming rather than leaving ctrl uninitialized. All write_region() failures are propagated. The function exits cleanly rather than leaving the decoder half-programmed.
Add cxl_fmws_base as a hwaddr global in cxl-host.c (and a stub in cxl-host-stubs.c). It is set once by cxl_fmws_set_memmap() and read later at machine_done time.
--- hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 8 ++ hw/vfio/pci.c | 176 +++++++++++++++++++++++++++++++++++++- hw/vfio/pci.h | 1 + hw/vfio/trace-events | 1 + include/hw/cxl/cxl_host.h | 10 +++ 6 files changed, 196 insertions(+), 2 deletions(-)
hw/cxl/cxl-host-stubs.c +2 -0
@@ -17,4 +17,6 @@ hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr) };void cxl_fmws_update_mmio (void ) {}; +hwaddr cxl_fmws_base; +const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c +8 -0
@@ -429,11 +429,19 @@ void cxl_fmws_update_mmio(void)object_child_foreach_recursive (object_get_root (), cxl_fmws_mmio_map,NULL ); } +/* +* GPA base of the first CXL Fixed Memory Window region placed in the memory +* map by cxl_fmws_set_memmap(). Set once at machine memory-map init time. +*/ +hwaddr cxl_fmws_base; +hwaddr cxl_fmws_set_memmap (hwaddr base,hwaddr max_addr) {GSList *cfmws_list, *iter;CXLFixedWindow *fw; + cxl_fmws_base = base; + cfmws_list =cxl_fmws_get_all_sorted ();for (iter = cfmws_list; iter; iter = iter->next) { fw =CXL_FMW (iter->data);
hw/vfio/pci.c +174 -2
206 lines not on this page · raw message
hw/vfio/pci.h +1 -0
@@ -133,6 +133,7 @@ typedef struct VFIOCXL {bool dpa_in_system_mem;VFIORegion region;VFIORegion comp_regs_region; +Notifier machine_done; }VFIOCXL ;struct VFIOPCIDevice {
hw/vfio/trace-events +1 -0
@@ -202,3 +202,4 @@ vfio_device_detach(const char *name, int group_id) " (%s) group %d" vfio_cxl_setup_params(const char *name, uint8_t bar, uint64_t hdm_off, uint64_t hdm_sz, uint64_t dpa_sz) " (%s) hdm_bar=%u hdm_regs_offset=0x%"PRIx64" hdm_regs_size=0x%"PRIx64" dpa_size=0x%"PRIx64 vfio_cxl_put_device(const char *name) " (%s) removing DPA region from system memory" vfio_cxl_bar_subregion(const char *name, int nr, uint64_t off) " (%s) BAR%d comp_regs overlay at BAR offset 0x%"PRIx64 +vfio_cxl_locked_hdm(const char *name, uint64_t fmws, uint32_t blo, uint32_t bhi, uint32_t ctrl) " (%s) fmws_base=0x%"PRIx64" wrote decoder0 base_lo=0x%08x base_hi=0x%08x ctrl=0x%08x"
include/hw/cxl/cxl_host.h +10 -0
@@ -20,6 +20,16 @@ hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr);void cxl_fmws_update_mmio (void );GSList *cxl_fmws_get_all_sorted (void ); +/** +* cxl_fmws_base - GPA base of the first CXL Fixed Memory Window region. +* +* Set by cxl_fmws_set_memmap() to the base address it receives (typically +* ROUND_UP(highest_gpa + 1, 256 MiB) on ARM virt). Valid after the +* machine memory-map init callback returns, i.e. at machine_done time. +* Zero when no machine has called cxl_fmws_set_memmap() (stub builds). +*/ +extern hwaddr cxl_fmws_base; +extern const MemoryRegionOps cfmws_ops;#endif
Signature · 1 line
2.25.1
Re: [RFC 7/9] hw/vfio+cxl: Program HDM decoder 0 at machine_done for firmware-committed devices
Quoted 4 lines · On Mon, 27 Apr 2026 23:42:33 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
setup_locked_hdm()runs as amachine_donenotifier after all devices have been realized. It programs HDM decoder 0 with the CFMWS base
Be more specific on this. Which HDM Decoder? i.e. what device is it in. Ultimately there may be several in the path. Are you supporting that? The reference to FLR() later would only apply to an endpoint (can't FLR a host bridge) + the code is clearly operating on the shadow registers for the EP.
I guess maybe you are relying on pass through host bridge decoders? If you are - you need to go check those - or better yet program and lock them as well.
Quoted 7 lines
address so the guest can fault into device memory from the first instruction.
The notifier is only registered when the kernel reports the device as firmware-committed (
VFIO_CXL_CAP_FIRMWARE_COMMITTED). The host is responsible for HDM decoder programming; the guest has no mechanism to remap host physical address mappings.
Maybe say why this matters. Presumably because only thing we can program from the guest is GPA to HPA mappings so linear ranges remain linear etc (more or less anyway given constraints on how many regions we can plug into qemu!)
Quoted 3 lines
The function uses
cxl->fmws_base(set by the optional cxl-fmws-base device property) if non-zero; otherwise it falls back to the
Why is that property needed? If it is add it in a separate patch where you can explain the need to force it. Smells like a hack so far.
Quoted 3 lines
cxl_fmws_baseglobal captured bycxl_fmws_set_memmap()during machine memory-map init. If neither is set, it warns and returns without programming anything.
Which CFMWS is that base in? How is that controlled? We may well have some type 3 devices in same VM and probably want those in different CFMWs (we'll need to actually upstream the patch to control constraints so the kernel won't try using your CFMWS for those - currently all CFMWS are entirely flexible).
Quoted 5 lines
If
COMMIT_LOCKis set in decoder 0 CTRL atmachine_donetime (left-over from a prior FLR?), it is cleared before writing BASE so the subsequent write is not blocked.COMMIT_LOCKis re-set after programming so the hardware enforces the committed base.
Hmm. We should look into whether there is a correct way to reset that. Though as above from the description I'm not sure which decoder this is resetting.
Quoted 14 lines
read_region()return is checked; failure aborts programming rather than leaving ctrl uninitialized. Allwrite_region()failures are propagated. The function exits cleanly rather than leaving the decoder half-programmed.Add
cxl_fmws_baseas a hwaddr global incxl-host.c(and a stub incxl-host-stubs.c). It is set once bycxl_fmws_set_memmap()and read later atmachine_donetime.Signed-off-by Zhi Wang <[email protected]>Signed-off-by Manish Honap <[email protected]>
hw/vfio/pci.c static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp) · line 3652
Quoted 7 lines
hw/vfio/pci.c +4
-0@@ -3486,6 +3646,18 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)trace_vfio_cxl_setup_params (vbasedev->name, cxl->hdm_regs_bar_index, cxl->hdm_regs_offset, cxl->hdm_regs_size, cxl->dpa_size); + +/* +* Only pre-program the HDM decoder if the kernel reported the device as +* firmware-committed. Non-committed devices need guest driver involvement
For wording these can we be very clear which kernel. Though I'm also unsure what FIRMWARE_COMMITTED means here. Does it actually mean host firmware, or does it mean something running on the host pre guest boot (e.g. could it be qemu?)
Quoted 11 lines
hw/vfio/pci.c +8
-0+* to commit the decoder; registering the notifier for them would write an +* uncommitted BASE value that the hardware ignores. +*/ +if (cap->flags &VFIO_CXL_CAP_FIRMWARE_COMMITTED ) { + cxl->machine_done.notify = setup_locked_hdm; +qemu_add_machine_init_done_notifier (&cxl->machine_done); + } +return true; }
RE: [RFC 7/9] hw/vfio+cxl: Program HDM decoder 0 at machine_done for firmware-committed devices
Quoted 38 lines
-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 19 May 2026 18:34
To: Manish Honap <[email protected]>
Cc: Alex Williamson <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Ankit Agrawal <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vikram Sethi <[email protected]>; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]; qemu- [email protected]
Subject: Re: [RFC 7/9] hw/vfio+cxl: Program HDM decoder 0 atmachine_donefor firmware-committed devicesExternal email: Use caution opening links or attachments
Quoted 4 lines · On Mon, 27 Apr 2026 23:42:33 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
setup_locked_hdm()runs as amachine_donenotifier after all devices have been realized. It programs HDM decoder 0 with the CFMWS baseBe more specific on this. Which HDM Decoder? i.e. what device is it in. Ultimately there may be several in the path. Are you supporting that? The reference to
FLR()later would only apply to an endpoint (can't FLR a host bridge) + the code is clearly operating on the shadow registers for the EP.
Quoted 3 lines
I guess maybe you are relying on pass through host bridge decoders? If you are - you need to go check those - or better yet program and lock them as well.
Yes, for current RFC I had restricted the scope to endpoint decoders with the setup having default value for hdm_for_passthrough. You are correct, this assumption was not enforced in the code.
For next V2, I will add realize-time checks in vfio_cxl_setup() that refuse to start unless:
- cxl_get_hb_passthrough(hb) is true
- pcie_count_ds_ports(hb->bus) == 1
- no switch is present between the cxl-rp and the vfio-pci
Quoted 9 lines
Quoted 8 lines
address so the guest can fault into device memory from the first instruction.
The notifier is only registered when the kernel reports the device as firmware-committed (
VFIO_CXL_CAP_FIRMWARE_COMMITTED). The host is responsible for HDM decoder programming; the guest has no mechanism to remap host physical address mappings.
Quoted 4 lines
Maybe say why this matters. Presumably because only thing we can program from the guest is GPA to HPA mappings so linear ranges remain linear etc (more or less anyway given constraints on how many regions we can plug into qemu!)
okay, I will add the details for this in v2.
Quoted 7 lines
Quoted 4 lines
The function uses
cxl->fmws_base(set by the optional cxl-fmws-base device property) if non-zero; otherwise it falls back to theWhy is that property needed? If it is add it in a separate patch where you can explain the need to force it. Smells like a hack so far.
Yes, I will update in v2 so that this part is handled by the value provided by CFMWS setting.
Quoted 13 lines
Quoted 4 lines
cxl_fmws_baseglobal captured bycxl_fmws_set_memmap()during machine memory-map init. If neither is set, it warns and returns without programming anything.Which CFMWS is that base in? How is that controlled? We may well have some type 3 devices in same VM and probably want those in different CFMWs (we'll need to actually upstream the patch to control constraints so the kernel won't try using your CFMWS for those - currently all CFMWS are entirely flexible).
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
If
COMMIT_LOCKis set in decoder 0 CTRL atmachine_donetime (left-over from a prior FLR?), it is cleared before writing BASE so the subsequent write is not blocked.COMMIT_LOCKis re-set after programming so the hardware enforces the committed base.Hmm. We should look into whether there is a correct way to reset that. Though as above from the description I'm not sure which decoder this is resetting.
Yes, I will rectify this in V2. This is referring to endpoint decoder.
Quoted 26 lines
Quoted 4 lines
read_region()return is checked; failure aborts programming rather than leaving ctrl uninitialized. Allwrite_region()failures arepropagated.
Quoted 1 line
The function exits cleanly rather than leaving the decoder half-
programmed.
Review static bool vfio_cxl_setup(VFIOPCIDevice *vdev,
Quoted 10 lines
Add
cxl_fmws_baseas a hwaddr global incxl-host.c(and a stub incxl-host-stubs.c). It is set once bycxl_fmws_set_memmap()and read later atmachine_donetime.Signed-off-by Zhi Wang <[email protected]>Signed-off-by Manish Honap <[email protected]>(quoted diff) +0
-0@@ -3486,6 +3646,18 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev,Error **errp)
Review static bool vfio_cxl_setup(VFIOPCIDevice *vdev, · line 3653
Quoted 8 lines
(quoted diff) +3
-0trace_vfio_cxl_setup_params(vbasedev->name, cxl- hdm_regs_bar_index, cxl->hdm_regs_offset, cxl- hdm_regs_size, cxl->dpa_size); + + /* + * Only pre-program the HDM decoder if the kernel reported the
Quoted 8 lines
device as
Review static bool vfio_cxl_setup(VFIOPCIDevice *vdev, · line 3655
Quoted 2 lines
(quoted diff) +2
-0+ * firmware-committed. Non-committed devices need guest driver + involvementFor wording these can we be very clear which kernel. Though I'm also unsure what
FIRMWARE_COMMITTEDmeans here. Does it actually mean host firmware, or does it mean something running on the host pre guest boot (e.g. could it be qemu?)
yes, this is referring to HOST_FIRMWARE. Sorry for the bad wording. I will update this flag name in the kernel-v3 and qemu-v2
Quoted 14 lines
Review static bool vfio_cxl_setup(VFIOPCIDevice *vdev, · line 3656
Quoted 2 lines
(quoted diff) +1
-0+ * to commit the decoder; registering the notifier for them wouldwrite an
Quoted 10 lines
hw/vfio/pci.c +7
-0+* uncommitted BASE value that the hardware ignores. +*/ +if (cap->flags &VFIO_CXL_CAP_FIRMWARE_COMMITTED ) { + cxl->machine_done.notify = setup_locked_hdm; +qemu_add_machine_init_done_notifier (&cxl->machine_done); + } +return true; }
RFC 8/9 hw/arm/smmu-common: Allow pxb-cxl as SMMUv3 primary bus
In reply to [email protected]From: Manish Honap <[email protected]>
The SMMUv3 primary bus check only accepted pxb-pcie as a valid root. pxb-cxl uses the same PCIe-compatible bus implementation; reject it and CXL devices behind it cannot reach the IOMMU.
Extend the check to also accept CXL buses so SMMUv3 translation applies to passthrough CXL devices. Update the comment above the check to mention pxb-cxl alongside pxb-pcie.
--- hw/arm/smmu-common.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
hw/arm/smmu-common.c +8 -9
@@ -963,19 +963,18 @@ static void smmu_base_realize(DeviceState *dev, Error **errp) s->iommu_ops = &smmu_ops; }/* -* We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based extra -* root complexes to be associated with SMMU. +* We only allow the default PCIe root complex (pcie.0) or pxb-pcie / +* pxb-cxl based extra root complexes to be associated with SMMU. */ if (pci_bus_is_express (pci_bus) &&pci_bus_is_root (pci_bus) &&object_dynamic_cast (OBJECT (pci_bus)->parent,TYPE_PCI_HOST_BRIDGE )) {/* -* This condition matches either the default pcie.0, pxb-pcie, or -* pxb-cxl. For both pxb-pcie and pxb-cxl, parent_dev will be set. -* Currently, we don't allow pxb-cxl as it requires further -* verification. Therefore, make sure this is indeed pxb-pcie. +* pcie.0 has no parent_dev; pxb-pcie and pxb-cxl do. Accept both +* bus types explicitly so other root complexes are still rejected. */ if (pci_bus->parent_dev) { -if (!object_dynamic_cast (OBJECT (pci_bus),TYPE_PXB_PCIE_BUS )) { +if (!object_dynamic_cast (OBJECT (pci_bus),TYPE_PXB_PCIE_BUS ) && + !object_dynamic_cast (OBJECT (pci_bus),TYPE_PXB_CXL_BUS )) { goto out_err; } }
@@ -988,8 +987,8 @@ static void smmu_base_realize(DeviceState *dev, Error **errp)return ; } out_err: -error_setg (errp,"SMMU should be attached to a default PCIe root complex" -"(pcie.0) or a pxb-pcie based root complex" ); +error_setg (errp,"SMMU should be attached to a default PCIe root complex " +"(pcie.0), a pxb-pcie, or a pxb-cxl based root complex" ); }/*
Signature · 1 line
2.25.1
RFC 9/9 vfio/listener: Skip DMA mapping for VFIO-owned RAM-device regions
In reply to [email protected]From: Manish Honap <[email protected]>
vfio_container_region_add() attempts an IOMMU DMA mapping for every RAM section that enters the guest address space. For VFIO mmap-backed regions (PCI BAR windows, CXL.mem regions), this mapping always fails:
the backing VMAs carry VM_IO | VM_PFNMAP flags and pin_user_pages() refuses to pin VM_IO pages, so IOMMU_IOAS_MAP returns -EFAULT.
CPU access to these regions goes through KVM Stage-2 page faults independently of the SMMU/IOMMU, so no IOMMU entry is required for correct operation.
Add an early return for RAM-device sections owned by a VFIO device. vfio_get_vfio_device(memory_region_owner(section->mr)) returns non-NULL for any mmap subregion created by vfio_region_mmap(), since memory_region_init_ram_device_ptr() propagates the VFIOPCIDevice owner from the containing region. Matching on ownership covers both normal PCI BAR windows and CXL.mem regions uniformly; non-VFIO RAM-device regions such as NVDIMMs are unaffected and continue through the normal mapping path.
--- hw/vfio/listener.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
hw/vfio/listener.c +14 -0
@@ -608,6 +608,20 @@ void vfio_container_region_add(VFIOContainer *bcontainer, pgmask + 1);return ; } + +/* +* VFIO MMAP backed regions (CXL.mem) uses VM_IO | VM_PFNMAP VMAs +* backed by physical device addresses. Skip vfio_container_dma_map +* as mapping is not needed for this region. +*/ +if (vfio_get_vfio_device (memory_region_owner (section->mr))) { +trace_vfio_listener_region_add_no_dma_map ( +memory_region_name (section->mr), + section->offset_within_address_space, +int128_getlo (section->size), + pgmask +1 ); +return ; + } } ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize),
Signature · 1 line
2.25.1
Quoted 5 lines · On Mon, 27 Apr 2026 23:42:26 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
This series adds QEMU-side support for passing CXL Type-2 devices (GPUs and accelerators with host-managed device memory) to VMs via vfio-pci.
Hi Manish,
Having read this description I'm not sure I really understand all the constraints this is operating under.
There are two basic paths to getting safe virtualisation of CXL.mem.
1) Lock down decoders etc against preconfigured mappings.
2) Constrain the more insane stuff via topology care - basically direct connect only for now and let the guest program stuff appropriately.
One possible constraint is making it all look like early CXL host stuff where the bios does the heavy lifting. I don't mind if we have to do that today but I'd like to understand if it's a long term requirement or not.
I think this is currently doing a mix of 1 and 2. You lock endpoint decoders but ignore host bridge ones. I think we need to handle those. Maybe I'm missing a check that a pass through decoder is in use (there is a commandline control to turn that on / off for single RP HB).
Quoted 19 lines
It pairs with the kernel series "vfio/pci: CXL Type-2 passthrough"[1] posted to the vfio mailing list. Patches 3-7 need that kernel series present to do anything useful. I am new to QEMU development, so please forgive and point me in the right direction for correct infrastructure decisions.
Background
----------CXL Type-2 devices expose device memory (CXL.mem) through HDM decoders. The kernel vfio-pci driver shadows the HDM Decoder Capability registers so userspace can observe and control decoder commits without touching the hardware register page directly.
Without this series, the guest never sees the device memory range and the HDM decoder goes unconfigured. The device shows up but its memory is unreachable.
Quoted 6 lines
Design decisions
----------------CXL.mem is exposed to the guest as a dedicated GPA window declared in ACPI (CEDT/CFMWS) rather than a PCI BAR. The HDM decoder BASE must match the CFMWS base and remain stable; BAR assignment is not stable.
Note a lot of my thinking here is based on the related but not identical question of how to do CXL type 3 emulation for virtualization cases.
My thinking for type 2 is we'd do similar tricks to for memory in constraining the topology so things like interleave don't cause us problems and CFMWS to particular device mappings are fixed (though in theory offsets with in CFMWS could change).
I may be missing the point entirely but why does HDM decoder base (here the GPA address where the routing is configured to put the DPA of the EP) need to match the CFMWS base or for that matter remain stable (i.e. Guest forced to put it in the same place)? Is it just so the type 2 driver knows where to find it? (There are easy ways around that - or is this something baked into an existing driver?)
There are a few easy ways to constrain things to reduce flexibility (e.g. we don't want interleave to be possible etc). If we absolutely have to we can also lock decoders. The only reason I was avoiding that for type 3 device was that it isn't needed + can get complex fast!
I think you are constraining the topology for this to work to PXB with 1 RP with pass through HDM decoders and no switches etc.
That's fine for now but I didn't immediately spot the checks that enforce that.
Can we have a command line example in this cover letter.
Quoted 3 lines
A separate
VIRT_HIGH_CXL_MMIOwindow in the ARM virt memory map carries this GPA range, independent of the existing PCIe MMIO slots.
I'm not immediately understanding why the CXL.mem path isn't via exiting CFMWS support. Why do we need a new memory window? Ultimately you seem to map to normal CFMWS regions. Also note you need to be very careful adding stuff to the memory map as bunch of other stuff above that will move - so it must simply not be there by default to avoid breaking migration of existing VMs. Maybe that is fine, I didn't check carefully
Quoted 10 lines
The Component Register BAR contains two distinct ranges. Accelerator register windows are passed through as direct hardware mmaps via
VFIO_REGION_INFO_CAP_SPARSE_MMAP. The HDM Decoder Capability block is excluded from that sparse list by the kernel and must be intercepted by QEMU to track decoder state. A single priority-1COMP_REGSoverlay placed athdm_regs_offsetinside the BAR container wins over any hardware-backed alias at the same offset, with no per-window aliasing required.
Quoted 3 lines
The guest has no mechanism to remap host physical mappings. QEMU programs decoder 0 with the CFMWS base through the kernel's
COMP_REGSshadow atmachine_donetime, after all devices are realized and before the guest starts.
Ah so this is locking down the decoder? I'm not necessarily against that but would like to understand what the constraints are that lead to that being necessary as opposed to guest programming them later (and late set up of the MemoryRegion like we do for type 3)
Quoted 12 lines
The notifier is registered only for devices the kernel reports as firmware-committed (
VFIO_CXL_CAP_FIRMWARE_COMMITTED).The CXL.mem MemoryRegion is a mmap-backed RAM-device region backed by a VM_IO|VM_PFNMAP VMA. The VFIO MemoryListener would attempt an IOMMU DMA mapping for it when it is added to
system_memory, which always
fails:pin_user_pages()refusesVM_IOpages. No IOMMU mapping is needed for these regions - CPU access goes via KVM Stage-2 page faults and device DMA to RAM uses separate per-RAM-section IOMMU entries. The listener is extended to skip the mapping attempt for VFIO-owned RAM-device regions.
Quoted 2 lines
pxb-cxl bridges had no
_DSMmethod. Without_DSMfunction 5 the OS defaults to treating PCI configuration as reassignable.
That one is annoyingly controversial. I see you have Shameer's patch so he can go into history of why. I tried to land that in the past and I was far from the first.
Quoted 3 lines
On machines with firmware-committed HDM decoders that reassignment breaks the CXL.mem mapping, so the
_DSMis added with preserve_config=true for ARM and false for x86.
What has that got to do with CXL.mem mapping? It can move the BARs around but those aren't related to the CXL.mem mapping.
Can we break this description up into two separate parts as it feels like BAR mappings and CXL.mem mappings are getting confused.
1) Deal with how the configuration stuff works - bars etc.
2) Deal with the CXL.mem mapping into a CFMWS.
Quoted 4 lines
Known issues:
- The bios-tables test will fail due to the_DSMaddition. A fix will be provided in a follow-up round.
Unless I'm missing some checks, a whole load more than that will fail as device memory / memory hotplug region + probably the CFMWS regions all move.
Quoted 9 lines
-
VFIO_CXL_CAP_CACHE_CAPABLEwill require additional handling.
- Devices with multiple firmware-committed HDM decoders are not fully supported.
- Non-firmware-committed devices are not supported.
- linux-headers sync is manual and temporary; once the kernel series is merged, this patch will be replaced with script generated update.
Anyhow, seems like a good starting point for discussion.
As we evolve the CXL support into including real virtualization use cases I'm sure we'll also throw up random corners around resetting etc.
Note I mostly skipped the VFIO stuff on basis others know a lot more on that side of things than I do!
Thanks,
Jonathan
Quoted 38 lines
Manish Honap (9):
hw/arm/virt: Add CXL FMWS PA window for device memory
cxl: Addpreserve_configto pxb-cxl OSC method
linux-headers: Updatevfio.hfor CXL Type-2 device passthrough hw/vfio/region: Addvfio_region_setup_with_ops()for custom region ops hw/vfio/pci: Add CXL Type-2 device detection and region setup hw/vfio/pci: Wire CXL component-register BAR withCOMP_REGSoverlay hw/vfio+cxl: Program HDM decoder 0 atmachine_donefor
firmware-committed devices
hw/arm/smmu-common: Allow pxb-cxl as SMMUv3 primary bus vfio/listener: Skip DMA mapping for VFIO-owned RAM-device regionshw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +- hw/arm/smmu-common.c | 17 +- hw/arm/virt-acpi-build.c | 5 + hw/arm/virt.c | 7 + hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 8 + hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 43 +++- hw/vfio/listener.c | 14 ++ hw/vfio/pci.c | 411 +++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 15 ++ hw/vfio/region.c | 15 +- hw/vfio/trace-events | 6 + hw/vfio/vfio-region.h | 3 + include/hw/acpi/cxl.h | 2 +- include/hw/arm/virt.h | 2 + include/hw/cxl/cxl_host.h | 10 + include/hw/pci-host/gpex.h | 2 + linux-headers/linux/vfio.h | 18 ++ 20 files changed, 570 insertions(+), 18 deletions(-)--
2.25.1
Quoted 103 lines
-----Original Message-----
From: Jonathan Cameron <[email protected]>
Sent: 19 May 2026 18:43
To: Manish Honap <[email protected]>
Cc: Alex Williamson <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Ankit Agrawal <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; alejandro.lucero- [email protected]; Vikram Sethi <[email protected]>; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; linux- [email protected]; [email protected]; [email protected]; qemu- [email protected]
Subject: Re: [RFC 0/9] QEMU: CXL Type-2 device passthrough via vfio-pciExternal email: Use caution opening links or attachments
Quoted 5 lines · On Mon, 27 Apr 2026 23:42:26 +0530 <[email protected]> wrote:
From: Manish Honap <[email protected]>
This series adds QEMU-side support for passing CXL Type-2 devices (GPUs and accelerators with host-managed device memory) to VMs via vfio-pci.
Hi Manish,
Having read this description I'm not sure I really understand all the constraints this is operating under.
There are two basic paths to getting safe virtualisation of CXL.mem.
1) Lock down decoders etc against preconfigured mappings.
2) Constrain the more insane stuff via topology care - basically direct connect
only for now and let the guest program stuff appropriately.One possible constraint is making it all look like early CXL host stuff where the bios does the heavy lifting. I don't mind if we have to do that today but I'd like to understand if it's a long term requirement or not.
I think this is currently doing a mix of 1 and 2. You lock endpoint decoders but ignore host bridge ones. I think we need to handle those. Maybe I'm missing a check that a pass through decoder is in use (there is a commandline control to turn that on / off for single RP HB).
Quoted 11 lines
It pairs with the kernel series "vfio/pci: CXL Type-2 passthrough"[1] posted to the vfio mailing list. Patches 3-7 need that kernel series present to do anything useful. I am new to QEMU development, so please forgive and point me in the right direction for correct infrastructure decisions.
Background
----------CXL Type-2 devices expose device memory (CXL.mem) through HDM
decoders.
Quoted 8 lines
The kernel vfio-pci driver shadows the HDM Decoder Capability registers so userspace can observe and control decoder commits without touching the hardware register page directly.
Without this series, the guest never sees the device memory range and the HDM decoder goes unconfigured. The device shows up but its memory is unreachable.
Quoted 7 lines
Design decisions
----------------CXL.mem is exposed to the guest as a dedicated GPA window declared in ACPI
(CEDT/CFMWS) rather than a PCI BAR. The HDM decoder BASE must match the CFMWS base and remain stable; BAR assignment is not stable.Note a lot of my thinking here is based on the related but not identical question of how to do CXL type 3 emulation for virtualization cases.
My thinking for type 2 is we'd do similar tricks to for memory in constraining the topology so things like interleave don't cause us problems and CFMWS to particular device mappings are fixed (though in theory offsets with in CFMWS could change).
I may be missing the point entirely but why does HDM decoder base (here the GPA address where the routing is configured to put the DPA of the EP) need to match the CFMWS base or for that matter remain stable (i.e. Guest forced to put it in the same place)? Is it just so the type 2 driver knows where to find it? (There are easy ways around that - or is this something baked into an existing driver?)
There are a few easy ways to constrain things to reduce flexibility (e.g.
we don't want interleave to be possible etc). If we absolutely have to we can also lock decoders. The only reason I was avoiding that for type 3 device was that it isn't needed + can get complex fast!
Quoted 6 lines
I think you are constraining the topology for this to work to PXB with 1 RP with pass through HDM decoders and no switches etc. That's fine for now but I didn't immediately spot the checks that enforce that.
Can we have a command line example in this cover letter.
Restating the paths for CXL.mem virtualization for v2 scope
Path 1: lock down decoders against preconfigured mappings. Path 2: constrain topology (direct-connect only, no switches,
no interleave) and let the guest program decoders.
For V2, I will:
(a) Document the Path 1 / Path 2 split explicitly in the cover letter, and state that this series implements Path 1 for the endpoint inside a Path-2-constrained topology.
(b) Add a command-line example to the cover letter (you asked for this on patch 7/9 as well).
(c) Add realize-time checks in vfio_cxl_setup() that refuse to start unless:
- cxl_get_hb_passthrough(hb) is true
- pcie_count_ds_ports(hb->bus) == 1
- no switch is present between the cxl-rp and the vfio-pci
(d) Use existing CFMWS infrastructure and drop VIRT_HIGH_CXL_MMIO
Quoted 25 lines
Quoted 4 lines
A separate
VIRT_HIGH_CXL_MMIOwindow in the ARM virt memory map carries this GPA range, independent of the existing PCIe MMIO slots.I'm not immediately understanding why the CXL.mem path isn't via exiting CFMWS support. Why do we need a new memory window? Ultimately you seem to map to normal CFMWS regions. Also note you need to be very careful adding stuff to the memory map as bunch of other stuff above that will move - so it must simply not be there by default to avoid breaking migration of existing VMs. Maybe that is fine, I didn't check carefully
Quoted 10 lines
The Component Register BAR contains two distinct ranges. Accelerator register windows are passed through as direct hardware mmaps via
VFIO_REGION_INFO_CAP_SPARSE_MMAP. The HDM Decoder Capability block is excluded from that sparse list by the kernel and must be intercepted by QEMU to track decoder state. A single priority-1COMP_REGSoverlay placed athdm_regs_offsetinside the BAR container wins over any hardware-backed alias at the same offset, with no per-window aliasing required.Quoted 3 lines
The guest has no mechanism to remap host physical mappings. QEMU programs decoder 0 with the CFMWS base through the kernel's
COMP_REGSshadow atmachine_donetime, after all devices are realized and before
Quoted 6 lines
the guest starts.
Ah so this is locking down the decoder? I'm not necessarily against that but would like to understand what the constraints are that lead to that being necessary as opposed to guest programming them later (and late set up of the MemoryRegion like we do for type 3)
Decoder locking is required for this series because the kernel-side endpoint driver expects a committed decoder before vfio-cxl probe completes. The host CXL stack commits the endpoint HDM decoder before vfio binds; QEMU then locks it again at machine_done with the GPA so the guest cannot move it.
Quoted 13 lines
Quoted 13 lines
The notifier is registered only for devices the kernel reports as firmware-committed (
VFIO_CXL_CAP_FIRMWARE_COMMITTED).The CXL.mem MemoryRegion is a mmap-backed RAM-device region backed by a VM_IO|VM_PFNMAP VMA. The VFIO MemoryListener would attempt an IOMMU DMA mapping for it when it is added to
system_memory, which always
fails:pin_user_pages()refusesVM_IOpages. No IOMMU mapping is needed for these regions - CPU access goes via KVM Stage-2 page faults and device DMA to RAM uses separate per-RAM-section IOMMU entries. The listener is extended to skip the mapping attempt for VFIO-owned RAM-device regions.
Quoted 6 lines
Quoted 2 lines
pxb-cxl bridges had no
_DSMmethod. Without_DSMfunction 5 the OS defaults to treating PCI configuration as reassignable.That one is annoyingly controversial. I see you have Shameer's patch so he can go into history of why. I tried to land that in the past and I was far from the first.
okay, I will check this part for additional details.
Quoted 5 lines
Quoted 4 lines
On machines with firmware-committed HDM decoders that reassignment breaks the CXL.mem mapping, so the
_DSMis added with preserve_config=true for ARM and false for x86.
Quoted 9 lines
What has that got to do with CXL.mem mapping? It can move the BARs around but those aren't related to the CXL.mem mapping.
Can we break this description up into two separate parts as it feels like BAR mappings and CXL.mem mappings are getting confused.
1) Deal with how the configuration stuff works - bars etc.
2) Deal with the CXL.mem mapping into a CFMWS.
Yes, agreed these two are independent.
For v2, I will split the cover-letter narrative accordingly and reorder patches so the CXL.mem story (HDM decoder, FMWS, decoder lock) sits in one block and the PCI/BAR story (_DSM preserve_config) sits in another.
Quoted 14 lines
Quoted 4 lines
Known issues:
- The bios-tables test will fail due to the_DSMaddition. A fix will be provided in a follow-up round.Unless I'm missing some checks, a whole load more than that will fail as device memory / memory hotplug region + probably the CFMWS regions all move.
Quoted 5 lines
-
VFIO_CXL_CAP_CACHE_CAPABLEwill require additional handling.
- Devices with multiple firmware-committed HDM decoders are not fully supported.
- Non-firmware-committed devices are not supported.
- linux-headers sync is manual and temporary; once the kernel series
Quoted 8 lines
is
Quoted 6 lines
merged, this patch will be replaced with script generated update.
[1]
https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidi
a.comAnyhow, seems like a good starting point for discussion.
Thank you for your suggestions.
Quoted 51 lines
As we evolve the CXL support into including real virtualization use cases I'm sure we'll also throw up random corners around resetting etc.
Note I mostly skipped the VFIO stuff on basis others know a lot more on that side of things than I do!
Thanks,
Jonathan
Quoted 5 lines
Manish Honap (9):
hw/arm/virt: Add CXL FMWS PA window for device memory
cxl: Addpreserve_configto pxb-cxl OSC method
linux-headers: Updatevfio.hfor CXL Type-2 device passthrough hw/vfio/region: Addvfio_region_setup_with_ops()for custom regionops
Quoted 7 lines
hw/vfio/pci: Add CXL Type-2 device detection and region setup hw/vfio/pci: Wire CXL component-register BAR with
COMP_REGSoverlay hw/vfio+cxl: Program HDM decoder 0 atmachine_donefor
firmware-committed devices
hw/arm/smmu-common: Allow pxb-cxl as SMMUv3 primary bus vfio/listener: Skip DMA mapping for VFIO-owned RAM-device regionsQuoted 11 lines
hw/acpi/cxl-stub.c | 2 +- hw/acpi/cxl.c | 4 +- hw/arm/smmu-common.c | 17 +- hw/arm/virt-acpi-build.c | 5 + hw/arm/virt.c | 7 + hw/cxl/cxl-host-stubs.c | 2 + hw/cxl/cxl-host.c | 8 + hw/i386/acpi-build.c | 2 +- hw/pci-host/gpex-acpi.c | 43 +++- hw/vfio/listener.c | 14 ++ hw/vfio/pci.c | 411+++++++++++++++++++++++++++++++++++++
Quoted 15 lines
hw/vfio/pci.h | 15 ++ hw/vfio/region.c | 15 +- hw/vfio/trace-events | 6 + hw/vfio/vfio-region.h | 3 + include/hw/acpi/cxl.h | 2 +- include/hw/arm/virt.h | 2 + include/hw/cxl/cxl_host.h | 10 + include/hw/pci-host/gpex.h | 2 + linux-headers/linux/vfio.h | 18 ++ 20 files changed, 570 insertions(+), 18 deletions(-)--
2.25.1
Quoted 21 lines
-----Original Message-----
From: Manish Honap <[email protected]>
Sent: 01 June 2026 08:56
To: Jonathan Cameron <[email protected]>
Cc: Alex Williamson <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Ankit Agrawal <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Vikram Sethi <[email protected]>; Neo Jia <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Krishnakant Jaju <[email protected]>; [email protected]; [email protected]; qemu- [email protected]; [email protected]; Manish Honap <[email protected]>
Subject: RE: [RFC 0/9] QEMU: CXL Type-2 device passthrough via vfio-pci
[...]
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
pxb-cxl bridges had no
_DSMmethod. Without_DSMfunction 5 the OS defaults to treating PCI configuration as reassignable.That one is annoyingly controversial. I see you have Shameer's patch so he can go into history of why. I tried to land that in the past and I was far from the first.
okay, I will check this part for additional details.
I guess the concern here is the QEMU regression reported with certain devices with legacy IO port BARs having issues on arm64 when _DSM #5 is specified:
https://lore.kernel.org/all/[email protected]/
For accel SMMUv3 this is fine as we restrict the devices to vfio-pci only. One option, until there is a proper fix in the Linux kernel or EDK2, might be to maintain a checklist so that devices known to have issues are not attached when _DSM #5 is advertised.
Thanks,
Shameer