hw/mem: CXL Type-3 Volatile Memory Support
v3 cover hw/mem: CXL Type-3 Volatile Memory Support
v3: Noticed whilst chasing an unrelated kernel bug.
- Drop setting of DVSEC range base addresses. Whilst harmless, expectation is that this will be 0 until the OS sets it (or uses HDM decoders instead)
Based on following series (in order)
1. [PATCH v4 00/10] hw/cxl: CXL emulation cleanups and minor fixes for upstream
2. [PATCH v4 0/8] hw/cxl: RAS error emulation and injection
3. [PATCH 0/2] hw/cxl: Passthrough HDM decoder emulation
Based on: Message-Id: [email protected]
Based-on: Message-id: [email protected]
Based-on: Message-id: [email protected]
Kernel code is queued up in kernel.org cxl/next for the merge window that just opened.
Now we have some kernel code to test this against (and it looks good) I'd like to propose this series for upstream following 3 other series already proposed for inclusion:
Original cover letter with minor updates.
This patches provides 2 features to the CXL Type-3 Device:
1) Volatile Memory Region Support
2) Multi-Region support (1 Volatile, 1 Persistent)
Summary of Changes per-commit:
1) Whitespace updates to docs and tests
2) Refactor CDAT DSMAS Initialization for multi-region initialization Multi-Region and Volatile Memory support for CXL Type-3 Devices Test and Documentation updates
The final patch in this series makes 6 major changes to the type-3 device in order to implement multi-region and volatile region support
1) The HostMemoryBackend [hostmem] has been replaced by two [hostvmem] and [hostpmem] to store volatile and persistent memory respectively
2) The single AddressSpace has been replaced by two AddressSpaces [hostvmem_as] and [hostpmem_as] to map respective memdevs.
3) Each memory region size and total region are stored separately
4) The CDAT and DVSEC memory map entries have been updated:
a) if vmem is present, vmem is mapped at DPA(0)
b) if pmem is present
i) and vmem is present, pmem is mapped at DPA(vmem->size) ii) else, pmem is mapped at DPA(0)
c) partitioning of pmem is not supported in this patch set but has been discussed and this design should suffice.
5) Read/Write functions have been updated to access AddressSpaces according to the mapping described in #4. Access to the persistent address space is calculated by (dpa-vmem_len)
6) cxl-mailbox has been updated to report the respective size of volatile and persistent memory region
Gregory Price (2):
tests/qtest/cxl-test: whitespace, line ending cleanup hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 26 +-- hw/mem/cxl_type3.c | 294 +++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 146 +++++++++++----- 6 files changed, 392 insertions(+), 142 deletions(-)
Signature · 1 line
2.37.2
PATCH v3 1/2 tests/qtest/cxl-test: whitespace, line ending cleanup
From: Gregory Price <[email protected]>
Defines are starting to exceed line length limits, align them for cleanliness before making modifications.
--- tests/qtest/cxl-test.c | 84 +++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 38 deletions(-)
tests/qtest/cxl-test.c +46 -38
@@ -8,50 +8,58 @@#include "qemu/osdep.h" #include "libqtest-single.h" -#define QEMU_PXB_CMD "-machine q35,cxl=on " \ - "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G " +#define QEMU_PXB_CMD \ + "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G " -#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ - "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ - "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G " +#define QEMU_2PXB_CMD \ + "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G " -#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " +#define QEMU_RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "/* Dual ports on first pxb */ -#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ - "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " +#define QEMU_2RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 "/* Dual ports on each of the pxb instances */ -#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ - "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ - "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ - "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " - -#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " - -#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " - -#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " \ - "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2,id=cxl-pmem2 " \ - "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3,id=cxl-pmem3 " +#define QEMU_4RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " + +#define QEMU_2T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " + +#define QEMU_4T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2,id=cxl-pmem2 " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3,id=cxl-pmem3 "static void cxl_basic_hb (void ) {
Signature · 1 line
2.37.2
PATCH v3 2/2 hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
In reply to Jonathan CameronFrom: Gregory Price <[email protected]>
This commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region.
Two new properties have been added to cxl-type3 device initialization:
[volatile-memdev] and [persistent-memdev]
The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property.
Partitioning volatile memory from persistent memory is not yet supported.
Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info.
---
v3:
- Don't set the DVSEC range register base address
v2:
- Fixed an off by one in address space selection.
- Gather tags.
--- docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 26 +-- hw/mem/cxl_type3.c | 294 +++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 76 +++++++-- 6 files changed, 353 insertions(+), 111 deletions(-)
docs/system/devices/cxl.rst +39 -10
@@ -300,7 +300,7 @@ Example topology involving a switch:: Example command lines --------------------- -A very simple setup with just one directly attached CXL Type 3 device:: +A very simple setup with just one directly attached CXL Type 3 Persistent Memory device:: qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ ...
@@ -308,7 +308,28 @@ A very simple setup with just one directly attached CXL Type 3 device:: -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +A very simple setup with just one directly attached CXL Type 3 Volatile Memory device:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +The same volatile setup may optionally include an LSA region:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way
@@ -328,13 +349,13 @@ the CXL Type3 device directly attached (no switches).:: -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ - -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ + -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ - -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ + -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ - -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ + -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave::
@@ -354,15 +375,23 @@ An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ -device cxl-upstream,bus=root_port0,id=us0 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ - -device cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ + -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ - -device cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ + -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ - -device cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ + -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ - -device cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ + -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k +Deprecations +------------ + +The Type 3 device [memdev] attribute has been deprecated in favor of the +[persistent-memdev] attributes. [memdev] will default to a persistent memory +device for backward compatibility and is incapable of being used in combination +with [persistent-memdev]. + Kernel Configuration Options ----------------------------
hw/cxl/cxl-mailbox-utils.c +15 -11
@@ -141,7 +141,8 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, }QEMU_PACKED *fw_info;QEMU_BUILD_BUG_ON (sizeof (*fw_info) !=0x50 ); -if (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER ) { +if ((cxl_dstate->vmem_size <CXL_CAPACITY_MULTIPLIER ) || + (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER )) {return CXL_MBOX_INTERNAL_ERROR ; }
@@ -288,20 +289,20 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate);CXLType3Class *cvc =CXL_TYPE3_GET_CLASS (ct3d); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } id = (void *)cmd->payload;memset (id,0 ,sizeof (*id)); -/* PMEM only */ snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); - id->total_capacity = size /CXL_CAPACITY_MULTIPLIER ; - id->persistent_capacity = size /CXL_CAPACITY_MULTIPLIER ; + id->total_capacity = cxl_dstate->mem_size /CXL_CAPACITY_MULTIPLIER ; + id->persistent_capacity = cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ; + id->volatile_capacity = cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ; id->lsa_size = cvc->get_lsa_size (ct3d); *len =sizeof (*id);
@@ -319,16 +320,19 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,uint64_t next_pmem; }QEMU_PACKED *part_info = (void *)cmd->payload;QEMU_BUILD_BUG_ON (sizeof (*part_info) !=0x20 ); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } -/* PMEM only */ - part_info->active_vmem =0 ; + part_info->active_vmem = cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ; +/* +* When both next_vmem and next_pmem are 0, there is no pending change to +* partitioning. +*/ part_info->next_vmem =0 ; - part_info->active_pmem = size /CXL_CAPACITY_MULTIPLIER ; + part_info->active_pmem = cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ; part_info->next_pmem =0 ; *len =sizeof (*part_info);
hw/mem/cxl_type3.c +221 -73
426 lines not on this page · raw message
include/hw/cxl/cxl_device.h +8 -3
@@ -119,8 +119,10 @@ typedef struct cxl_device_state {uint64_t host_set; } timestamp; -/* memory region for persistent memory, HDM */ +/* memory region size, HDM */ +uint64_t mem_size;uint64_t pmem_size; +uint64_t vmem_size; } CXLDeviceState;/* Initialize the register block for a device */
@@ -245,12 +247,15 @@ struct CXLType3Dev {PCIDevice parent_obj;/* Properties */ -HostMemoryBackend *hostmem; +HostMemoryBackend *hostmem;/* deprecated */ +HostMemoryBackend *hostvmem; +HostMemoryBackend *hostpmem;HostMemoryBackend *lsa;uint64_t sn;/* State */ -AddressSpace hostmem_as; +AddressSpace hostvmem_as; +AddressSpace hostpmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate;
tests/qtest/bios-tables-test.c +4 -4
@@ -1922,13 +1922,13 @@ static void test_acpi_q35_cxl(void)" -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1" " -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2" " -device cxl-rp,port=0,bus=cxl.1,id=rp1,chassis=0,slot=2" -" -device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1" +" -device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1" " -device cxl-rp,port=1,bus=cxl.1,id=rp2,chassis=0,slot=3" -" -device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2" +" -device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2" " -device cxl-rp,port=0,bus=cxl.2,id=rp3,chassis=0,slot=5" -" -device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3" +" -device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3" " -device cxl-rp,port=1,bus=cxl.2,id=rp4,chassis=0,slot=6" -" -device cxl-type3,bus=rp4,memdev=cxl-mem4,lsa=lsa4" +" -device cxl-type3,bus=rp4,persistent-memdev=cxl-mem4,lsa=lsa4" " -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k," "cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.targets.1=cxl.2,cxl-fmw.1.size=4G,cxl-fmw.1.interleave-granularity=8k" , tmp_path, tmp_path, tmp_path, tmp_path,
tests/qtest/cxl-test.c +66 -10
119 lines not on this page · raw message
Signature · 1 line
2.37.2
Re: [PATCH v3 2/2] hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
Quoted 115 lines · Jonathan Cameron
From: Gregory Price <[email protected]>
This commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region.
Two new properties have been added to cxl-type3 device initialization:
[volatile-memdev] and [persistent-memdev]The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property.
Partitioning volatile memory from persistent memory is not yet supported.
Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at
DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info.Signed-off-by Gregory Price <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Reviewed-by Fan Ni <[email protected]>Tested-by Fan Ni <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>---
v3:
- Don't set the DVSEC range register base address
v2:
- Fixed an off by one in address space selection.
- Gather tags.--- docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 26 +-- hw/mem/cxl_type3.c | 294 +++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 76 +++++++-- 6 files changed, 353 insertions(+), 111 deletions(-)docs/system/devices/cxl.rst +31
-10@@ -300,7 +300,7 @@ Example topology involving a switch:: Example command lines --------------------- -A very simple setup with just one directly attached CXL Type 3 device:: +A very simple setup with just one directly attached CXL Type 3 Persistent Memory device:: qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \@@ -308,7 +308,28 @@ A very simple setup with just one directly attached CXL Type 3 device:: -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +A very simple setup with just one directly attached CXL Type 3 Volatile Memory device:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +The same volatile setup may optionally include an LSA region:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way@@ -328,13 +349,13 @@ the CXL Type3 device directly attached (no switches).:: -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ - -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ + -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ - -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ + -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ - -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ + -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave::@@ -354,15 +375,23 @@ An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ -device cxl-upstream,bus=root_port0,id=us0 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ - -device cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ + -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ - -device cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ + -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ - -device cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ + -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ - -device cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ + -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \
docs/system/devices/cxl.rst An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: · line 394
Quoted 10 lines
docs/system/devices/cxl.rst +8
-0-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k +Deprecations +------------ + +The Type 3 device [memdev] attribute has been deprecated in favor of the +[persistent-memdev] attributes. [memdev] will default to a persistent memory +device for backward compatibility and is incapable of being used in combination +with [persistent-memdev]. +
Probably worth mentioning & referring this in docs/about/deprecated.rst.
Quoted 8 lines · On Tue, 21 Feb 2023 14:00:21 +0000 Jonathan Cameron…
v3: Noticed whilst chasing an unrelated kernel bug.
- Drop setting of DVSEC range base addresses. Whilst harmless, expectation is that this will be 0 until the OS sets it (or uses HDM decoders instead)Based on following series (in order)
1. [PATCH v4 00/10] hw/cxl: CXL emulation cleanups and minor fixes for upstream
2. [PATCH v4 0/8] hw/cxl: RAS error emulation and injection
I sent these in the wrong order. Now v5 (comment change and some tags) Other series unchanged.
Quoted 4 lines
3. [PATCH 0/2] hw/cxl: Passthrough HDM decoder emulation
Based on: Message-Id: [email protected]
Based-on: Message-id: [email protected]
Based-on: Message-id: [email protected]
Quoted 55 lines
Based-on: Message-id: [email protected]
Kernel code is queued up in kernel.org cxl/next for the merge window that just opened.
Now we have some kernel code to test this against (and it looks good) I'd like to propose this series for upstream following 3 other series already proposed for inclusion:
Original cover letter with minor updates.
This patches provides 2 features to the CXL Type-3 Device:
1) Volatile Memory Region Support
2) Multi-Region support (1 Volatile, 1 Persistent)Summary of Changes per-commit:
1) Whitespace updates to docs and tests
2) Refactor CDAT DSMAS Initialization for multi-region initialization Multi-Region and Volatile Memory support for CXL Type-3 Devices Test and Documentation updatesThe final patch in this series makes 6 major changes to the type-3 device in order to implement multi-region and volatile region support
1) The HostMemoryBackend [hostmem] has been replaced by two [hostvmem] and [hostpmem] to store volatile and persistent memory respectively
2) The single AddressSpace has been replaced by two AddressSpaces [hostvmem_as] and [hostpmem_as] to map respective memdevs.
3) Each memory region size and total region are stored separately
4) The CDAT and DVSEC memory map entries have been updated:
a) if vmem is present, vmem is mapped at DPA(0)
b) if pmem is present
i) and vmem is present, pmem is mapped atDPA(vmem->size)ii) else, pmem is mapped at DPA(0)
c) partitioning of pmem is not supported in this patch set but has been discussed and this design should suffice.
5) Read/Write functions have been updated to access AddressSpaces according to the mapping described in #4. Access to the persistent address space is calculated by (dpa-vmem_len)
6) cxl-mailbox has been updated to report the respective size of volatile and persistent memory regionGregory Price (2):
tests/qtest/cxl-test: whitespace, line ending cleanup hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 26 +-- hw/mem/cxl_type3.c | 294 +++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 146 +++++++++++----- 6 files changed, 392 insertions(+), 142 deletions(-)