hw/mem: CXL Type-3 Volatile Memory Support
v5 cover hw/mem: CXL Type-3 Volatile Memory Support
v5: Thanks to Michael Tsirkin for review
- Use memory_region_size() instead of incorrect open coding. Add a precursor patch tidying up existing uses.
Depends on some of the fix series posted (mostly fuzz). [PATCH 0/2] hw/cxl: CDAT file handling fixes. [PATCH v2 0/3] hw/cxl: Fix decoder commit and uncommit handlin [PATCH 0/3] docs/cxl: Gathering of fixes for 8.0 CXL docs.
Based on: Message-ID: [email protected] Based on: Message-ID: [email protected] Based on: Message_ID: [email protected]/
Now kernel support is in place, lets catch up with the emulation of volatile memory on CXL type 3.
Original cover letter with minor updates.
This patches provides 2 features to the CXL Type-3 Device:
1) Volatile Memory Region Support
2) Multi-Region support (1 Volatile, 1 Persistent)
Summary of Changes per-commit:
1) Whitespace updates to docs and tests
2) Refactor CDAT DSMAS Initialization for multi-region initialization Multi-Region and Volatile Memory support for CXL Type-3 Devices Test and Documentation updates
The final patch in this series makes 6 major changes to the type-3 device in order to implement multi-region and volatile region support
1) The HostMemoryBackend [hostmem] has been replaced by two [hostvmem] and [hostpmem] to store volatile and persistent memory respectively
2) The single AddressSpace has been replaced by two AddressSpaces [hostvmem_as] and [hostpmem_as] to map respective memdevs.
3) Each memory region size and total region are stored separately
4) The CDAT and DVSEC memory map entries have been updated:
a) if vmem is present, vmem is mapped at DPA(0)
b) if pmem is present
i) and vmem is present, pmem is mapped at DPA(vmem->size) ii) else, pmem is mapped at DPA(0)
c) partitioning of pmem is not supported in this patch set but has been discussed and this design should suffice.
5) Read/Write functions have been updated to access AddressSpaces according to the mapping described in #4. Access to the persistent address space is calculated by (dpa-vmem_len)
6) cxl-mailbox has been updated to report the respective size of volatile and persistent memory region.
Gregory Price (2):
tests/qtest/cxl-test: whitespace, line ending cleanup hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
Jonathan Cameron (1):
hw/mem: Use memory_region_size() in cxl_type3
docs/about/deprecated.rst | 8 + docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 32 ++-- hw/mem/cxl_type3.c | 300 ++++++++++++++++++++++++--------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 146 +++++++++++----- 7 files changed, 407 insertions(+), 147 deletions(-)
Signature · 1 line
2.37.2
PATCH v5 1/3 tests/qtest/cxl-test: whitespace, line ending cleanup
From: Gregory Price <[email protected]>
Defines are starting to exceed line length limits, align them for cleanliness before making modifications.
--- tests/qtest/cxl-test.c | 84 +++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 38 deletions(-)
tests/qtest/cxl-test.c +46 -38
@@ -8,50 +8,58 @@#include "qemu/osdep.h" #include "libqtest-single.h" -#define QEMU_PXB_CMD "-machine q35,cxl=on " \ - "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G " +#define QEMU_PXB_CMD \ + "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G " -#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ - "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ - "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G " +#define QEMU_2PXB_CMD \ + "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G " -#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " +#define QEMU_RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "/* Dual ports on first pxb */ -#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ - "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " +#define QEMU_2RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 "/* Dual ports on each of the pxb instances */ -#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ - "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ - "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ - "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " - -#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " - -#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " - -#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " \ - "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2,id=cxl-pmem2 " \ - "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3,id=cxl-pmem3 " +#define QEMU_4RP \ + "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " + +#define QEMU_2T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " + +#define QEMU_4T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2,id=cxl-pmem2 " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3,id=cxl-pmem3 "static void cxl_basic_hb (void ) {
Signature · 1 line
2.37.2
PATCH v5 2/3 hw/mem: Use memory_region_size() in cxl_type3
In reply to Jonathan CameronAccessors prefered over direct use of int128_get64() as they clamp out of range values. None are expected here but cleaner to always use the accessor than mix and match.
---
v5: New patch to tidy up existing instance before adding more of
them.
- Use memory_region_size() to access the size of memory regions. We may eventually need to allow for larger addresses but it is unlikely to be a problem any time soon.
--- hw/mem/cxl_type3.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
hw/mem/cxl_type3.c +4 -4
@@ -52,7 +52,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = CDAT_DSMAS_FLAG_NV, .DPA_base = 0, - .DPA_length = int128_get64(mr->size), + .DPA_length = memory_region_size(mr), };/* For now, no memory side cache, plausiblish numbers */
@@ -133,7 +133,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table,/* Reserved - the non volatile from DSMAS matters */ .EFI_memory_type_attr = 2, .DPA_offset = 0, - .DPA_length = int128_get64(mr->size), + .DPA_length = memory_region_size(mr), };/* Header always at start of structure */
@@ -698,7 +698,7 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data,return MEMTX_ERROR ; } -if (dpa_offset >int128_get64 (mr->size)) { +if (dpa_offset >memory_region_size (mr)) {return MEMTX_ERROR ; }
@@ -721,7 +721,7 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,return MEMTX_OK ; } -if (dpa_offset >int128_get64 (mr->size)) { +if (dpa_offset >memory_region_size (mr)) {return MEMTX_OK ; } return address_space_write(&ct3d->hostmem_as, dpa_offset, attrs,
Signature · 1 line
2.37.2
Re: [PATCH v5 2/3] hw/mem: Use memory_region_size() in cxl_type3
Quoted 59 lines · Jonathan Cameron
Accessors prefered over direct use of
int128_get64()as they clamp out of range values. None are expected here but cleaner to always use the accessor than mix and match.Signed-off-by Jonathan Cameron <[email protected]>---
v5: New patch to tidy up existing instance before adding more of
them.
- Usememory_region_size()to access the size of memory regions. We may eventually need to allow for larger addresses but it is unlikely to be a problem any time soon.--- hw/mem/cxl_type3.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)hw/mem/cxl_type3.c +4
-4@@ -52,7 +52,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = CDAT_DSMAS_FLAG_NV, .DPA_base = 0, - .DPA_length = int128_get64(mr->size), + .DPA_length = memory_region_size(mr), };/* For now, no memory side cache, plausiblish numbers */ @@ -133,7 +133,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table,/* Reserved - the non volatile from DSMAS matters */ .EFI_memory_type_attr = 2, .DPA_offset = 0, - .DPA_length = int128_get64(mr->size), + .DPA_length = memory_region_size(mr), };/* Header always at start of structure */ @@ -698,7 +698,7 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data,return MEMTX_ERROR ; } -if (dpa_offset >int128_get64 (mr->size)) { +if (dpa_offset >memory_region_size (mr)) {return MEMTX_ERROR ; }@@ -721,7 +721,7 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,return MEMTX_OK ; } -if (dpa_offset >int128_get64 (mr->size)) { +if (dpa_offset >memory_region_size (mr)) {return MEMTX_OK ; } return address_space_write(&ct3d->hostmem_as, dpa_offset, attrs,Signature · 1 line
2.37.2
lgtm
PATCH v5 3/3 hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
In reply to Jonathan CameronFrom: Gregory Price <[email protected]>
This commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region.
Two new properties have been added to cxl-type3 device initialization:
[volatile-memdev] and [persistent-memdev]
The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property.
Partitioning volatile memory from persistent memory is not yet supported.
Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info.
---
v5:
- memory_region_size() to access the size where needed.
- fix a bug where the mem_size element of cxl_dstate was being treated as an int128. It's 64bit.
--- docs/about/deprecated.rst | 8 + docs/system/devices/cxl.rst | 49 ++++-- hw/cxl/cxl-mailbox-utils.c | 32 ++-- hw/mem/cxl_type3.c | 296 +++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/bios-tables-test.c | 8 +- tests/qtest/cxl-test.c | 76 +++++++-- 7 files changed, 366 insertions(+), 114 deletions(-)
docs/about/deprecated.rst +8 -0
@@ -284,6 +284,14 @@ from Intel that was not properly allocated. Since version 5.2, the controller has used a properly allocated identifier. Deprecate the ``use-intel-id`` machine compatibility parameter. +``-device cxl-type3,memdev=xxxx`` (since 8.0) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``cxl-type3`` device initially only used a single memory backend. With +the addition of volatile memory support, it is now necessary to distinguish +between persistent and volatile memory backends. As such, memdev is deprecated +in favor of persistent-memdev. + Block device options ''''''''''''''''''''
docs/system/devices/cxl.rst +39 -10
81 lines not on this page · raw message
hw/cxl/cxl-mailbox-utils.c +18 -14
@@ -141,7 +141,8 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, }QEMU_PACKED *fw_info;QEMU_BUILD_BUG_ON (sizeof (*fw_info) !=0x50 ); -if (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER ) { +if ((cxl_dstate->vmem_size <CXL_CAPACITY_MULTIPLIER ) || + (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER )) {return CXL_MBOX_INTERNAL_ERROR ; }
@@ -288,21 +289,21 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate);CXLType3Class *cvc =CXL_TYPE3_GET_CLASS (ct3d); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } id = (void *)cmd->payload;memset (id,0 ,sizeof (*id)); -/* PMEM only */ snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); - id->total_capacity = size /CXL_CAPACITY_MULTIPLIER ; - id->persistent_capacity = size /CXL_CAPACITY_MULTIPLIER ; - id->lsa_size = cvc->get_lsa_size (ct3d); +stq_le_p (&id->total_capacity, cxl_dstate->mem_size /CXL_CAPACITY_MULTIPLIER ); +stq_le_p (&id->persistent_capacity, cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ); +stq_le_p (&id->volatile_capacity, cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ); +stl_le_p (&id->lsa_size, cvc->get_lsa_size (ct3d)); *len =sizeof (*id);return CXL_MBOX_SUCCESS ;
@@ -319,17 +320,20 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,uint64_t next_pmem; }QEMU_PACKED *part_info = (void *)cmd->payload;QEMU_BUILD_BUG_ON (sizeof (*part_info) !=0x20 ); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } -/* PMEM only */ - part_info->active_vmem =0 ; - part_info->next_vmem =0 ; - part_info->active_pmem = size /CXL_CAPACITY_MULTIPLIER ; - part_info->next_pmem =0 ; +stq_le_p (&part_info->active_vmem, cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ); +/* +* When both next_vmem and next_pmem are 0, there is no pending change to +* partitioning. +*/ +stq_le_p (&part_info->next_vmem,0 ); +stq_le_p (&part_info->active_pmem, cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ); +stq_le_p (&part_info->next_pmem,0 ); *len =sizeof (*part_info);return CXL_MBOX_SUCCESS ;
hw/mem/cxl_type3.c +223 -73
428 lines not on this page · raw message
include/hw/cxl/cxl_device.h +8 -3
@@ -119,8 +119,10 @@ typedef struct cxl_device_state {uint64_t host_set; } timestamp; -/* memory region for persistent memory, HDM */ +/* memory region size, HDM */ +uint64_t mem_size;uint64_t pmem_size; +uint64_t vmem_size; } CXLDeviceState;/* Initialize the register block for a device */
@@ -245,12 +247,15 @@ struct CXLType3Dev {PCIDevice parent_obj;/* Properties */ -HostMemoryBackend *hostmem; +HostMemoryBackend *hostmem;/* deprecated */ +HostMemoryBackend *hostvmem; +HostMemoryBackend *hostpmem;HostMemoryBackend *lsa;uint64_t sn;/* State */ -AddressSpace hostmem_as; +AddressSpace hostvmem_as; +AddressSpace hostpmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate;
tests/qtest/bios-tables-test.c +4 -4
@@ -1902,13 +1902,13 @@ static void test_acpi_q35_cxl(void)" -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1" " -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2" " -device cxl-rp,port=0,bus=cxl.1,id=rp1,chassis=0,slot=2" -" -device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1" +" -device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1" " -device cxl-rp,port=1,bus=cxl.1,id=rp2,chassis=0,slot=3" -" -device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2" +" -device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2" " -device cxl-rp,port=0,bus=cxl.2,id=rp3,chassis=0,slot=5" -" -device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3" +" -device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3" " -device cxl-rp,port=1,bus=cxl.2,id=rp4,chassis=0,slot=6" -" -device cxl-type3,bus=rp4,memdev=cxl-mem4,lsa=lsa4" +" -device cxl-type3,bus=rp4,persistent-memdev=cxl-mem4,lsa=lsa4" " -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k," "cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.targets.1=cxl.2,cxl-fmw.1.size=4G,cxl-fmw.1.interleave-granularity=8k" , tmp_path, tmp_path, tmp_path, tmp_path,
tests/qtest/cxl-test.c +66 -10
119 lines not on this page · raw message
Signature · 1 line
2.37.2