QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
v7 cover QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
Whilst I have carried on Huai-Cheng Kuo's series version numbering and naming, there have been very substantial changes since v6 so I would suggest fresh review makes sense for anyone who has looked at this before. In particularly if the Avery design folks could check I haven't broken anything that would be great.
For reference v6: QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
https://lore.kernel.org/qemu-devel/[email protected]/
Summary of changes:
1) Linux headers definitions for DOE are now upstream so drop that patch.
2) Add CDAT for switch upstream port.
3) Generate 'plausible' default CDAT tables when a file is not provided.
4) General refactoring to calculate the correct table sizes and allocate based on that rather than copying from a local static array.
5) Changes from earlier reviews such as matching QEMU type naming style.
6) Moved compliance and SPDM usecases to future patch sets.
Sign-offs on these are complex because the patches were originally developed by Huai-Cheng Kuo, but posted by Chris Browy and then picked up by Jonathan Cameron who made substantial changes.
Huai-Cheng Kuo / Chris Browy, please confirm you are still happy to maintain this code as per the original MAINTAINERS entry.
What's here?
This series brings generic PCI Express Data Object Exchange support (DOE) DOE is defined in the PCIe Base Spec r6.0. It consists of a mailbox in PCI config space via a PCIe Extended Capability Structure.
The PCIe spec defines several protocols (including one to discover what protocols a given DOE instance supports) and other specification such as CXL define additional protocols using their own vendor IDs.
In this series we make use of the DOE to support the CXL spec defined Table Access Protocol, specifically to provide access to CDAT - a table specified in a specification that is hosted by the UEFI forum and is used to provide runtime discoverability of the sort of information that would otherwise be available in firmware tables (memory types, latency and bandwidth information etc).
The Linux kernel gained support for DOE / CDAT on CXL type 3 EPs in 6.0. The version merged did not support interrupts (earlier versions did so that support in the emulation was tested a while back).
This series provides CDAT emulation for CXL switch upstream ports and CXL type 3 memory devices. Note that to exercise the switch support additional Linux kernel patches are needed.
https://lore.kernel.org/linux-cxl/[email protected]/
(I'll post a new version of that support shortly)
Additional protocols will be supported by follow on patch sets:
* CXL compliance protocol.
* CMA / SPDM device attestation.
(Old version at https://gitlab.com/jic23/qemu/-/commits/cxl-next - will refresh that tree next week)
Huai-Cheng Kuo (3):
hw/pci: PCIe Data Object Exchange emulation hw/cxl/cdat: CXL CDAT Data Object Exchange implementation hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Jonathan Cameron (2):
hw/mem/cxl-type3: Add MSIX support hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE
MAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 222 ++++++++++++++++++++ hw/cxl/meson.build | 1 + hw/mem/cxl_type3.c | 236 +++++++++++++++++++++ hw/pci-bridge/cxl_upstream.c | 182 +++++++++++++++- hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_cdat.h | 166 +++++++++++++++ include/hw/cxl/cxl_component.h | 7 + include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++ include/hw/pci/pcie_regs.h | 4 + 15 files changed, 1323 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/pci/pcie_doe.h
Signature · 1 line
2.37.2
PATCH v7 1/5 hw/pci: PCIe Data Object Exchange emulation
From: Huai-Cheng Kuo <[email protected]>
Emulation of PCIe Data Object Exchange (DOE) PCIE Base Specification r6.0 6.3 Data Object Exchange
Supports multiple DOE PCIe Extended Capabilities for a single PCIe device. For each capability, a static array of DOEProtocol should be passed to pcie_doe_init(). The protocols in that array will be registered under the DOE capability structure. For each protocol, vendor ID, type, and corresponding callback function (handle_request()) should be implemented. This callback function represents how the DOE request for corresponding protocol will be handled.
pcie_doe_{read/write}_config() must be appended to corresponding PCI device's config_read/write() handler to enable DOE access. In pcie_doe_read_config(), false will be returned if pci_config_read() offset is not within DOE capability range. In pcie_doe_write_config(), the function will have no affect if the address is not within the related DOE PCIE extended capability.
--- MAINTAINERS | 7 + hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++++++ include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++++ include/hw/pci/pcie_regs.h | 4 + 7 files changed, 506 insertions(+)
MAINTAINERS +7 -0
@@ -1832,6 +1832,13 @@ F: qapi/pci.json F: docs/pci* F: docs/specs/*pci* +PCIE DOE +M: Huai-Cheng Kuo <[email protected]> +M: Chris Browy <[email protected]> +S: Supported +F: include/hw/pci/pcie_doe.h +F: hw/pci/pcie_doe.c + ACPI/SMBIOS M: Michael S. Tsirkin <[email protected]> M: Igor Mammedov <[email protected]>
hw/pci/meson.build +1 -0
@@ -13,6 +13,7 @@ pci_ss.add(files( # allow plugging PCIe devices into PCI buses, include them even if # CONFIG_PCI_EXPRESS=n. pci_ss.add(files('pcie.c', 'pcie_aer.c')) +pci_ss.add(files('pcie_doe.c')) softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c', 'pcie_host.c')) softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
hw/pci/pcie_doe.c new file +367 -0
367 lines not on this page · raw message
include/hw/pci/pci_ids.h +3 -0
@@ -157,6 +157,9 @@/* Vendors and devices. Sort key: vendor first, device next. */ +/* Ref: PCIe r6.0 Table 6-32 */ +#define PCI_VENDOR_ID_PCI_SIG 0x0001 +#define PCI_VENDOR_ID_LSI_LOGIC 0x1000#define PCI_DEVICE_ID_LSI_53C810 0x0001#define PCI_DEVICE_ID_LSI_53C895A 0x0012
include/hw/pci/pcie.h +1 -0
@@ -26,6 +26,7 @@#include "hw/pci/pcie_aer.h" #include "hw/pci/pcie_sriov.h" #include "hw/hotplug.h" +#include "hw/pci/pcie_doe.h" typedef enum { /* for attention and power indicator */
include/hw/pci/pcie_doe.h new file +123 -0
123 lines not on this page · raw message
include/hw/pci/pcie_regs.h +4 -0
@@ -179,4 +179,8 @@ typedef enum PCIExpLinkWidth {#define PCI_ACS_VER 0x1#define PCI_ACS_SIZEOF 8 +/* DOE Capability Register Fields */ +#define PCI_DOE_VER 0x1 +#define PCI_DOE_SIZEOF 24 +#endif /* QEMU_PCIE_REGS_H */
Signature · 1 line
2.37.2
PATCH v7 2/5 hw/mem/cxl-type3: Add MSIX support
In reply to Jonathan CameronThis will be used by several upcoming patch sets so break it out such that it doesn't matter which one lands first.
--- hw/mem/cxl_type3.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/mem/cxl_type3.c +9 -0
@@ -13,6 +13,7 @@#include "qemu/rcu.h" #include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" +#include "hw/pci/msix.h" /* * Null value of all Fs suggested by IEEE RA guidelines for use of
@@ -146,6 +147,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)ComponentRegisters *regs = &cxl_cstate->crb;MemoryRegion *mr = ®s->component_registers;uint8_t *pci_conf = pci_dev->config; +unsigned short msix_num =1 ; +int i;if (!cxl_setup_memory (ct3d, errp)) {return ;
@@ -180,6 +183,12 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , &ct3d->cxl_dstate.device_registers); + +/* MSI(-X) Initailization */ +msix_init_exclusive_bar (pci_dev, msix_num,4 ,NULL ); +for (i =0 ; i < msix_num; i++) { +msix_vector_use (pci_dev, i); + } }static void ct3_exit (PCIDevice *pci_dev)
Signature · 1 line
2.37.2
PATCH v7 3/5 hw/cxl/cdat: CXL CDAT Data Object Exchange implementation
In reply to Jonathan CameronFrom: Huai-Cheng Kuo <[email protected]>
The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.02, Oct. 2020" and "Compute Express Link Specification, Rev. 2.0, Oct. 2020"
This patch adds core support that will be shared by both end-points and switch port emulation.
---
Changes since RFC:
- Split out libary code from specific device.
--- hw/cxl/cxl-cdat.c | 222 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_cdat.h | 165 ++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 7 ++ include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + 6 files changed, 399 insertions(+)
hw/cxl/cxl-cdat.c new file +222 -0
@@ -0,0 +1,222 @@ +/* +* CXL CDAT Structure +* +* Copyright (C) 2021 Avery Design Systems, Inc. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" +#include "qapi/error.h" +#include "qemu/error-report.h" + +static void cdat_len_check (CDATSubHeader *hdr,Error **errp) +{ +assert (hdr->length); +assert (hdr->reserved ==0 ); + +switch (hdr->type) { +case CDAT_TYPE_DSMAS : +assert (hdr->length ==sizeof (CDATDsmas)); +break ; +case CDAT_TYPE_DSLBIS : +assert (hdr->length ==sizeof (CDATDslbis)); +break ; +case CDAT_TYPE_DSMSCIS : +assert (hdr->length ==sizeof (CDATDsmscis)); +break ; +case CDAT_TYPE_DSIS : +assert (hdr->length ==sizeof (CDATDsis)); +break ; +case CDAT_TYPE_DSEMTS : +assert (hdr->length ==sizeof (CDATDsemts)); +break ; +case CDAT_TYPE_SSLBIS : +assert (hdr->length >=sizeof (CDATSslbisHeader)); +assert ((hdr->length -sizeof (CDATSslbisHeader)) % +sizeof (CDATSslbe) ==0 ); +break ; +default : +error_setg (errp,"Type %d is reserved" , hdr->type); + } +} + +static void ct3_build_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATTableHeader *cdat_header = NULL; + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int ent, i; + +/* Use default table if fopen == NULL */ +assert (cdat->build_cdat_table); + + cdat_header =g_malloc0 (sizeof (*cdat_header)); +if (!cdat_header) { +error_setg (errp,"Failed to allocate CDAT header" ); +return ; + } + + cdat->built_buf_len = cdat->build_cdat_table (&cdat->built_buf, cdat->private); + +if (!cdat->built_buf_len) { +/* Build later as not all data available yet */ + cdat->to_update = true; +return ; + } + cdat->to_update = false; + + cdat_st =g_malloc0 (sizeof (*cdat_st) * (cdat->built_buf_len +1 )); +if (!cdat_st) { +error_setg (errp,"Failed to allocate CDAT entry array" ); +return ; + } + +/* Entry 0 for CDAT header, starts with Entry 1 */ +for (ent =1 ; ent < cdat->built_buf_len +1 ; ent++) { +CDATSubHeader *hdr = cdat->built_buf[ent -1 ]; +uint8_t *buf = (uint8_t *)cdat->built_buf[ent -1 ]; + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + + cdat_header->length += hdr->length; +for (i =0 ; i < hdr->length; i++) { + sum += buf[i]; + } + } + +/* CDAT header */ + cdat_header->revision =CXL_CDAT_REV ; +/* For now, no runtime updates */ + cdat_header->sequence =0 ; + cdat_header->length +=sizeof (CDATTableHeader); + sum += cdat_header->revision + cdat_header->sequence + + cdat_header->length; +/* Sum of all bytes including checksum must be 0 */ + cdat_header->checksum = ~sum +1 ; + + cdat_st[0 ].base =g_steal_pointer (&cdat_header); + cdat_st[0 ].length =sizeof (*cdat_header); + cdat->entry_len =1 + cdat->built_buf_len; + cdat->entry =g_steal_pointer (&cdat_st); +} + +static void ct3_load_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int num_ent; +int i =0 , ent =1 , file_size =0 ; +CDATSubHeader *hdr; +FILE *fp =NULL ; + +/* Read CDAT file and create its cache */ + fp =fopen (cdat->filename,"r" ); +if (!fp) { +error_setg (errp,"CDAT: Unable to open file" ); +return ; + } + +fseek (fp,0 ,SEEK_END ); + file_size =ftell (fp); +fseek (fp,0 ,SEEK_SET ); + cdat->buf =g_malloc0 (file_size); + +if (fread (cdat->buf, file_size,1 , fp) ==0 ) { +error_setg (errp,"CDAT: File read failed" ); +return ; + } + +fclose (fp); + +if (file_size <sizeof (CDATTableHeader)) { +error_setg (errp,"CDAT: File too short" ); +return ; + } + i =sizeof (CDATTableHeader); + num_ent =1 ; +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + i += hdr->length; + num_ent++; + } +if (i != file_size) { +error_setg (errp,"CDAT: File length missmatch" ); +return ; + } + + cdat_st =g_malloc0 (sizeof (*cdat_st) * num_ent); +if (!cdat_st) { +error_setg (errp,"CDAT: Failed to allocate entry array" ); +return ; + } + +/* Set CDAT header, Entry = 0 */ + cdat_st[0 ].base = cdat->buf; + cdat_st[0 ].length =sizeof (CDATTableHeader); + i =0 ; + +while (i < cdat_st[0 ].length) { + sum += cdat->buf[i++]; + } + +/* Read CDAT structures */ +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + +while (cdat->buf + i < + (uint8_t *)cdat_st[ent].base + cdat_st[ent].length) { +assert (i < file_size); + sum += cdat->buf[i++]; + } + + ent++; + } + +if (sum !=0 ) { +warn_report ("CDAT: Found checksum mismatch in %s" , cdat->filename); + } + cdat->entry_len = num_ent; + cdat->entry =g_steal_pointer (&cdat_st); +} + +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->filename) { +ct3_load_cdat (cdat, errp); + }else { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->to_update) { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +free (cdat->entry); +if (cdat->built_buf) + cdat->free_cdat_table (cdat->built_buf, cdat->built_buf_len, + cdat->private); +if (cdat->buf) +free (cdat->buf); +}
hw/cxl/meson.build +1 -0
@@ -4,6 +4,7 @@ softmmu_ss.add(when: 'CONFIG_CXL', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c', + 'cxl-cdat.c', ), if_false: files( 'cxl-host-stubs.c',
include/hw/cxl/cxl_cdat.h new file +165 -0
165 lines not on this page · raw message
include/hw/cxl/cxl_component.h +7 -0
@@ -19,6 +19,7 @@#include "qemu/range.h" #include "qemu/typedefs.h" #include "hw/register.h" +#include "qapi/error.h" enum reg_type {CXL2_DEVICE ,
@@ -184,6 +185,8 @@ typedef struct cxl_component {struct PCIDevice *pdev; }; }; + +CDATObject cdat; } CXLComponentState; void cxl_component_register_block_init(Object *obj,
@@ -220,4 +223,8 @@ static inline hwaddr cxl_decode_ig(int ig)CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp); +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate); +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp); +#endif
include/hw/cxl/cxl_device.h +3 -0
@@ -243,6 +243,9 @@ struct CXLType3Dev {AddressSpace hostmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate; + +/* DOE */ +DOECap doe_cdat; };#define TYPE_CXL_TYPE3 "cxl-type3"
include/hw/cxl/cxl_pci.h +1 -0
@@ -13,6 +13,7 @@#include "qemu/compiler.h" #include "hw/pci/pci.h" #include "hw/pci/pcie.h" +#include "hw/cxl/cxl_cdat.h" #define CXL_VENDOR_ID 0x1e98
Signature · 1 line
2.37.2
PATCH v7 4/5 hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Jonathan CameronFrom: Huai-Cheng Kuo <[email protected]>
The CDAT can be specified in two ways. One is to add ",cdat=<filename>" in "-device cxl-type3"'s command option. The file is required to provide the whole CDAT table in binary mode. The other is to use the default that provides some 'reasonable' numbers based on type of memory and size.
The DOE capability supporting CDAT is added to hw/mem/cxl_type3.c with capability offset 0x190. The config read/write to this capability range can be generated in the OS to request the CDAT data.
--
Changes since RFC:
- Break out type 3 user of library as separate patch.
- Change reported data for default to be based on the options provided for the type 3 device.
--- hw/mem/cxl_type3.c | 227 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+)
hw/mem/cxl_type3.c +227 -0
@@ -12,9 +12,218 @@#include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" +#include "sysemu/numa.h" #include "hw/cxl/cxl.h" #include "hw/pci/msix.h" +#define DWORD_BYTE 4 + +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, +void *priv) +{ + g_autofree CDATDsmas *dsmas_nonvolatile = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile = NULL; + g_autofree CDATDsemts *dsemts_nonvolatile = NULL; +CXLType3Dev *ct3d = priv; +int len =0 ; +int i =0 ; +int next_dsmad_handle =0 ; +int nonvolatile_dsmad =-1 ; +int dslbis_nonvolatile_num =4 ; +MemoryRegion *mr; + +/* Non volatile aspects */ +if (ct3d->hostmem) { + dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); +if (!dsmas_nonvolatile) { +return -ENOMEM ; + } + nonvolatile_dsmad = next_dsmad_handle++; + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } + *dsmas_nonvolatile = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas_nonvolatile), + }, + .DSMADhandle = nonvolatile_dsmad, + .flags =CDAT_DSMAS_FLAG_NV , + .DPA_base =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; + +/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) +return -ENOMEM ; + + dslbis_nonvolatile[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis_nonvolatile[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis_nonvolatile[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis_nonvolatile[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); + *dsemts_nonvolatile = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts_nonvolatile), + }, + .DSMAS_handle = nonvolatile_dsmad, + .EFI_memory_type_attr =2 ,/* Reserved - the non volatile from DSMAS matters */ + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; + } + + *cdat_table =g_malloc0 (len *sizeof (*cdat_table)); +/* Header always at start of structure */ +if (dsmas_nonvolatile) { + (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); + } +if (dslbis_nonvolatile) { +CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); +int j; + +for (j =0 ; j < dslbis_nonvolatile_num; j++) { + (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; + } + } +if (dsemts_nonvolatile) { + (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + } + +return len; +} + +static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) +{ +int i; + +for (i =0 ; i < num; i++) { +g_free (cdat_table[i]); + } +g_free (cdat_table); +} + +static bool cxl_doe_cdat_rsp (DOECap *doe_cap) +{ +CDATObject *cdat = &CXL_TYPE3 (doe_cap->pdev)->cxl_cstate.cdat; +uint16_t ent; +void *base; +uint32_t len; +CDATReq *req =pcie_doe_get_write_mbox_ptr (doe_cap); +CDATRsp rsp; + +assert (cdat->entry_len); + +/* Discard if request length mismatched */ +if (pcie_doe_get_obj_len (req) < +DIV_ROUND_UP (sizeof (CDATReq),DWORD_BYTE )) { +return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp ) { + .header = { + .vendor_id =CXL_VENDOR_ID , + .data_obj_type =CXL_DOE_TABLE_ACCESS , + .reserved =0x0 , + .length =DIV_ROUND_UP ((sizeof (rsp) + len),DWORD_BYTE ), + }, + .rsp_code =CXL_DOE_TAB_RSP , + .table_type =CXL_DOE_TAB_TYPE_CDAT , + .entry_handle = (ent < cdat->entry_len -1 ) ? + ent +1 :CXL_DOE_TAB_ENT_MAX , + }; + +memcpy (doe_cap->read_mbox, &rsp,sizeof (rsp)); +memcpy (doe_cap->read_mbox +DIV_ROUND_UP (sizeof (rsp),DWORD_BYTE ), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + +return true; +} + +static uint32_t ct3d_config_read (PCIDevice *pci_dev,uint32_t addr,int size) +{ +CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev); +uint32_t val; + +if (pcie_doe_read_config (&ct3d->doe_cdat, addr, size, &val)) { +return val; + } + +return pci_default_read_config (pci_dev, addr, size); +} + +static void ct3d_config_write (PCIDevice *pci_dev,uint32_t addr,uint32_t val, +int size) +{ +CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev); + +pcie_doe_write_config (&ct3d->doe_cdat, addr, val, size); +pci_default_write_config (pci_dev, addr, val, size); +} +/* * Null value of all Fs suggested by IEEE RA guidelines for use of * EU, OUI and CID
@@ -140,6 +349,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)return true; } +static DOEProtocol doe_cdat_prot[] = { + {CXL_VENDOR_ID ,CXL_DOE_TABLE_ACCESS , cxl_doe_cdat_rsp }, + { } +}; +static void ct3_realize (PCIDevice *pci_dev,Error **errp) {CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev);
@@ -189,6 +403,14 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)for (i =0 ; i < msix_num; i++) {msix_vector_use (pci_dev, i); } + +/* DOE Initailization */ +pcie_doe_init (pci_dev, &ct3d->doe_cdat,0x190 , doe_cdat_prot, true,0 ); + + cxl_cstate->cdat.build_cdat_table = ct3_build_cdat_table; + cxl_cstate->cdat.free_cdat_table = ct3_free_cdat_table; + cxl_cstate->cdat.private = ct3d; +cxl_doe_cdat_init (cxl_cstate, errp); }static void ct3_exit (PCIDevice *pci_dev)
@@ -197,6 +419,7 @@ static void ct3_exit(PCIDevice *pci_dev)CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;ComponentRegisters *regs = &cxl_cstate->crb; +cxl_doe_cdat_release (cxl_cstate);g_free (regs->special_ops);address_space_destroy (&ct3d->hostmem_as); }
@@ -296,6 +519,7 @@ static Property ct3_props[] = {DEFINE_PROP_LINK ("lsa" , CXLType3Dev, lsa,TYPE_MEMORY_BACKEND , HostMemoryBackend *),DEFINE_PROP_UINT64 ("sn" , CXLType3Dev, sn,UI64_NULL ), +DEFINE_PROP_STRING ("cdat" , CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_END_OF_LIST(), };
@@ -361,6 +585,9 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->device_id =0xd93 ;/* LVF for now */ pc->revision =1 ; + pc->config_write = ct3d_config_write; + pc->config_read = ct3d_config_read; +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;
Signature · 1 line
2.37.2
PATCH v7 5/5 hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE
In reply to Jonathan CameronThis Data Object Exchange Mailbox allows software to query the latency and bandwidth between ports on the switch. For now only provide information on routes between the upstream port and each downstream port (not p2p).
--- hw/pci-bridge/cxl_upstream.c | 182 ++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_cdat.h | 1 + 2 files changed, 182 insertions(+), 1 deletion(-)
hw/pci-bridge/cxl_upstream.c +181 -1
@@ -10,11 +10,12 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "hw/qdev-properties.h" #include "hw/pci/msi.h" #include "hw/pci/pcie.h" #include "hw/pci/pcie_port.h" -#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 1 +#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 2#define CXL_UPSTREAM_PORT_MSI_OFFSET 0x70#define CXL_UPSTREAM_PORT_PCIE_CAP_OFFSET 0x90
@@ -28,6 +29,7 @@ typedef struct CXLUpstreamPort {/*< public >*/ CXLComponentState cxl_cstate; +DOECap doe_cdat; } CXLUpstreamPort;CXLComponentState *cxl_usp_to_cstate (CXLUpstreamPort *usp)
@@ -60,6 +62,9 @@ static void cxl_usp_dvsec_write_config(PCIDevice *dev, uint32_t addr,static void cxl_usp_write_config (PCIDevice *d,uint32_t address,uint32_t val,int len) { +CXLUpstreamPort *usp =CXL_USP (d); + +pcie_doe_write_config (&usp->doe_cdat, address, val, len);pci_bridge_write_config (d, address, val, len);pcie_cap_flr_write_config (d, address, val, len);pcie_aer_write_config (d, address, val, len);
@@ -67,6 +72,18 @@ static void cxl_usp_write_config(PCIDevice *d, uint32_t address,cxl_usp_dvsec_write_config (d, address, val, len); } +static uint32_t cxl_usp_read_config (PCIDevice *d,uint32_t address,int len) +{ +CXLUpstreamPort *usp =CXL_USP (d); +uint32_t val; + +if (pcie_doe_read_config (&usp->doe_cdat, address, len, &val)) { +return val; + } + +return pci_default_read_config (d, address, len); +} +static void latch_registers (CXLUpstreamPort *usp) {uint32_t *reg_state = usp->cxl_cstate.crb.cache_mem_registers;
@@ -119,6 +136,155 @@ static void build_dvsecs(CXLComponentState *cxl) REG_LOC_DVSEC_REVID, dvsec); } +static bool cxl_doe_cdat_rsp (DOECap *doe_cap) +{ +CDATObject *cdat = &CXL_USP (doe_cap->pdev)->cxl_cstate.cdat; +uint16_t ent; +void *base; +uint32_t len; +CDATReq *req =pcie_doe_get_write_mbox_ptr (doe_cap); +CDATRsp rsp; + +cxl_doe_cdat_update (&CXL_USP (doe_cap->pdev)->cxl_cstate, &error_fatal); +assert (cdat->entry_len); + +/* Discard if request length mismatched */ +if (pcie_doe_get_obj_len (req) < +DIV_ROUND_UP (sizeof (CDATReq),sizeof (uint32_t ))) { +return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp ) { + .header = { + .vendor_id =CXL_VENDOR_ID , + .data_obj_type =CXL_DOE_TABLE_ACCESS , + .reserved =0x0 , + .length =DIV_ROUND_UP ((sizeof (rsp) + len),sizeof (uint32_t )), + }, + .rsp_code =CXL_DOE_TAB_RSP , + .table_type =CXL_DOE_TAB_TYPE_CDAT , + .entry_handle = (ent < cdat->entry_len -1 ) ? + ent +1 :CXL_DOE_TAB_ENT_MAX , + }; + +memcpy (doe_cap->read_mbox, &rsp,sizeof (rsp)); +memcpy (doe_cap->read_mbox +DIV_ROUND_UP (sizeof (rsp),sizeof (uint32_t )), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + +return true; +} + +static DOEProtocol doe_cdat_prot[] = { + {CXL_VENDOR_ID ,CXL_DOE_TABLE_ACCESS , cxl_doe_cdat_rsp }, + { } +}; + +static int build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSslbis *sslbis_latency = NULL; + g_autofree CDATSslbis *sslbis_bandwidth = NULL; +CXLUpstreamPort *us =CXL_USP (priv); +PCIBus *bus = &PCI_BRIDGE (us)->sec_bus; +int devfn, sslbis_size; +int len =0 ; +int i =0 ; +int count =0 ; +uint16_t port_ids[256 ]; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; +PCIEPort *port; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } + +/* +* Whilst the PCI express spec doesn't allow anything other than +* downstream ports on this bus, let us be a little paranoid +*/ +if (!object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { +continue ; + } + + port =PCIE_PORT (d); + port_ids[count] = port->port; + count++; + } + +/* May not yet have any ports - try again later */ +if (count ==0 ) { +return 0 ; + } + + sslbis_size =sizeof (CDATSslbis) +sizeof (*sslbis_latency->sslbe) * count; + sslbis_latency =g_malloc (sslbis_size); + *sslbis_latency = (CDATSslbis ) { + .sslbis_header = { + .header = { + .type =CDAT_TYPE_SSLBIS , + .length = sslbis_size, + }, + .data_type =HMATLB_DATA_TYPE_ACCESS_LATENCY , + .entry_base_unit =10000 , + }, + }; + +for (i =0 ; i < count; i++) { + sslbis_latency->sslbe[i] = (CDATSslbe ) { + .port_x_id =CDAT_PORT_ID_USP , + .port_y_id = port_ids[i], + .latency_bandwidth =15 ,/* 150ns */ + }; + } + len++; + + sslbis_bandwidth =g_malloc (sslbis_size); + *sslbis_bandwidth = (CDATSslbis ) { + .sslbis_header = { + .header = { + .type =CDAT_TYPE_SSLBIS , + .length = sslbis_size, + }, + .data_type =HMATLB_DATA_TYPE_ACCESS_BANDWIDTH , + .entry_base_unit =1000 , + }, + }; + +for (i =0 ; i < count; i++) { + sslbis_bandwidth->sslbe[i] = (CDATSslbe ) { + .port_x_id =CDAT_PORT_ID_USP , + .port_y_id = port_ids[i], + .latency_bandwidth =16 ,/* 16 GB/s */ + }; + } + len++; + *cdat_table =g_malloc0 (len *sizeof (*cdat_table)); +/* Header always at start of structure */ + i =0 ; + (*cdat_table)[i++] =g_steal_pointer (&sslbis_latency); + (*cdat_table)[i++] =g_steal_pointer (&sslbis_bandwidth); + +return len; +} + +static void free_default_cdat_table (CDATSubHeader **cdat_table,int num, +void *priv) +{ +int i; + +for (i =0 ; i < num; i++) { +g_free (cdat_table[i]); + } +g_free (cdat_table); +} +static void cxl_usp_realize (PCIDevice *d,Error **errp) {PCIEPort *p =PCIE_PORT (d);
@@ -161,6 +327,13 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp)PCI_BASE_ADDRESS_MEM_TYPE_64 , component_bar); +pcie_doe_init (d, &usp->doe_cdat, cxl_cstate->dvsec_offset, doe_cdat_prot, true,1 ); + + cxl_cstate->cdat.build_cdat_table = build_cdat_table; + cxl_cstate->cdat.free_cdat_table = free_default_cdat_table; + cxl_cstate->cdat.private = d; +cxl_doe_cdat_init (cxl_cstate, errp); +return ; err_cap:
@@ -179,6 +352,11 @@ static void cxl_usp_exitfn(PCIDevice *d)pci_bridge_exitfn (d); } +static Property cxl_upstream_props[] = { +DEFINE_PROP_STRING ("cdat" , CXLUpstreamPort, cxl_cstate.cdat.filename), +DEFINE_PROP_END_OF_LIST () +}; +static void cxl_upstream_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);
@@ -186,6 +364,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data) k->is_bridge = true; k->config_write = cxl_usp_write_config; + k->config_read = cxl_usp_read_config; k->realize = cxl_usp_realize; k->exit = cxl_usp_exitfn; k->vendor_id =0x19e5 ;/* Huawei */
@@ -194,6 +373,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data)set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); dc->desc ="CXL Switch Upstream Port" ; dc->reset = cxl_usp_reset; +device_class_set_props (dc, cxl_upstream_props); }static const TypeInfo cxl_usp_info = {
include/hw/cxl/cxl_cdat.h +1 -0
@@ -131,6 +131,7 @@ typedef struct CDATSslbisHeader {uint64_t entry_base_unit; }QEMU_PACKED CDATSslbisHeader; +#define CDAT_PORT_ID_USP 0x100/* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ typedef struct CDATSslbe { uint16_t port_x_id;
Signature · 1 line
2.37.2
Quoted 5 lines · On Fri, 7 Oct 2022 16:21:51 +0100 Jonathan Cameron…
Whilst I have carried on Huai-Cheng Kuo's series version numbering and naming, there have been very substantial changes since v6 so I would suggest fresh review makes sense for anyone who has looked at this before. In particularly if the Avery design folks could check I haven't broken anything that would be great.
I forgot to run checkpatch on these and there is some white space that will need cleaning up and one instance of missing brackets. As that doesn't greatly affect review, I'll wait for a few days to see if there is other feedback to incorporate in v8.
Sorry for the resulting noise!
These are now available at
https://gitlab.com/jic23/qemu/-/commits/cxl-2022-10-09
along with a bunch of other CXL features:
* Compliance DOE protocol
* SPDM / CMA over DOE supprot
* ARM64 support in general.
* Various small emulation additions.
* CPMU support
I'll add a few more features to similarly named branches over the next week or so including initial support for standalone switch CCI mailboxes.
Jonathan
Quoted 82 lines
For reference v6: QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
https://lore.kernel.org/qemu-devel/[email protected]/Summary of changes:
1) Linux headers definitions for DOE are now upstream so drop that patch.
2) Add CDAT for switch upstream port.
3) Generate 'plausible' default CDAT tables when a file is not provided.
4) General refactoring to calculate the correct table sizes and allocate based on that rather than copying from a local static array.
5) Changes from earlier reviews such as matching QEMU type naming style.
6) Moved compliance and SPDM usecases to future patch sets.Sign-offs on these are complex because the patches were originally developed by Huai-Cheng Kuo, but posted by Chris Browy and then picked up by Jonathan Cameron who made substantial changes.
Huai-Cheng Kuo / Chris Browy, please confirm you are still happy to maintain this code as per the original
MAINTAINERSentry.What's here?
This series brings generic PCI Express Data Object Exchange support (DOE) DOE is defined in the PCIe Base Spec r6.0. It consists of a mailbox in PCI config space via a PCIe Extended Capability Structure.
The PCIe spec defines several protocols (including one to discover what protocols a given DOE instance supports) and other specification such as CXL define additional protocols using their own vendor IDs.In this series we make use of the DOE to support the CXL spec defined Table Access Protocol, specifically to provide access to CDAT - a table specified in a specification that is hosted by the UEFI forum and is used to provide runtime discoverability of the sort of information that would otherwise be available in firmware tables (memory types, latency and bandwidth information etc).
The Linux kernel gained support for DOE / CDAT on CXL type 3 EPs in 6.0. The version merged did not support interrupts (earlier versions did so that support in the emulation was tested a while back).
This series provides CDAT emulation for CXL switch upstream ports and CXL type 3 memory devices. Note that to exercise the switch support additional Linux kernel patches are needed.
https://lore.kernel.org/linux-cxl/[email protected]/
(I'll post a new version of that support shortly)Additional protocols will be supported by follow on patch sets:
* CXL compliance protocol.
* CMA / SPDM device attestation.
(Old version at https://gitlab.com/jic23/qemu/-/commits/cxl-next - will refresh that tree next week)Huai-Cheng Kuo (3):
hw/pci: PCIe Data Object Exchange emulation hw/cxl/cdat: CXL CDAT Data Object Exchange implementation hw/mem/cxl-type3: Add CXL CDAT Data Object ExchangeJonathan Cameron (2):
hw/mem/cxl-type3: Add MSIX support hw/pci-bridge/cxl-upstream: Add a CDAT table access DOEMAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 222 ++++++++++++++++++++ hw/cxl/meson.build | 1 + hw/mem/cxl_type3.c | 236 +++++++++++++++++++++ hw/pci-bridge/cxl_upstream.c | 182 +++++++++++++++- hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_cdat.h | 166 +++++++++++++++ include/hw/cxl/cxl_component.h | 7 + include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++ include/hw/pci/pcie_regs.h | 4 + 15 files changed, 1323 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/pci/pcie_doe.h
Hi Jonathan,
We've reviewed the patches related to DOE and everything looks good. And we are glad to maintain the code as the maintainers.
Thanks for applying the changes.
Best Regards,
Huai-Cheng Kuo
Quoted 122 lines · Jonathan Cameron
Quoted 3 lines · On Fri, 7 Oct 2022 16:21:51 +0100 Jonathan Cameron…
Whilst I have carried on Huai-Cheng Kuo's series version numbering and naming, there have been very substantial changes since v6 so I would suggest fresh review makes sense for anyone who has looked at this
before.
Quoted 2 lines
In particularly if the Avery design folks could check I haven't broken anything that would be great.
I forgot to run checkpatch on these and there is some white space that will need cleaning up and one instance of missing brackets. As that doesn't greatly affect review, I'll wait for a few days to see if there is other feedback to incorporate in v8.
Sorry for the resulting noise!
These are now available at
https://gitlab.com/jic23/qemu/-/commits/cxl-2022-10-09
along with a bunch of other CXL features:
* Compliance DOE protocol
* SPDM / CMA over DOE supprot
* ARM64 support in general.
* Various small emulation additions.
* CPMU supportI'll add a few more features to similarly named branches over the next week or so including initial support for standalone switch CCI mailboxes.
Jonathan
Quoted 3 lines
For reference v6: QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
https://lore.kernel.org/qemu-devel/[email protected]/
Quoted 11 lines
Summary of changes:
1) Linux headers definitions for DOE are now upstream so drop that patch.
2) Add CDAT for switch upstream port.
3) Generate 'plausible' default CDAT tables when a file is not provided.
4) General refactoring to calculate the correct table sizes and allocate based on that rather than copying from a local static array.
5) Changes from earlier reviews such as matching QEMU type naming style.
6) Moved compliance and SPDM usecases to future patch sets.Sign-offs on these are complex because the patches were originally
developed
Quoted 1 line
by Huai-Cheng Kuo, but posted by Chris Browy and then picked up by
Jonathan
Quoted 3 lines
Cameron who made substantial changes.
Huai-Cheng Kuo / Chris Browy, please confirm you are still happy to
maintain this
Quoted 6 lines
code as per the original
MAINTAINERSentry.What's here?
This series brings generic PCI Express Data Object Exchange support (DOE) DOE is defined in the PCIe Base Spec r6.0. It consists of a mailbox in
PCI
Quoted 12 lines
config space via a PCIe Extended Capability Structure.
The PCIe spec defines several protocols (including one to discover what protocols a given DOE instance supports) and other specification such as CXL define additional protocols using their own vendor IDs.In this series we make use of the DOE to support the CXL spec defined Table Access Protocol, specifically to provide access to CDAT - a table specified in a specification that is hosted by the UEFI forum and is used to provide runtime discoverability of the sort of information that would otherwise be available in firmware tables (memory types, latency and bandwidth information etc).
Quoted 8 lines
The Linux kernel gained support for DOE / CDAT on CXL type 3 EPs in 6.0. The version merged did not support interrupts (earlier versions did so that support in the emulation was tested a while back).
This series provides CDAT emulation for CXL switch upstream ports and CXL type 3 memory devices. Note that to exercise the switch support additional Linux kernel patches are needed.
https://lore.kernel.org/linux-cxl/[email protected]/
Quoted 6 lines
(I'll post a new version of that support shortly)
Additional protocols will be supported by follow on patch sets:
* CXL compliance protocol.
* CMA / SPDM device attestation.
(Old version at https://gitlab.com/jic23/qemu/-/commits/cxl-next - willrefresh
Quoted 32 lines
that tree next week)
Huai-Cheng Kuo (3):
hw/pci: PCIe Data Object Exchange emulation hw/cxl/cdat: CXL CDAT Data Object Exchange implementation hw/mem/cxl-type3: Add CXL CDAT Data Object ExchangeJonathan Cameron (2):
hw/mem/cxl-type3: Add MSIX support hw/pci-bridge/cxl-upstream: Add a CDAT table access DOEMAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 222 ++++++++++++++++++++ hw/cxl/meson.build | 1 + hw/mem/cxl_type3.c | 236 +++++++++++++++++++++ hw/pci-bridge/cxl_upstream.c | 182 +++++++++++++++- hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_cdat.h | 166 +++++++++++++++ include/hw/cxl/cxl_component.h | 7 + include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++ include/hw/pci/pcie_regs.h | 4 + 15 files changed, 1323 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/pci/pcie_doe.h
cover Multi-Region and Volatile Memory support for CXL Type-3 Devices
In reply to Jonathan CameronSummary of Changes:
1) Correction of PCI_CLASS from STORAGE_EXPRESS to MEMORY_CXL on init
2) Add CXL_CAPACITY_MULTIPLIER definition to replace magic numbers
3) Refactor CDAT DSMAS Initialization for multi-region initialization
4) Multi-Region and Volatile Memory support for CXL Type-3 Devices
5) Test and Documentation updates
Developed with input from Jonathan Cameron and Davidloh Bueso.
This series brings 2 features to CXL Type-3 Devices:
1) Volatile Memory Region support
2) Multi-Region support (1 Volatile, 1 Persistent)
In this series we implement multi-region and volatile region support through 6 major changes to CXL devices
1) The HostMemoryBackend [hostmem] has been replaced by two [hostvmem] and [hostpmem] to store volatile and persistent memory respectively
2) The single AddressSpace has been replaced by two AddressSpaces [hostvmem_as] and [hostpmem_as] to map respective memdevs.
3) Each memory region size and total region are stored separately
4) The CDAT and DVSEC memory map entries have been updated:
a) if vmem is present, vmem is mapped at DPA(0)
b) if pmem is present
i) and vmem is present, pmem is mapped at DPA(vmem->size) ii) else, pmem is mapped at DPA(0)
c) partitioning of pmem is not supported in this patch set but has been discussed and this design should suffice.
5) Read/Write functions have been updated to access AddressSpaces according to the mapping described in #4
6) cxl-mailbox has been updated to report the respective size of volatile and persistent memory regions
CXL Spec (3.0) Section 8.2.9.8.2.0 - Get Partition Info
Active Volatile Memory
The device shall provide this volatile capacity starting at DPA 0
Active Persistent Memory
The device shall provide this persistent capacity starting at the DPA immediately following the volatile capacity
Partitioning of Persistent Memory regions may be supported on following patch sets.
Submitted as an extention to the CDAT emulation because the CDAT DSMAS entry concerns memory mapping and is required to successfully map memory regions correctly in bios/efi.
Gregory Price (5):
hw/cxl: set cxl-type3 device type to PCI_CLASS_MEMORY_CXL hw/cxl: Add CXL_CAPACITY_MULTIPLIER definition hw/mem/cxl_type: Generalize CDATDsmas initialization for Memory
Regions
hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
cxl: update tests and documentation for new cxl properties
docs/system/devices/cxl.rst | 53 ++++- hw/cxl/cxl-mailbox-utils.c | 23 +- hw/mem/cxl_type3.c | 449 +++++++++++++++++++++++------------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/cxl-test.c | 81 ++++++- 5 files changed, 416 insertions(+), 201 deletions(-)
Signature · 1 line
2.37.3
PATCH 1/5 hw/cxl: set cxl-type3 device type to PCI_CLASS_MEMORY_CXL
Current code sets to STORAGE_EXPRESS and then overrides it.
--- hw/mem/cxl_type3.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
hw/mem/cxl_type3.c +1 -2
@@ -535,7 +535,6 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) }pci_config_set_prog_interface (pci_conf,0x10 ); -pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL );pcie_endpoint_cap_init (pci_dev,0x80 );if (ct3d->sn !=UI64_NULL ) {
@@ -763,7 +762,7 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->config_read = ct3d_config_read; pc->realize = ct3_realize; pc->exit = ct3_exit; - pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->class_id =PCI_CLASS_MEMORY_CXL ; pc->vendor_id =PCI_VENDOR_ID_INTEL ; pc->device_id =0xd93 ;/* LVF for now */ pc->revision =1 ;
Signature · 1 line
2.37.3
PATCH 2/5 hw/cxl: Add CXL_CAPACITY_MULTIPLIER definition
In reply to Gregory PriceRemove usage of magic numbers when accessing capacity fields and replace with CXL_CAPACITY_MULTIPLIER, matching the kernel definition.
--- hw/cxl/cxl-mailbox-utils.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
hw/cxl/cxl-mailbox-utils.c +8 -6
@@ -14,6 +14,8 @@#include "qemu/log.h" #include "qemu/uuid.h" +#define CXL_CAPACITY_MULTIPLIER 0x10000000/* SZ_256M */ +/* * How to add a new command, example. The command set FOO, with cmd BAR. * 1. Add the command set and cmd to the enum.
@@ -140,7 +142,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, }QEMU_PACKED *fw_info;QEMU_BUILD_BUG_ON (sizeof (*fw_info) !=0x50 ); -if (cxl_dstate->pmem_size < (256 <<20 )) { +if (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER ) {return CXL_MBOX_INTERNAL_ERROR ; }
@@ -285,7 +287,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,CXLType3Class *cvc =CXL_TYPE3_GET_CLASS (ct3d);uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) {return CXL_MBOX_INTERNAL_ERROR ; }
@@ -295,8 +297,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,/* PMEM only */ snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); - id->total_capacity = size / (256 <<20 ); - id->persistent_capacity = size / (256 <<20 ); + id->total_capacity = size /CXL_CAPACITY_MULTIPLIER ; + id->persistent_capacity = size /CXL_CAPACITY_MULTIPLIER ; id->lsa_size = cvc->get_lsa_size (ct3d); id->poison_list_max_mer[1 ] =0x1 ;/* 256 poison records */
@@ -317,14 +319,14 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,QEMU_BUILD_BUG_ON (sizeof (*part_info) !=0x20 );uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) {return CXL_MBOX_INTERNAL_ERROR ; }/* PMEM only */ part_info->active_vmem =0 ; part_info->next_vmem =0 ; - part_info->active_pmem = size / (256 <<20 ); + part_info->active_pmem = size /CXL_CAPACITY_MULTIPLIER ; part_info->next_pmem =0 ; *len =sizeof (*part_info);
Signature · 1 line
2.37.3
PATCH 3/5 hw/mem/cxl_type: Generalize CDATDsmas initialization for Memory Regions
In reply to Gregory PriceThis is a preparatory commit for enabling multiple memory regions within a single CXL Type-3 device. We will need to initialize multiple CDAT DSMAS regions (and subsequent DSLBIS, and DSEMTS entries), so generalize the intialization into a function.
--- hw/mem/cxl_type3.c | 275 +++++++++++++++++++++++++-------------------- 1 file changed, 154 insertions(+), 121 deletions(-)
hw/mem/cxl_type3.c +154 -121
@@ -24,145 +24,178 @@#define UI64_NULL ~(0ULL)#define DWORD_BYTE 4 +static int ct3_build_dsmas (CDATDsmas *dsmas, +CDATDslbis *dslbis, +CDATDsemts *dsemts, +MemoryRegion *mr, +int dsmad_handle, +bool is_pmem, +uint64_t dpa_base) +{ +int len =0 ; +/* ttl_len should be incremented for every entry */ + +/* Device Scoped Memory Affinity Structure */ + *dsmas = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas), + }, + .DSMADhandle = dsmad_handle, + .flags = (is_pmem ?CDAT_DSMAS_FLAG_NV :0 ), + .DPA_base = dpa_base, + .DPA_length =int128_get64 (mr->size), + }; + len++; + +/* For now, no memory side cache, plausiblish numbers */ + dslbis[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + *dsemts = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts), + }, + .DSMAS_handle = dsmad_handle, +/* EFI_MEMORY_NV implies EfiReservedMemoryType */ + .EFI_memory_type_attr = is_pmem ?2 :0 , +/* Reserved - the non volatile from DSMAS matters */ + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; +return len; +} +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) { - g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile = NULL; - g_autofree CDATDsemts *dsemts_nonvolatile = NULL; + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis = NULL; + g_autofree CDATDsemts *dsemts = NULL;CXLType3Dev *ct3d = priv; -int len =0 ; -int i =0 ; -int next_dsmad_handle =0 ; -int nonvolatile_dsmad =-1 ; -int dslbis_nonvolatile_num =4 ; +int cdat_len =0 ; +int cdat_idx =0 , sub_idx =0 ; +int dsmas_num, dslbis_num, dsemts_num; +int dsmad_handle =0 ; +uint64_t dpa_base =0 ;MemoryRegion *mr; -/* Non volatile aspects */ -if (ct3d->hostmem) { - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { -return -ENOMEM ; - } - nonvolatile_dsmad = next_dsmad_handle++; - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - *dsmas_nonvolatile = (CDATDsmas ) { - .header = { - .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), - }, - .DSMADhandle = nonvolatile_dsmad, - .flags =CDAT_DSMAS_FLAG_NV , - .DPA_base =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; - -/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) -return -ENOMEM ; - - dslbis_nonvolatile[0 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_LATENCY , - .entry_base_unit =10000 ,/* 10ns base */ - .entry[0 ] =15 ,/* 150ns */ - }; - len++; - - dslbis_nonvolatile[1 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_LATENCY , - .entry_base_unit =10000 , - .entry[0 ] =25 ,/* 250ns */ - }; - len++; - - dslbis_nonvolatile[2 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - dslbis_nonvolatile[3 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - *dsemts_nonvolatile = (CDATDsemts ) { - .header = { - .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), - }, - .DSMAS_handle = nonvolatile_dsmad, - .EFI_memory_type_attr =2 ,/* Reserved - the non volatile from DSMAS matters */ - .DPA_offset =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; +if (!ct3d->hostmem | !host_memory_backend_get_memory (ct3d->hostmem)) { +return -EINVAL ; + } + + dsmas_num =1 ; + dslbis_num =4 * dsmas_num; + dsemts_num = dsmas_num; + + dsmas =g_malloc (sizeof (*dsmas) * dsmas_num); + dslbis =g_malloc (sizeof (*dslbis) * dslbis_num); + dsemts =g_malloc (sizeof (*dsemts) * dsemts_num); + +if (!dsmas || !dslbis || !dsemts) { +return -ENOMEM ; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); + cdat_len +=ct3_build_dsmas (&dsmas[dsmad_handle], + &dslbis[4 * dsmad_handle], + &dsemts[dsmad_handle], + mr, + dsmad_handle, + false, + dpa_base); + dpa_base += mr->size; + dsmad_handle++; + +/* Allocate and fill in the CDAT table */ + *cdat_table =g_malloc0 (cdat_len *sizeof (*cdat_table)); +if (!*cdat_table) { +return -ENOMEM ; } - *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ -if (dsmas_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); +CDATDsmas *dsmas_ent =g_steal_pointer (&dsmas); +for (sub_idx =0 ; sub_idx < dsmas_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dsmas_ent[sub_idx]; } -if (dslbis_nonvolatile) { -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); -int j; -for (j =0 ; j < dslbis_nonvolatile_num; j++) { - (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } +CDATDslbis *dslbis_ent =g_steal_pointer (&dslbis); +for (sub_idx =0 ; sub_idx < dslbis_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dslbis_ent[sub_idx]; } -if (dsemts_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + +CDATDsemts *dsemts_ent =g_steal_pointer (&dsemts); +for (sub_idx =0 ; sub_idx < dsemts_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dsemts_ent[sub_idx]; } - -return len; + +return cdat_len; }static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) { -int i; +int dsmas_num =1 ; +int dslbis_idx = dsmas_num; +int dsemts_idx = dsmas_num + (dsmas_num *4 ); + +/* There are only 3 sub-tables to free: dsmas, dslbis, dsemts */ +assert (num == (dsmas_num + (dsmas_num *4 ) + (dsmas_num))); + +g_free (cdat_table[0 ]); +g_free (cdat_table[dslbis_idx]); +g_free (cdat_table[dsemts_idx]); -for (i =0 ; i < num; i++) { -g_free (cdat_table[i]); - }g_free (cdat_table); }
Signature · 1 line
2.37.3
PATCH 4/5 hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
In reply to Gregory PriceThis commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region.
Two new properties have been added to cxl-type3 device initialization:
[volatile-memdev] and [persistent-memdev]
The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property.
Partitioning volatile memory from persistent memory is not yet supported.
Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info.
--- hw/cxl/cxl-mailbox-utils.c | 21 ++-- hw/mem/cxl_type3.c | 197 ++++++++++++++++++++++++++---------- include/hw/cxl/cxl_device.h | 11 +- 3 files changed, 162 insertions(+), 67 deletions(-)
hw/cxl/cxl-mailbox-utils.c +10 -11
@@ -142,7 +142,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, }QEMU_PACKED *fw_info;QEMU_BUILD_BUG_ON (sizeof (*fw_info) !=0x50 ); -if (cxl_dstate->pmem_size <CXL_CAPACITY_MULTIPLIER ) { +if (cxl_dstate->mem_size <CXL_CAPACITY_MULTIPLIER ) {return CXL_MBOX_INTERNAL_ERROR ; }
@@ -285,20 +285,20 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate);CXLType3Class *cvc =CXL_TYPE3_GET_CLASS (ct3d); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } id = (void *)cmd->payload;memset (id,0 ,sizeof (*id)); -/* PMEM only */ snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); - id->total_capacity = size /CXL_CAPACITY_MULTIPLIER ; - id->persistent_capacity = size /CXL_CAPACITY_MULTIPLIER ; + id->total_capacity = cxl_dstate->mem_size /CXL_CAPACITY_MULTIPLIER ; + id->persistent_capacity = cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ; + id->volatile_capacity = cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ; id->lsa_size = cvc->get_lsa_size (ct3d); id->poison_list_max_mer[1 ] =0x1 ;/* 256 poison records */
@@ -317,16 +317,15 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,uint64_t next_pmem; }QEMU_PACKED *part_info = (void *)cmd->payload;QEMU_BUILD_BUG_ON (sizeof (*part_info) !=0x20 ); -uint64_t size = cxl_dstate->pmem_size; -if (!QEMU_IS_ALIGNED (size,CXL_CAPACITY_MULTIPLIER )) { +if ((!QEMU_IS_ALIGNED (cxl_dstate->vmem_size,CXL_CAPACITY_MULTIPLIER )) || + (!QEMU_IS_ALIGNED (cxl_dstate->pmem_size,CXL_CAPACITY_MULTIPLIER ))) {return CXL_MBOX_INTERNAL_ERROR ; } -/* PMEM only */ - part_info->active_vmem =0 ; + part_info->active_vmem = cxl_dstate->vmem_size /CXL_CAPACITY_MULTIPLIER ; part_info->next_vmem =0 ; - part_info->active_pmem = size /CXL_CAPACITY_MULTIPLIER ; + part_info->active_pmem = cxl_dstate->pmem_size /CXL_CAPACITY_MULTIPLIER ; part_info->next_pmem =0 ; *len =sizeof (*part_info);
hw/mem/cxl_type3.c +144 -53
293 lines not on this page · raw message
include/hw/cxl/cxl_device.h +8 -3
@@ -141,8 +141,10 @@ typedef struct cxl_device_state {uint64_t host_set; } timestamp; -/* memory region for persistent memory, HDM */ +/* memory region size, HDM */ +uint64_t mem_size;uint64_t pmem_size; +uint64_t vmem_size;/* Move me later */ CPMUState cpmu[CXL_NUM_CPMU_INSTANCES ];
@@ -270,12 +272,15 @@ struct CXLType3Dev {PCIDevice parent_obj;/* Properties */ -HostMemoryBackend *hostmem; +HostMemoryBackend *hostmem;/* deprecated */ +HostMemoryBackend *hostvmem; +HostMemoryBackend *hostpmem;HostMemoryBackend *lsa;uint64_t sn;/* State */ -AddressSpace hostmem_as; +AddressSpace hostvmem_as; +AddressSpace hostpmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate;
Signature · 1 line
2.37.3
PATCH 5/5 cxl: update tests and documentation for new cxl properties
In reply to Gregory PriceAdds explicit examples for --persistent-memdev and --volatile-memdev
--- docs/system/devices/cxl.rst | 53 ++++++++++++++++++------ tests/qtest/cxl-test.c | 81 +++++++++++++++++++++++++++++++------ 2 files changed, 110 insertions(+), 24 deletions(-)
docs/system/devices/cxl.rst +41 -12
@@ -300,15 +300,36 @@ Example topology involving a switch:: Example command lines --------------------- -A very simple setup with just one directly attached CXL Type 3 device:: +A very simple setup with just one directly attached CXL Type 3 Persistent Memory device:: qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ ... - -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \ - -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -object memory-backend-file,pmem=true,id=pmem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \ + -object memory-backend-file,pmem=true,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,persistent-memdev=pmem0,lsa=cxl-lsa1,id=cxl-pmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +A very simple setup with just one directly attached CXL Type 3 Volatile Memory device:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +The same volatile setup may optionally include an LSA region:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way
@@ -328,13 +349,13 @@ the CXL Type3 device directly attached (no switches).:: -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ - -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ + -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ - -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ + -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ - -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ + -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave::
@@ -354,15 +375,23 @@ An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ -device cxl-upstream,bus=root_port0,id=us0 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ - -device cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ + -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ - -device cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ + -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ - -device cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ + -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ - -device cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ + -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k +Deprecations +------------ + +The Type 3 device [memdev] attribute has been deprecated in favor +of the [persistent-memdev] and [volatile-memdev] attributes. [memdev] +will default to a persistent memory device for backward compatibility +and is incapable of being used in combination with [persistent-memdev]. + Kernel Configuration Options ----------------------------
tests/qtest/cxl-test.c +69 -12
119 lines not on this page · raw message
Signature · 1 line
2.37.3
Re: [PATCH 0/5] Multi-Region and Volatile Memory support for CXL Type-3 Devices
In reply to Gregory PriceQuoted 40 lines · Gregory Price
Summary of Changes:
1) Correction ofPCI_CLASSfromSTORAGE_EXPRESStoMEMORY_CXLon init
2) AddCXL_CAPACITY_MULTIPLIERdefinition to replace magic numbers
3) Refactor CDAT DSMAS Initialization for multi-region initialization
4) Multi-Region and Volatile Memory support for CXL Type-3 Devices
5) Test and Documentation updatesDeveloped with input from Jonathan Cameron and Davidloh Bueso.
This series brings 2 features to CXL Type-3 Devices:
1) Volatile Memory Region support
2) Multi-Region support (1 Volatile, 1 Persistent)In this series we implement multi-region and volatile region support through 6 major changes to CXL devices
1) The HostMemoryBackend [hostmem] has been replaced by two [hostvmem] and [hostpmem] to store volatile and persistent memory respectively
2) The single AddressSpace has been replaced by two AddressSpaces [hostvmem_as] and [hostpmem_as] to map respective memdevs.
3) Each memory region size and total region are stored separately
4) The CDAT and DVSEC memory map entries have been updated:
a) if vmem is present, vmem is mapped at DPA(0)
b) if pmem is present
i) and vmem is present, pmem is mapped atDPA(vmem->size)ii) else, pmem is mapped at DPA(0)
c) partitioning of pmem is not supported in this patch set but has been discussed and this design should suffice.
5) Read/Write functions have been updated to access AddressSpaces according to the mapping described in #4
6) cxl-mailbox has been updated to report the respective size of volatile and persistent memory regionsCXL Spec (3.0) Section 8.2.9.8.2.0 - Get Partition Info
Active Volatile Memory
The device shall provide this volatile capacity starting at DPA 0
Active Persistent Memory
The device shall provide this persistent capacity starting at the DPA immediately following the volatile capacity
Quoted 6 lines
Partitioning of Persistent Memory regions may be supported on following patch sets.
Submitted as an extention to the CDAT emulation because the CDAT DSMAS entry concerns memory mapping and is required to successfully map memory regions correctly in bios/efi.
As there will be v8 of CDAT patches I expect there will be a rebase of this patchset too.
Quoted 17 lines
Gregory Price (5):
hw/cxl: set cxl-type3 device type toPCI_CLASS_MEMORY_CXLhw/cxl: AddCXL_CAPACITY_MULTIPLIERdefinition hw/mem/cxl_type: Generalize CDATDsmas initialization for Memory
Regions
hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent)
cxl: update tests and documentation for new cxl propertiesdocs/system/devices/cxl.rst | 53 ++++- hw/cxl/cxl-mailbox-utils.c | 23 +- hw/mem/cxl_type3.c | 449 +++++++++++++++++++++++------------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/cxl-test.c | 81 ++++++- 5 files changed, 416 insertions(+), 201 deletions(-)Signature · 1 line
2.37.3
Re: [PATCH 3/5] hw/mem/cxl_type: Generalize CDATDsmas initialization for Memory Regions
In reply to Gregory PriceQuoted 6 lines · On Tue, 11 Oct 2022 17:19:14 -0400 Gregory Price <[email protected]>…
This is a preparatory commit for enabling multiple memory regions within a single CXL Type-3 device. We will need to initialize multiple CDAT DSMAS regions (and subsequent DSLBIS, and DSEMTS entries), so generalize the intialization into a function.
Signed-off-by Gregory Price <[email protected]>
Hi Gregory,
Main comment here is that DOE isn't in yet. Some of the changes you have made in here should be review comments on that series rather than here.
I'm also not keen on taking the various allocations to arrays, particularly when seeing the hacky result in the free routine.
Just spin some more pointers and 3 more allocations (once persistent is added).
Jonathan
Quoted 9 lines
--- hw/mem/cxl_type3.c | 275 +++++++++++++++++++++++++-------------------- 1 file changed, 154 insertions(+), 121 deletions(-)hw/mem/cxl_type3.c +0
-0
hw/mem/cxl_type3.c line 33
Quoted 10 lines
hw/mem/cxl_type3.c +7
-0@@ -24,145 +24,178 @@#define UI64_NULL ~(0ULL)#define DWORD_BYTE 4 +static int ct3_build_dsmas (CDATDsmas *dsmas, +CDATDslbis *dslbis, +CDATDsemts *dsemts, +MemoryRegion *mr, +int dsmad_handle, +bool is_pmem, +uint64_t dpa_base)
Rewrap this to be just under 80 characters per line.
This is building a lot more than DSMAS.
Could rename it, or could break it down into functions that deal with each type of entry.
hw/mem/cxl_type3.c line 36
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+{ +int len =0 ; +/* ttl_len should be incremented for every entry */
ttl_ ?
Given you now allocate outside of this function, probably more appropriate to just add the entries up there.
Quoted 84 lines
hw/mem/cxl_type3.c +83
-0+ +/* Device Scoped Memory Affinity Structure */ + *dsmas = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas), + }, + .DSMADhandle = dsmad_handle, + .flags = (is_pmem ?CDAT_DSMAS_FLAG_NV :0 ), + .DPA_base = dpa_base, + .DPA_length =int128_get64 (mr->size), + }; + len++; + +/* For now, no memory side cache, plausiblish numbers */ + dslbis[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + *dsemts = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts), + }, + .DSMAS_handle = dsmad_handle, +/* EFI_MEMORY_NV implies EfiReservedMemoryType */ + .EFI_memory_type_attr = is_pmem ?2 :0 , +/* Reserved - the non volatile from DSMAS matters */ + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; +return len; +} + static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,
hw/mem/cxl_type3.c line 34
Quoted 10 lines
hw/mem/cxl_type3.c +3
-4void *priv) { - g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile = NULL; - g_autofree CDATDsemts *dsemts_nonvolatile = NULL; + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis = NULL; + g_autofree CDATDsemts *dsemts = NULL;CXLType3Dev *ct3d = priv; -int len =0 ;
There are changes in here that aren't necessary and just result in a much harder diff to review. Why rename len to cdat_len?
Quoted 98 lines
hw/mem/cxl_type3.c +5
-91-int i =0 ; -int next_dsmad_handle =0 ; -int nonvolatile_dsmad =-1 ; -int dslbis_nonvolatile_num =4 ; +int cdat_len =0 ; +int cdat_idx =0 , sub_idx =0 ; +int dsmas_num, dslbis_num, dsemts_num; +int dsmad_handle =0 ; +uint64_t dpa_base =0 ;MemoryRegion *mr; -/* Non volatile aspects */ -if (ct3d->hostmem) { - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { -return -ENOMEM ; - } - nonvolatile_dsmad = next_dsmad_handle++; - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - *dsmas_nonvolatile = (CDATDsmas ) { - .header = { - .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), - }, - .DSMADhandle = nonvolatile_dsmad, - .flags =CDAT_DSMAS_FLAG_NV , - .DPA_base =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; - -/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) -return -ENOMEM ; - - dslbis_nonvolatile[0 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_LATENCY , - .entry_base_unit =10000 ,/* 10ns base */ - .entry[0 ] =15 ,/* 150ns */ - }; - len++; - - dslbis_nonvolatile[1 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_LATENCY , - .entry_base_unit =10000 , - .entry[0 ] =25 ,/* 250ns */ - }; - len++; - - dslbis_nonvolatile[2 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - dslbis_nonvolatile[3 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - *dsemts_nonvolatile = (CDATDsemts ) { - .header = {
hw/mem/cxl_type3.c line 134
Quoted 10 lines
hw/mem/cxl_type3.c +1
-9- .type = CDAT_TYPE_DSEMTS, - .length = sizeof(*dsemts_nonvolatile), - }, - .DSMAS_handle = nonvolatile_dsmad, - .EFI_memory_type_attr = 2, /* Reserved - the non volatile from DSMAS matters */ - .DPA_offset = 0, - .DPA_length = int128_get64(mr->size), - }; - len++; +if (!ct3d->hostmem | !host_memory_backend_get_memory (ct3d->hostmem)) {
I don't immediately see why we need this test here and didn't previously. If it should always have been there, put that as a review on the DOE patches not here.
hw/mem/cxl_type3.c line 142
Quoted 8 lines
hw/mem/cxl_type3.c +8
-0+return -EINVAL ; + } + + dsmas_num =1 ; + dslbis_num =4 * dsmas_num; + dsemts_num = dsmas_num; + + dsmas =g_malloc (sizeof (*dsmas) * dsmas_num);
As we aren't likely to add any more entries after non volatile I'm not convinced making everything arrays then indexing into them is worth while, particularly as it's causing huge amounts of churn.
If this style of preallocate then fill makes more sense (I don't particularly like it breaks up the handling of each element), then propose that in review of the original series. Having this level of 'style' of function change here makes for a hard to read set. We can still change the original patch. I'm not yet convinced it's worth making that change.
I think I'd rather see the allocation and fill all in the factored out function.
hw/mem/cxl_type3.c line 151
Quoted 9 lines
hw/mem/cxl_type3.c +9
-0+ dslbis =g_malloc (sizeof (*dslbis) * dslbis_num); + dsemts =g_malloc (sizeof (*dsemts) * dsemts_num); + +if (!dsmas || !dslbis || !dsemts) { +return -ENOMEM ; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); + cdat_len += ct3_build_dsmas(&dsmas[dsmad_handle],
There isn't a specific connection between dsmad_handle. This code kind of makes it look like it's not just that we've decided to use handle 0 and later 1.
That's another reason I'd rather not do this with arrays.
Quoted 6 lines
hw/mem/cxl_type3.c +6
-0+ &dslbis[4 * dsmad_handle], + &dsemts[dsmad_handle], + mr, + dsmad_handle, + false, + dpa_base);
hw/mem/cxl_type3.c line 139
Quoted 10 lines
hw/mem/cxl_type3.c +7
-1+ dpa_base += mr->size; + dsmad_handle++; + +/* Allocate and fill in the CDAT table */ + *cdat_table =g_malloc0 (cdat_len *sizeof (*cdat_table)); +if (!*cdat_table) { +return -ENOMEM ; } - *cdat_table =g_malloc0 (len *sizeof (*cdat_table));
Looks like I'm missing an allocation failure check here in original code. Please put that in a review of that series rather than introducing the change hidden down in here.
hw/mem/cxl_type3.c line 168
Quoted 4 lines
hw/mem/cxl_type3.c +1
-2/* Header always at start of structure */ -if (dsmas_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); +CDATDsmas *dsmas_ent =g_steal_pointer (&dsmas);
We should not be introducing new local variables down here. Style wise stick to old school C style of all definitions at the top or within a scoped region {}.
hw/mem/cxl_type3.c line 170
Quoted 2 lines
hw/mem/cxl_type3.c +2
-0+for (sub_idx =0 ; sub_idx < dsmas_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dsmas_ent[sub_idx];
for a local index j is fine.
Using a more specific name for what was i is fair enough. Belongs in review of original patch given that hasn't been accepted yet.
Quoted 24 lines
hw/mem/cxl_type3.c +9
-10} -if (dslbis_nonvolatile) { -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); -int j; -for (j =0 ; j < dslbis_nonvolatile_num; j++) { - (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } +CDATDslbis *dslbis_ent =g_steal_pointer (&dslbis); +for (sub_idx =0 ; sub_idx < dslbis_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dslbis_ent[sub_idx]; } -if (dsemts_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + +CDATDsemts *dsemts_ent =g_steal_pointer (&dsemts); +for (sub_idx =0 ; sub_idx < dsemts_num; sub_idx++) { + (*cdat_table)[cdat_idx++] = (CDATSubHeader *)&dsemts_ent[sub_idx]; } - -return len; + +return cdat_len; }
hw/mem/cxl_type3.c line 193
Quoted 10 lines
hw/mem/cxl_type3.c +6
-1static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) { -int i; +int dsmas_num =1 ; +int dslbis_idx = dsmas_num; +int dsemts_idx = dsmas_num + (dsmas_num *4 ); + +/* There are only 3 sub-tables to free: dsmas, dslbis, dsemts */ +assert (num == (dsmas_num + (dsmas_num *4 ) + (dsmas_num)));
This alone is a very good reason not to do allocations as arrays. It looks extremely fragile to me. Lets just pay the cost of a few more small allocations to keep the code easier to maintain.
Quoted 11 lines
hw/mem/cxl_type3.c +4
-3+ +g_free (cdat_table[0 ]); +g_free (cdat_table[dslbis_idx]); +g_free (cdat_table[dsemts_idx]); -for (i =0 ; i < num; i++) { -g_free (cdat_table[i]); - }g_free (cdat_table); }
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Jonathan CameronThis code contains heap corruption on free, and I think should be refactored to pre-allocate all the entries we're interested in putting into the table. This would flatten the code and simplify the error handling steps.
Also, should we consider making a union with all the possible entries to make entry allocation easier? It may eat a few extra bytes of memory, but it would simplify the allocation/cleanup code here further.
Given that every allocation has to be checked, i'm also not convinced the use of g_autofree is worth the potential footguns associated with it.
Quoted 8 lines
hw/mem/cxl_type3.c +3
-0@@ -12,9 +12,218 @@ +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, +void *priv) +{
(snip)
hw/mem/cxl_type3.c line 61
Quoted 4 lines
hw/mem/cxl_type3.c +4
-0+/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) +return -ENOMEM ;
this allocation creates a table of entries, which is later freed incorrectly
hw/mem/cxl_type3.c
Quoted 2 lines
hw/mem/cxl_type3.c +2
-0+ + *cdat_table =g_malloc0 (len *sizeof (*cdat_table));
this allocation needs to be checked
hw/mem/cxl_type3.c line 139
Quoted 6 lines
hw/mem/cxl_type3.c +6
-0+/* Header always at start of structure */ +if (dsmas_nonvolatile) { + (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); + } +if (dslbis_nonvolatile) { +CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile);
using a local reference used to avoid a g_autofree footgun suggests we should not use g_autofree here, and possibly reconsider the overall strategy for allocation and cleanup
hw/mem/cxl_type3.c line 144
Quoted 5 lines
hw/mem/cxl_type3.c +5
-0+int j; + +for (j =0 ; j < dslbis_nonvolatile_num; j++) { + (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; + }
this fills the CDAT table with sub-references to the table allocated above, which leads to heap corruption with the current code, or complicated cleanup if we decide to keep it
hw/mem/cxl_type3.c line 156
Quoted 8 lines
hw/mem/cxl_type3.c +8
-0+ +return len; +} + +static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) +{ +int i; +
And here we free every entry of the table, which can/will cause heap corruption when the sub-table entries are freed
Quoted 5 lines
hw/mem/cxl_type3.c +5
-0+for (i =0 ; i < num; i++) { +g_free (cdat_table[i]); + } +g_free (cdat_table); +}
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Jonathan CameronIncluded in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.
Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
function
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)
Signature · 1 line
2.37.3
PATCH 1/5 hw/mem/cxl_type3: fix checkpatch errors
This fixes checkpatch errors in the prior commit.
--- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
hw/mem/cxl_type3.c +9 -6
@@ -56,9 +56,11 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, len++;/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) {return -ENOMEM ; + } dslbis_nonvolatile[0 ] = (CDATDslbis ) { .header = {
@@ -85,7 +87,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 25, /* 250ns */ }; len++; - + dslbis_nonvolatile[2 ] = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS ,
@@ -123,7 +125,8 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .length = sizeof(*dsemts_nonvolatile), }, .DSMAS_handle = nonvolatile_dsmad, - .EFI_memory_type_attr = 2, /* Reserved - the non volatile from DSMAS matters */ + /* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr = 2, .DPA_offset = 0, .DPA_length = int128_get64(mr->size), };
@@ -136,7 +139,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); }if (dslbis_nonvolatile) { -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); +CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile);int j;for (j =0 ; j < dslbis_nonvolatile_num; j++) {
@@ -146,7 +149,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,if (dsemts_nonvolatile) { (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); } - +return len; }
Signature · 1 line
2.37.3
PATCH 3/5 hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work
In reply to Gregory PriceMakes the size of the allocated cdat table static (6 entries), flattens the code, and reduces the number of exit conditions
--- hw/mem/cxl_type3.c | 52 ++++++++++++++++++++-------------------------- 1 file changed, 22 insertions(+), 30 deletions(-)
hw/mem/cxl_type3.c +22 -30
@@ -17,6 +17,7 @@#include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CT3_CDAT_SUBTABLE_SIZE 6static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv)
@@ -25,7 +26,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, g_autofree CDATDslbis *dslbis_nonvolatile = NULL; g_autofree CDATDsemts *dsemts_nonvolatile = NULL;CXLType3Dev *ct3d = priv; -int len =0 ;int i =0 ;int next_dsmad_handle =0 ;int nonvolatile_dsmad =-1 ;
@@ -33,7 +33,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,MemoryRegion *mr;if (!ct3d->hostmem) { -return len; +return 0 ; } mr =host_memory_backend_get_memory (ct3d->hostmem);
@@ -41,11 +41,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,return -EINVAL ; } + *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table)); +if (!*cdat_table) { +return -ENOMEM ; + } +/* Non volatile aspects */ dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); +if (!dsmas_nonvolatile || !dslbis_nonvolatile || !dsemts_nonvolatile) { +g_free (*cdat_table); + *cdat_table =NULL ;return -ENOMEM ; } + nonvolatile_dsmad = next_dsmad_handle++; *dsmas_nonvolatile = (CDATDsmas ) { .header = {
@@ -57,15 +68,8 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .DPA_base = 0, .DPA_length = int128_get64(mr->size), }; - len++;/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) { -return -ENOMEM ; - } - dslbis_nonvolatile[0 ] = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS ,
@@ -77,7 +81,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 10000, /* 10ns base */ .entry[0] = 15, /* 150ns */ }; - len++; dslbis_nonvolatile[1 ] = (CDATDslbis ) { .header = {
@@ -90,7 +93,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 10000, .entry[0] = 25, /* 250ns */ }; - len++; dslbis_nonvolatile[2 ] = (CDATDslbis ) { .header = {
@@ -103,7 +105,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 1000, /* GB/s */ .entry[0] = 16, }; - len++; dslbis_nonvolatile[3 ] = (CDATDslbis ) { .header = {
@@ -116,9 +117,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 1000, /* GB/s */ .entry[0] = 16, }; - len++; - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); *dsemts_nonvolatile = (CDATDsemts ) { .header = { .type =CDAT_TYPE_DSEMTS ,
@@ -130,26 +129,19 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .DPA_offset = 0, .DPA_length = int128_get64(mr->size), }; - len++; - *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ -if (dsmas_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); - } -if (dslbis_nonvolatile) { -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); -int j; + (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); -for (j =0 ; j < dslbis_nonvolatile_num; j++) { - (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } - } -if (dsemts_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); +CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); +int j; +for (j =0 ; j < dslbis_nonvolatile_num; j++) { + (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; } -return len; + (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + +return CT3_CDAT_SUBTABLE_SIZE ; }static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv)
Signature · 1 line
2.37.3
PATCH 2/5 hw/mem/cxl_type3: Pull validation checks ahead of functional code
In reply to Gregory PriceFor style - pulling these validations ahead flattens the code.
--- hw/mem/cxl_type3.c | 193 ++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 97 deletions(-)
hw/mem/cxl_type3.c +96 -97
@@ -32,107 +32,106 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,int dslbis_nonvolatile_num =4 ;MemoryRegion *mr; +if (!ct3d->hostmem) { +return len; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } +/* Non volatile aspects */ -if (ct3d->hostmem) { - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { -return -ENOMEM ; - } - nonvolatile_dsmad = next_dsmad_handle++; - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - *dsmas_nonvolatile = (CDATDsmas ) { - .header = { - .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), - }, - .DSMADhandle = nonvolatile_dsmad, - .flags =CDAT_DSMAS_FLAG_NV , - .DPA_base =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; - -/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) { -return -ENOMEM ; - } + dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); +if (!dsmas_nonvolatile) { +return -ENOMEM ; + } + nonvolatile_dsmad = next_dsmad_handle++; + *dsmas_nonvolatile = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas_nonvolatile), + }, + .DSMADhandle = nonvolatile_dsmad, + .flags =CDAT_DSMAS_FLAG_NV , + .DPA_base =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; - dslbis_nonvolatile[0 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_LATENCY , - .entry_base_unit =10000 ,/* 10ns base */ - .entry[0 ] =15 ,/* 150ns */ - }; - len++; - - dslbis_nonvolatile[1 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_LATENCY , - .entry_base_unit =10000 , - .entry[0 ] =25 ,/* 250ns */ - }; - len++; - - dslbis_nonvolatile[2 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - dslbis_nonvolatile[3 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - *dsemts_nonvolatile = (CDATDsemts ) { - .header = { - .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), - }, - .DSMAS_handle = nonvolatile_dsmad, -/* Reserved - the non volatile from DSMAS matters */ - .EFI_memory_type_attr =2 , - .DPA_offset =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; +/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) { +return -ENOMEM ; } + dslbis_nonvolatile[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis_nonvolatile[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis_nonvolatile[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis_nonvolatile[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); + *dsemts_nonvolatile = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts_nonvolatile), + }, + .DSMAS_handle = nonvolatile_dsmad, +/* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr =2 , + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; + *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ if (dsmas_nonvolatile) {
Signature · 1 line
2.37.3
PATCH 4/5 hw/mem/cxl_type3: Change the CDAT allocation/free strategy
In reply to Gregory PriceThe existing code allocates a subtable for SLBIS entries, uses a local variable to avoid a g_autofree footgun, and the cleanup code causes heap corruption.
Rather than allocate a table, explicitly allocate each individual entry and make the sub-table size static.
--- hw/mem/cxl_type3.c | 49 ++++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 23 deletions(-)
hw/mem/cxl_type3.c +26 -23
@@ -23,13 +23,14 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) { g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile1 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile2 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile3 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile4 = NULL; g_autofree CDATDsemts *dsemts_nonvolatile = NULL;CXLType3Dev *ct3d = priv; -int i =0 ;int next_dsmad_handle =0 ;int nonvolatile_dsmad =-1 ; -int dslbis_nonvolatile_num =4 ;MemoryRegion *mr;if (!ct3d->hostmem) {
@@ -48,10 +49,15 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,/* Non volatile aspects */ dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); + dslbis_nonvolatile1 =g_malloc (sizeof (*dslbis_nonvolatile1)); + dslbis_nonvolatile2 =g_malloc (sizeof (*dslbis_nonvolatile2)); + dslbis_nonvolatile3 =g_malloc (sizeof (*dslbis_nonvolatile3)); + dslbis_nonvolatile4 =g_malloc (sizeof (*dslbis_nonvolatile4)); dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); -if (!dsmas_nonvolatile || !dslbis_nonvolatile || !dsemts_nonvolatile) { + +if (!dsmas_nonvolatile || !dsemts_nonvolatile || + !dslbis_nonvolatile1 || !dslbis_nonvolatile2 || + !dslbis_nonvolatile3 || !dslbis_nonvolatile4) {g_free (*cdat_table); *cdat_table =NULL ;return -ENOMEM ;
@@ -70,10 +76,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile[0 ] = (CDATDslbis ) { + *dslbis_nonvolatile1 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile1), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,
@@ -82,10 +88,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 15, /* 150ns */ }; - dslbis_nonvolatile[1 ] = (CDATDslbis ) { + *dslbis_nonvolatile2 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile2), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,
@@ -94,10 +100,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 25, /* 250ns */ }; - dslbis_nonvolatile[2 ] = (CDATDslbis ) { + *dslbis_nonvolatile3 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile3), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,
@@ -106,10 +112,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 16, }; - dslbis_nonvolatile[3 ] = (CDATDslbis ) { + *dslbis_nonvolatile4 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile4), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,
@@ -131,15 +137,12 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* Header always at start of structure */ - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); - -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); -int j; -for (j =0 ; j < dslbis_nonvolatile_num; j++) { - (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } - - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + (*cdat_table)[0 ] =g_steal_pointer (&dsmas_nonvolatile); + (*cdat_table)[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile1); + (*cdat_table)[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile2); + (*cdat_table)[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile3); + (*cdat_table)[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile4); + (*cdat_table)[5 ] =g_steal_pointer (&dsemts_nonvolatile);return CT3_CDAT_SUBTABLE_SIZE ; }
Signature · 1 line
2.37.3
PATCH 5/5 hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a function
In reply to Gregory PriceThe CDAT can contain multiple entries for multiple memory regions, this will allow us to re-use the initialization code when volatile memory region support is added.
--- hw/mem/cxl_type3.c | 137 ++++++++++++++++++++++++--------------------- 1 file changed, 72 insertions(+), 65 deletions(-)
hw/mem/cxl_type3.c +72 -65
@@ -19,117 +19,93 @@#define DWORD_BYTE 4#define CT3_CDAT_SUBTABLE_SIZE 6 -static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, -void *priv) +static int ct3_build_cdat_subtable (CDATSubHeader **cdat_table, +MemoryRegion *mr,int dsmad_handle) { - g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile1 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile2 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile3 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile4 = NULL; - g_autofree CDATDsemts *dsemts_nonvolatile = NULL; -CXLType3Dev *ct3d = priv; -int next_dsmad_handle =0 ; -int nonvolatile_dsmad =-1 ; -MemoryRegion *mr; - -if (!ct3d->hostmem) { -return 0 ; - } - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - - *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table)); -if (!*cdat_table) { -return -ENOMEM ; - } - -/* Non volatile aspects */ - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); - dslbis_nonvolatile1 =g_malloc (sizeof (*dslbis_nonvolatile1)); - dslbis_nonvolatile2 =g_malloc (sizeof (*dslbis_nonvolatile2)); - dslbis_nonvolatile3 =g_malloc (sizeof (*dslbis_nonvolatile3)); - dslbis_nonvolatile4 =g_malloc (sizeof (*dslbis_nonvolatile4)); - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - -if (!dsmas_nonvolatile || !dsemts_nonvolatile || - !dslbis_nonvolatile1 || !dslbis_nonvolatile2 || - !dslbis_nonvolatile3 || !dslbis_nonvolatile4) { -g_free (*cdat_table); - *cdat_table =NULL ; + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis1 = NULL; + g_autofree CDATDslbis *dslbis2 = NULL; + g_autofree CDATDslbis *dslbis3 = NULL; + g_autofree CDATDslbis *dslbis4 = NULL; + g_autofree CDATDsemts *dsemts = NULL; + + dsmas =g_malloc (sizeof (*dsmas)); + dslbis1 =g_malloc (sizeof (*dslbis1)); + dslbis2 =g_malloc (sizeof (*dslbis2)); + dslbis3 =g_malloc (sizeof (*dslbis3)); + dslbis4 =g_malloc (sizeof (*dslbis4)); + dsemts =g_malloc (sizeof (*dsemts)); + +if (!dsmas || !dslbis1 || !dslbis2 || !dslbis3 || !dslbis4 || !dsemts) {return -ENOMEM ; } - nonvolatile_dsmad = next_dsmad_handle++; - *dsmas_nonvolatile = (CDATDsmas ) { + *dsmas = (CDATDsmas ) { .header = { .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), + .length =sizeof (*dsmas), }, - .DSMADhandle = nonvolatile_dsmad, + .DSMADhandle = dsmad_handle, .flags =CDAT_DSMAS_FLAG_NV , .DPA_base =0 , .DPA_length =int128_get64 (mr->size), };/* For now, no memory side cache, plausiblish numbers */ - *dslbis_nonvolatile1 = (CDATDslbis ) { + *dslbis1 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile1), + .length =sizeof (*dslbis1), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_READ_LATENCY , .entry_base_unit =10000 ,/* 10ns base */ .entry[0 ] =15 ,/* 150ns */ }; - *dslbis_nonvolatile2 = (CDATDslbis ) { + *dslbis2 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile2), + .length =sizeof (*dslbis2), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_WRITE_LATENCY , .entry_base_unit =10000 , .entry[0 ] =25 ,/* 250ns */ }; - *dslbis_nonvolatile3 = (CDATDslbis ) { + *dslbis3 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile3), + .length =sizeof (*dslbis3), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_READ_BANDWIDTH , .entry_base_unit =1000 ,/* GB/s */ .entry[0 ] =16 , }; - *dslbis_nonvolatile4 = (CDATDslbis ) { + *dslbis4 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile4), + .length =sizeof (*dslbis4), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , .entry_base_unit =1000 ,/* GB/s */ .entry[0 ] =16 , }; - *dsemts_nonvolatile = (CDATDsemts ) { + *dsemts = (CDATDsemts ) { .header = { .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), + .length =sizeof (*dsemts), }, - .DSMAS_handle = nonvolatile_dsmad, + .DSMAS_handle = dsmad_handle,/* Reserved - the non volatile from DSMAS matters */ .EFI_memory_type_attr =2 , .DPA_offset =0 ,
@@ -137,16 +113,47 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* Header always at start of structure */ - (*cdat_table)[0 ] =g_steal_pointer (&dsmas_nonvolatile); - (*cdat_table)[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile1); - (*cdat_table)[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile2); - (*cdat_table)[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile3); - (*cdat_table)[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile4); - (*cdat_table)[5 ] =g_steal_pointer (&dsemts_nonvolatile); + cdat_table[0 ] =g_steal_pointer (&dsmas); + cdat_table[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis1); + cdat_table[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis2); + cdat_table[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis3); + cdat_table[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis4); + cdat_table[5 ] =g_steal_pointer (&dsemts);return CT3_CDAT_SUBTABLE_SIZE ; } +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, +void *priv) +{ +CXLType3Dev *ct3d = priv; +MemoryRegion *mr; +int ret =0 ; + +if (!ct3d->hostmem) { +return 0 ; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } + + *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table)); +if (!*cdat_table) { +return -ENOMEM ; + } + +/* Non volatile aspects */ + ret =ct3_build_cdat_subtable (*cdat_table, mr,0 ); +if (ret <0 ) { +g_free (*cdat_table); + *cdat_table =NULL ; + } + +return ret; +} +static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) {int i;
Signature · 1 line
2.37.3
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Gregory PriceQuoted 25 lines · On Wed, 12 Oct 2022 14:21:15 -0400 Gregory Price <[email protected]>…
Included in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
function
Quoted 3 lines
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)
Thanks, I'm going to roll this stuff into the original patch set for v8. Some of this I already have (like the check patch stuff). Some I may disagree with in which case I'll reply to the patches - note I haven't looked at them in detail yet!
Jonathan
Re: [PATCH 2/5] hw/mem/cxl_type3: Pull validation checks ahead of functional code
In reply to Gregory PriceQuoted 1 line · On Wed, 12 Oct 2022 14:21:17 -0400 Gregory Price <[email protected]>…
For style - pulling these validations ahead flattens the code.
True, but at the cost of separating the check from where it is obvious why we have the check. I'd prefer to see it next to the use.
Inverting the hostmem check is resonable so I'll make that change.
My original thinking is that doing so would make adding non volatile support messier but given you plan to factor out most of this the change won't be too bad anyway.
Quoted 214 lines
Signed-off-by Gregory Price <[email protected]>--- hw/mem/cxl_type3.c | 193 ++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 97 deletions(-)hw/mem/cxl_type3.c +96
-97@@ -32,107 +32,106 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,int dslbis_nonvolatile_num =4 ;MemoryRegion *mr; +if (!ct3d->hostmem) { +return len; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } +/* Non volatile aspects */ -if (ct3d->hostmem) { - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { -return -ENOMEM ; - } - nonvolatile_dsmad = next_dsmad_handle++; - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - *dsmas_nonvolatile = (CDATDsmas ) { - .header = { - .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), - }, - .DSMADhandle = nonvolatile_dsmad, - .flags =CDAT_DSMAS_FLAG_NV , - .DPA_base =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; - -/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) { -return -ENOMEM ; - } + dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); +if (!dsmas_nonvolatile) { +return -ENOMEM ; + } + nonvolatile_dsmad = next_dsmad_handle++; + *dsmas_nonvolatile = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas_nonvolatile), + }, + .DSMADhandle = nonvolatile_dsmad, + .flags =CDAT_DSMAS_FLAG_NV , + .DPA_base =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; - dslbis_nonvolatile[0 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_LATENCY , - .entry_base_unit =10000 ,/* 10ns base */ - .entry[0 ] =15 ,/* 150ns */ - }; - len++; - - dslbis_nonvolatile[1 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_LATENCY , - .entry_base_unit =10000 , - .entry[0 ] =25 ,/* 250ns */ - }; - len++; - - dslbis_nonvolatile[2 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - dslbis_nonvolatile[3 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - *dsemts_nonvolatile = (CDATDsemts ) { - .header = { - .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), - }, - .DSMAS_handle = nonvolatile_dsmad, -/* Reserved - the non volatile from DSMAS matters */ - .EFI_memory_type_attr =2 , - .DPA_offset =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; +/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) { +return -ENOMEM ; } + dslbis_nonvolatile[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis_nonvolatile[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis_nonvolatile[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis_nonvolatile[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); + *dsemts_nonvolatile = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts_nonvolatile), + }, + .DSMAS_handle = nonvolatile_dsmad, +/* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr =2 , + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; + *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ if (dsmas_nonvolatile) {
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Gregory PriceQuoted 3 lines · On Wed, 12 Oct 2022 12:01:54 -0400 Gregory Price <[email protected]>…
This code contains heap corruption on free, and I think should be refactored to pre-allocate all the entries we're interested in putting into the table.
Good point on the heap corruption.. (oops. Particularly as I raised that I didn't like the complexity of your free in your previous version and still failed to notice the current code was wrong...)
Quoted 2 lines
This would flatten the code and simplify the error
handling steps.
I'm not so keen on this. Error handling is pretty trivial because of the autofree magic. It will get a tiny bit harder once we have two calls to the factored out function, but not too bad - we just need to free the handed off pointers in reverse from wherever we got to before the error.
Quoted 4 lines
Also, should we consider making a union with all the possible entries to make entry allocation easier? It may eat a few extra bytes of memory, but it would simplify the allocation/cleanup code here further.
An interesting point, though gets trickier once we have variable numbers of elements. I'm not sure it's worth the effort to save a few lines of code.
Quoted 4 lines
Given that every allocation has to be checked, i'm also not convinced the use of
g_autofreeis worth the potential footguns associated with it.
After rolling a version with some of your suggested changes incorporated the autofree logic is all nice and localized so I think it's well worth having. Only slightly messy bit is we end up with 4 separate pointers for the bandwidth and latency elements.
Re: [PATCH 2/5] hw/mem/cxl_type3: Pull validation checks ahead of functional code
In reply to Jonathan CameronQuoted 8 lines · On Thu, 13 Oct 2022 10:07:40 +0100 Jonathan Cameron…
Quoted 1 line · On Wed, 12 Oct 2022 14:21:17 -0400 Gregory Price <[email protected]>…
For style - pulling these validations ahead flattens the code.
True, but at the cost of separating the check from where it is obvious why we have the check. I'd prefer to see it next to the use.
That separation made a bit more sense after factoring out the code as then we want to pass the mr in rather than the HostMemBackend.
So in the end I did what you suggested :)
Jonathan
Quoted 224 lines
Inverting the hostmem check is resonable so I'll make that change.
My original thinking is that doing so would make adding non volatile support messier but given you plan to factor out most of this the change won't be too bad anyway.
Quoted 214 lines
Signed-off-by Gregory Price <[email protected]>--- hw/mem/cxl_type3.c | 193 ++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 97 deletions(-)hw/mem/cxl_type3.c +96
-97@@ -32,107 +32,106 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,int dslbis_nonvolatile_num =4 ;MemoryRegion *mr; +if (!ct3d->hostmem) { +return len; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } +/* Non volatile aspects */ -if (ct3d->hostmem) { - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { -return -ENOMEM ; - } - nonvolatile_dsmad = next_dsmad_handle++; - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - *dsmas_nonvolatile = (CDATDsmas ) { - .header = { - .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), - }, - .DSMADhandle = nonvolatile_dsmad, - .flags =CDAT_DSMAS_FLAG_NV , - .DPA_base =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; - -/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) { -return -ENOMEM ; - } + dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); +if (!dsmas_nonvolatile) { +return -ENOMEM ; + } + nonvolatile_dsmad = next_dsmad_handle++; + *dsmas_nonvolatile = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas_nonvolatile), + }, + .DSMADhandle = nonvolatile_dsmad, + .flags =CDAT_DSMAS_FLAG_NV , + .DPA_base =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; - dslbis_nonvolatile[0 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_LATENCY , - .entry_base_unit =10000 ,/* 10ns base */ - .entry[0 ] =15 ,/* 150ns */ - }; - len++; - - dslbis_nonvolatile[1 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_LATENCY , - .entry_base_unit =10000 , - .entry[0 ] =25 ,/* 250ns */ - }; - len++; - - dslbis_nonvolatile[2 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_READ_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - dslbis_nonvolatile[3 ] = (CDATDslbis ) { - .header = { - .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), - }, - .handle = nonvolatile_dsmad, - .flags =HMAT_LB_MEM_MEMORY , - .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , - .entry_base_unit =1000 ,/* GB/s */ - .entry[0 ] =16 , - }; - len++; - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - *dsemts_nonvolatile = (CDATDsemts ) { - .header = { - .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), - }, - .DSMAS_handle = nonvolatile_dsmad, -/* Reserved - the non volatile from DSMAS matters */ - .EFI_memory_type_attr =2 , - .DPA_offset =0 , - .DPA_length =int128_get64 (mr->size), - }; - len++; +/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) { +return -ENOMEM ; } + dslbis_nonvolatile[0 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + len++; + + dslbis_nonvolatile[1 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + len++; + + dslbis_nonvolatile[2 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dslbis_nonvolatile[3 ] = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis_nonvolatile), + }, + .handle = nonvolatile_dsmad, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + len++; + + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); + *dsemts_nonvolatile = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts_nonvolatile), + }, + .DSMAS_handle = nonvolatile_dsmad, +/* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr =2 , + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + len++; + *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ if (dsmas_nonvolatile) {
Re: [PATCH 3/5] hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work
In reply to Gregory PriceQuoted 4 lines · On Wed, 12 Oct 2022 14:21:18 -0400 Gregory Price <[email protected]>…
Makes the size of the allocated cdat table static (6 entries), flattens the code, and reduces the number of exit conditions
Signed-off-by Gregory Price <[email protected]>
Hmm. I don't entirely like this as it stands because it leads to more fragile code as we don't have clear association between number of entries and actual assignments.
So, what I've done (inspired by this) is moved to a local enum in the factored out building function that has an element for each of the entries (used ultimately to assign them) and a trailing NUM_ENTRIES element we can then use in place of the CT3_CDAT_SUBTABLE_SIZE define you have here.
I went with the 2 pass approach mentioned in a later patch, so if cdat_table passed to the factored out code is NULL, we just return NUM_ENTRIES directly.
Quoted 40 lines
--- hw/mem/cxl_type3.c | 52 ++++++++++++++++++++-------------------------- 1 file changed, 22 insertions(+), 30 deletions(-)hw/mem/cxl_type3.c +1
-0@@ -17,6 +17,7 @@#include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CT3_CDAT_SUBTABLE_SIZE 6 static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,void *priv)
hw/mem/cxl_type3.c +3
-2@@ -25,7 +26,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, g_autofree CDATDslbis *dslbis_nonvolatile = NULL; g_autofree CDATDsemts *dsemts_nonvolatile = NULL;CXLType3Dev *ct3d = priv; -int len =0 ;int i =0 ;int next_dsmad_handle =0 ;int nonvolatile_dsmad =-1 ;@@ -33,7 +33,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,MemoryRegion *mr;if (!ct3d->hostmem) { -return len; +return 0 ; } mr =host_memory_backend_get_memory (ct3d->hostmem);@@ -41,11 +41,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,return -EINVAL ; } + *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table)); +if (!*cdat_table) {
hw/mem/cxl_type3.c static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, · line 54
Quoted 10 lines
hw/mem/cxl_type3.c +7
-1+return -ENOMEM ; + } +/* Non volatile aspects */ dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); -if (!dsmas_nonvolatile) { + dslbis_nonvolatile = +g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); + dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); +if (!dsmas_nonvolatile || !dslbis_nonvolatile || !dsemts_nonvolatile) {
I don't like aggregated error checking. It saves lines of code, but leads to generally less mantainable code. I prefer to do one thing, check it and handle necessary errors - provides a small localized chunk of code that is easy to review and maintain.
1. Allocate structure
2. Fill structure.
We have to leave the assignment till later as only want to steal the pointers once we know there are no error paths.
hw/mem/cxl_type3.c static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, · line 55
Quoted 1 line
hw/mem/cxl_type3.c +1
-0+g_free (*cdat_table);
We have auto free to clean this up. So if this did make sense, use a local g_autofree CDATSubHeader **cdat_table = NULL;
and steal the pointer when assigning *cdat_table at the end of this function after all the failure paths.
This code all ends up in the caller of the factored out code anyway so that comment becomes irrelevant on the version I've ended up with.
Jonathan
Quoted 70 lines
hw/mem/cxl_type3.c +2
-19+ *cdat_table =NULL ;return -ENOMEM ; } + nonvolatile_dsmad = next_dsmad_handle++; *dsmas_nonvolatile = (CDATDsmas ) { .header = {@@ -57,15 +68,8 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .DPA_base = 0, .DPA_length = int128_get64(mr->size), }; - len++;/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); -if (!dslbis_nonvolatile) { -return -ENOMEM ; - } - dslbis_nonvolatile[0 ] = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS ,@@ -77,7 +81,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 10000, /* 10ns base */ .entry[0] = 15, /* 150ns */ }; - len++; dslbis_nonvolatile[1 ] = (CDATDslbis ) { .header = {@@ -90,7 +93,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 10000, .entry[0] = 25, /* 250ns */ }; - len++; dslbis_nonvolatile[2 ] = (CDATDslbis ) { .header = {@@ -103,7 +105,6 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 1000, /* GB/s */ .entry[0] = 16, }; - len++; dslbis_nonvolatile[3 ] = (CDATDslbis ) { .header = {@@ -116,9 +117,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry_base_unit = 1000, /* GB/s */ .entry[0] = 16, }; - len++; - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); *dsemts_nonvolatile = (CDATDsemts ) { .header = { .type =CDAT_TYPE_DSEMTS ,@@ -130,26 +129,19 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .DPA_offset = 0, .DPA_length = int128_get64(mr->size), }; - len++; - *cdat_table =g_malloc0 (len *sizeof (*cdat_table));/* Header always at start of structure */ -if (dsmas_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); - } -if (dslbis_nonvolatile) { -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile);
hw/mem/cxl_type3.c static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, · line 136
Quoted 10 lines
hw/mem/cxl_type3.c +2
-7-int j; + (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); -for (j =0 ; j < dslbis_nonvolatile_num; j++) { - (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } - } -if (dsemts_nonvolatile) { - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); +CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile);
Removing the paranoid checking makes sense if we are going to handle the volatile / non volatile as 'whole sets of tables'.
Quoted 12 lines
hw/mem/cxl_type3.c +6
-1+int j; +for (j =0 ; j < dslbis_nonvolatile_num; j++) { + (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; } -return len; + (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + +return CT3_CDAT_SUBTABLE_SIZE ; }static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv)
Re: [PATCH 4/5] hw/mem/cxl_type3: Change the CDAT allocation/free strategy
In reply to Gregory PriceQuoted 3 lines · On Wed, 12 Oct 2022 14:21:19 -0400 Gregory Price <[email protected]>…
The existing code allocates a subtable for SLBIS entries, uses a local variable to avoid a
g_autofreefootgun, and the cleanup code causes heap corruption.
Ah good point (particularly given I moaned about how you were handling the frees and still failed to notice the current code was broken!)
Quoted 5 lines
Rather than allocate a table, explicitly allocate each individual entry and make the sub-table size static.
Signed-off-by Gregory Price <[email protected]>
I'll integrate a change in the spirit of what you have here, but without aggregating the error handling paths.
Quoted 106 lines
--- hw/mem/cxl_type3.c | 49 ++++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 23 deletions(-)hw/mem/cxl_type3.c +20
-19@@ -23,13 +23,14 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) { g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile1 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile2 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile3 = NULL; + g_autofree CDATDslbis *dslbis_nonvolatile4 = NULL; g_autofree CDATDsemts *dsemts_nonvolatile = NULL;CXLType3Dev *ct3d = priv; -int i =0 ;int next_dsmad_handle =0 ;int nonvolatile_dsmad =-1 ; -int dslbis_nonvolatile_num =4 ;MemoryRegion *mr;if (!ct3d->hostmem) {@@ -48,10 +49,15 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,/* Non volatile aspects */ dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); - dslbis_nonvolatile = -g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); + dslbis_nonvolatile1 =g_malloc (sizeof (*dslbis_nonvolatile1)); + dslbis_nonvolatile2 =g_malloc (sizeof (*dslbis_nonvolatile2)); + dslbis_nonvolatile3 =g_malloc (sizeof (*dslbis_nonvolatile3)); + dslbis_nonvolatile4 =g_malloc (sizeof (*dslbis_nonvolatile4)); dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); -if (!dsmas_nonvolatile || !dslbis_nonvolatile || !dsemts_nonvolatile) { + +if (!dsmas_nonvolatile || !dsemts_nonvolatile || + !dslbis_nonvolatile1 || !dslbis_nonvolatile2 || + !dslbis_nonvolatile3 || !dslbis_nonvolatile4) {g_free (*cdat_table); *cdat_table =NULL ;return -ENOMEM ;@@ -70,10 +76,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* For now, no memory side cache, plausiblish numbers */ - dslbis_nonvolatile[0 ] = (CDATDslbis ) { + *dslbis_nonvolatile1 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile1), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,@@ -82,10 +88,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 15, /* 150ns */ }; - dslbis_nonvolatile[1 ] = (CDATDslbis ) { + *dslbis_nonvolatile2 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile2), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,@@ -94,10 +100,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 25, /* 250ns */ }; - dslbis_nonvolatile[2 ] = (CDATDslbis ) { + *dslbis_nonvolatile3 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile3), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,@@ -106,10 +112,10 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, .entry[0] = 16, }; - dslbis_nonvolatile[3 ] = (CDATDslbis ) { + *dslbis_nonvolatile4 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile), + .length =sizeof (*dslbis_nonvolatile4), }, .handle = nonvolatile_dsmad, .flags =HMAT_LB_MEM_MEMORY ,@@ -131,15 +137,12 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* Header always at start of structure */ - (*cdat_table)[i++] =g_steal_pointer (&dsmas_nonvolatile); - -CDATDslbis *dslbis =g_steal_pointer (&dslbis_nonvolatile); -int j; -for (j =0 ; j < dslbis_nonvolatile_num; j++) {
hw/mem/cxl_type3.c static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, · line 145
Quoted 10 lines
hw/mem/cxl_type3.c +6
-4- (*cdat_table)[i++] = (CDATSubHeader *)&dslbis[j]; - } - - (*cdat_table)[i++] =g_steal_pointer (&dsemts_nonvolatile); + (*cdat_table)[0 ] =g_steal_pointer (&dsmas_nonvolatile); + (*cdat_table)[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile1); + (*cdat_table)[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile2); + (*cdat_table)[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile3); + (*cdat_table)[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile4); + (*cdat_table)[5 ] =g_steal_pointer (&dsemts_nonvolatile);
Moving to simple indexing makes sense now they are all in one place (making introducing a bug much less likely!)
I've introduced an enum so that we have an automatic agreement between number of elements and these assignments.
Quoted 3 lines
hw/mem/cxl_type3.c +0
-0return CT3_CDAT_SUBTABLE_SIZE ; }
Re: [PATCH 5/5] hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a function
In reply to Gregory PriceQuoted 5 lines · On Wed, 12 Oct 2022 14:21:20 -0400 Gregory Price <[email protected]>…
The CDAT can contain multiple entries for multiple memory regions, this will allow us to re-use the initialization code when volatile memory region support is added.
Signed-off-by Gregory Price <[email protected]>
I'm in two minds about this... We could integrate it in the original series, but at that time the change is justified. Or we could leave it as a first patch in your follow on series.
Anyhow, I went with a similar refactor inspired by this.
Quoted 9 lines
--- hw/mem/cxl_type3.c | 137 ++++++++++++++++++++++++--------------------- 1 file changed, 72 insertions(+), 65 deletions(-)hw/mem/cxl_type3.c +0
-0
hw/mem/cxl_type3.c line 23
Quoted 7 lines
hw/mem/cxl_type3.c +2
-2@@ -19,117 +19,93 @@#define DWORD_BYTE 4#define CT3_CDAT_SUBTABLE_SIZE 6 -static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, -void *priv) +static int ct3_build_cdat_subtable (CDATSubHeader **cdat_table, +MemoryRegion *mr,int dsmad_handle)
subtable is particularly well defined. Maybect3_build_cdat_entries_for_mr()?
Quoted 76 lines
hw/mem/cxl_type3.c +19
-43{ - g_autofree CDATDsmas *dsmas_nonvolatile = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile1 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile2 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile3 = NULL; - g_autofree CDATDslbis *dslbis_nonvolatile4 = NULL; - g_autofree CDATDsemts *dsemts_nonvolatile = NULL; -CXLType3Dev *ct3d = priv; -int next_dsmad_handle =0 ; -int nonvolatile_dsmad =-1 ; -MemoryRegion *mr; - -if (!ct3d->hostmem) { -return 0 ; - } - - mr =host_memory_backend_get_memory (ct3d->hostmem); -if (!mr) { -return -EINVAL ; - } - - *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table)); -if (!*cdat_table) { -return -ENOMEM ; - } - -/* Non volatile aspects */ - dsmas_nonvolatile =g_malloc (sizeof (*dsmas_nonvolatile)); - dslbis_nonvolatile1 =g_malloc (sizeof (*dslbis_nonvolatile1)); - dslbis_nonvolatile2 =g_malloc (sizeof (*dslbis_nonvolatile2)); - dslbis_nonvolatile3 =g_malloc (sizeof (*dslbis_nonvolatile3)); - dslbis_nonvolatile4 =g_malloc (sizeof (*dslbis_nonvolatile4)); - dsemts_nonvolatile =g_malloc (sizeof (*dsemts_nonvolatile)); - -if (!dsmas_nonvolatile || !dsemts_nonvolatile || - !dslbis_nonvolatile1 || !dslbis_nonvolatile2 || - !dslbis_nonvolatile3 || !dslbis_nonvolatile4) { -g_free (*cdat_table); - *cdat_table =NULL ; + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis1 = NULL; + g_autofree CDATDslbis *dslbis2 = NULL; + g_autofree CDATDslbis *dslbis3 = NULL; + g_autofree CDATDslbis *dslbis4 = NULL; + g_autofree CDATDsemts *dsemts = NULL; + + dsmas =g_malloc (sizeof (*dsmas)); + dslbis1 =g_malloc (sizeof (*dslbis1)); + dslbis2 =g_malloc (sizeof (*dslbis2)); + dslbis3 =g_malloc (sizeof (*dslbis3)); + dslbis4 =g_malloc (sizeof (*dslbis4)); + dsemts =g_malloc (sizeof (*dsemts)); + +if (!dsmas || !dslbis1 || !dslbis2 || !dslbis3 || !dslbis4 || !dsemts) {return -ENOMEM ; } - nonvolatile_dsmad = next_dsmad_handle++; - *dsmas_nonvolatile = (CDATDsmas ) { + *dsmas = (CDATDsmas ) { .header = { .type =CDAT_TYPE_DSMAS , - .length =sizeof (*dsmas_nonvolatile), + .length =sizeof (*dsmas), }, - .DSMADhandle = nonvolatile_dsmad, + .DSMADhandle = dsmad_handle, .flags =CDAT_DSMAS_FLAG_NV , .DPA_base =0 , .DPA_length =int128_get64 (mr->size), };/* For now, no memory side cache, plausiblish numbers */ - *dslbis_nonvolatile1 = (CDATDslbis ) { + *dslbis1 = (CDATDslbis ) { .header = {
hw/mem/cxl_type3.c line 64
Quoted 10 lines
hw/mem/cxl_type3.c +2
-2.type = CDAT_TYPE_DSLBIS, - .length = sizeof(*dslbis_nonvolatile1), + .length = sizeof(*dslbis1), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags = HMAT_LB_MEM_MEMORY, .data_type = HMAT_LB_DATA_READ_LATENCY, .entry_base_unit = 10000, /* 10ns base */ .entry[0] = 15, /* 150ns */
If we are going to wrap this up for volatile / non-volatile we probably need to pass in a reasonable value for these. Whilst not technically always true, to test the Linux handling I'd want non-volatile to report as longer latency.
Quoted 86 lines
hw/mem/cxl_type3.c +25
-18}; - *dslbis_nonvolatile2 = (CDATDslbis ) { + *dslbis2 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile2), + .length =sizeof (*dslbis2), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_WRITE_LATENCY , .entry_base_unit =10000 , .entry[0 ] =25 ,/* 250ns */ }; - *dslbis_nonvolatile3 = (CDATDslbis ) { + *dslbis3 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile3), + .length =sizeof (*dslbis3), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_READ_BANDWIDTH , .entry_base_unit =1000 ,/* GB/s */ .entry[0 ] =16 , }; - *dslbis_nonvolatile4 = (CDATDslbis ) { + *dslbis4 = (CDATDslbis ) { .header = { .type =CDAT_TYPE_DSLBIS , - .length =sizeof (*dslbis_nonvolatile4), + .length =sizeof (*dslbis4), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags =HMAT_LB_MEM_MEMORY , .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , .entry_base_unit =1000 ,/* GB/s */ .entry[0 ] =16 , }; - *dsemts_nonvolatile = (CDATDsemts ) { + *dsemts = (CDATDsemts ) { .header = { .type =CDAT_TYPE_DSEMTS , - .length =sizeof (*dsemts_nonvolatile), + .length =sizeof (*dsemts), }, - .DSMAS_handle = nonvolatile_dsmad, + .DSMAS_handle = dsmad_handle,/* Reserved - the non volatile from DSMAS matters */ .EFI_memory_type_attr =2 , .DPA_offset =0 ,@@ -137,16 +113,47 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, };/* Header always at start of structure */ - (*cdat_table)[0 ] =g_steal_pointer (&dsmas_nonvolatile); - (*cdat_table)[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile1); - (*cdat_table)[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile2); - (*cdat_table)[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile3); - (*cdat_table)[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis_nonvolatile4); - (*cdat_table)[5 ] =g_steal_pointer (&dsemts_nonvolatile); + cdat_table[0 ] =g_steal_pointer (&dsmas); + cdat_table[1 ] = (CDATSubHeader *)g_steal_pointer (&dslbis1); + cdat_table[2 ] = (CDATSubHeader *)g_steal_pointer (&dslbis2); + cdat_table[3 ] = (CDATSubHeader *)g_steal_pointer (&dslbis3); + cdat_table[4 ] = (CDATSubHeader *)g_steal_pointer (&dslbis4); + cdat_table[5 ] =g_steal_pointer (&dsemts);return CT3_CDAT_SUBTABLE_SIZE ; } +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, +void *priv) +{ +CXLType3Dev *ct3d = priv; +MemoryRegion *mr; +int ret =0 ; +
hw/mem/cxl_type3.c static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, · line 142
Quoted 10 lines
hw/mem/cxl_type3.c +10
-0+if (!ct3d->hostmem) { +return 0 ; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return -EINVAL ; + } + + *cdat_table =g_malloc0 (CT3_CDAT_SUBTABLE_SIZE *sizeof (*cdat_table));
This bakes in assumptions at the wrong layer in the code. Out here we should not know how big the table is - that is a job just for the ct3_build_cdat_subtable() part.
Various options come to mind..
1) Two pass approach. First call ct3_build_cdat_subtable() with NULL pointer passed in. For that all it does it return the number of elements. The the caller calls it again providing suitable storage.
2) Allocate in ct3_build_cdat_subtable() then copy in the caller or use directly if only one type of memory present.
I've gone with the 2 pass approach. Let me know what you think of it once I send the patches out in a few mins.
Thanks,
Jonathan
Quoted 17 lines
hw/mem/cxl_type3.c +14
-0+if (!*cdat_table) { +return -ENOMEM ; + } + +/* Non volatile aspects */ + ret =ct3_build_cdat_subtable (*cdat_table, mr,0 ); +if (ret <0 ) { +g_free (*cdat_table); + *cdat_table =NULL ; + } + +return ret; +} +static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) {int i;
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Gregory PriceQuoted 27 lines · On Wed, 12 Oct 2022 12:01:54 -0400 Gregory Price <[email protected]>…
This code contains heap corruption on free, and I think should be refactored to pre-allocate all the entries we're interested in putting into the table. This would flatten the code and simplify the error handling steps.
Also, should we consider making a union with all the possible entries to make entry allocation easier? It may eat a few extra bytes of memory, but it would simplify the allocation/cleanup code here further.
Given that every allocation has to be checked, i'm also not convinced the use of
g_autofreeis worth the potential footguns associated with it.Quoted 8 lines
hw/mem/cxl_type3.c +3
-0@@ -12,9 +12,218 @@ +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table, +void *priv) +{(snip)
hw/mem/cxl_type3.c line 61
Quoted 4 lines
hw/mem/cxl_type3.c +4
-0+/* For now, no memory side cache, plausiblish numbers */ + dslbis_nonvolatile =g_malloc (sizeof (*dslbis_nonvolatile) * dslbis_nonvolatile_num); +if (!dslbis_nonvolatile) +return -ENOMEM ;
Quoted 7 lines
this allocation creates a table of entries, which is later freed incorrectly
hw/mem/cxl_type3.c
Quoted 2 lines
hw/mem/cxl_type3.c +2
-0+ + *cdat_table =g_malloc0 (len *sizeof (*cdat_table));this allocation needs to be checked
I just realized that sizeof should be sizeof(**cdat_table)
I've moved to a local autofree pointer after factoring out the guts of the code so this gets simpler anyway (and was more obviously wrong!)
Jonathan
Re: [PATCH v7 3/5] hw/cxl/cdat: CXL CDAT Data Object Exchange implementation
In reply to Jonathan CameronQuoted 7 lines · On Fri, 7 Oct 2022 16:21:54 +0100 Jonathan Cameron…
From: Huai-Cheng Kuo <[email protected]>
The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.02, Oct. 2020" and "Compute Express Link Specification, Rev. 2.0, Oct. 2020"
Quoted 6 lines
This patch adds core support that will be shared by both end-points and switch port emulation.
Signed-off-by Huai-Cheng Kuo <[email protected]>Signed-off-by Chris Browy <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Whilst doing v8 I'll uprev this to the 1.03 CDAT spec and CXL 3.0 Changes are minor and it's backwards compatible but it's hard to get the older CXL spec now 3.0 is out.
Quoted 480 lines
---
Changes since RFC:
- Split out libary code from specific device.--- hw/cxl/cxl-cdat.c | 222 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_cdat.h | 165 ++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 7 ++ include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + 6 files changed, 399 insertions(+)hw/cxl/cxl-cdat.c new file +222
-0@@ -0,0 +1,222 @@ +/* +* CXL CDAT Structure +* +* Copyright (C) 2021 Avery Design Systems, Inc. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" +#include "qapi/error.h" +#include "qemu/error-report.h" + +static void cdat_len_check (CDATSubHeader *hdr,Error **errp) +{ +assert (hdr->length); +assert (hdr->reserved ==0 ); + +switch (hdr->type) { +case CDAT_TYPE_DSMAS : +assert (hdr->length ==sizeof (CDATDsmas)); +break ; +case CDAT_TYPE_DSLBIS : +assert (hdr->length ==sizeof (CDATDslbis)); +break ; +case CDAT_TYPE_DSMSCIS : +assert (hdr->length ==sizeof (CDATDsmscis)); +break ; +case CDAT_TYPE_DSIS : +assert (hdr->length ==sizeof (CDATDsis)); +break ; +case CDAT_TYPE_DSEMTS : +assert (hdr->length ==sizeof (CDATDsemts)); +break ; +case CDAT_TYPE_SSLBIS : +assert (hdr->length >=sizeof (CDATSslbisHeader)); +assert ((hdr->length -sizeof (CDATSslbisHeader)) % +sizeof (CDATSslbe) ==0 ); +break ; +default : +error_setg (errp,"Type %d is reserved" , hdr->type); + } +} + +static void ct3_build_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATTableHeader *cdat_header = NULL; + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int ent, i; + +/* Use default table if fopen == NULL */ +assert (cdat->build_cdat_table); + + cdat_header =g_malloc0 (sizeof (*cdat_header)); +if (!cdat_header) { +error_setg (errp,"Failed to allocate CDAT header" ); +return ; + } + + cdat->built_buf_len = cdat->build_cdat_table (&cdat->built_buf, cdat->private); + +if (!cdat->built_buf_len) { +/* Build later as not all data available yet */ + cdat->to_update = true; +return ; + } + cdat->to_update = false; + + cdat_st =g_malloc0 (sizeof (*cdat_st) * (cdat->built_buf_len +1 )); +if (!cdat_st) { +error_setg (errp,"Failed to allocate CDAT entry array" ); +return ; + } + +/* Entry 0 for CDAT header, starts with Entry 1 */ +for (ent =1 ; ent < cdat->built_buf_len +1 ; ent++) { +CDATSubHeader *hdr = cdat->built_buf[ent -1 ]; +uint8_t *buf = (uint8_t *)cdat->built_buf[ent -1 ]; + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + + cdat_header->length += hdr->length; +for (i =0 ; i < hdr->length; i++) { + sum += buf[i]; + } + } + +/* CDAT header */ + cdat_header->revision =CXL_CDAT_REV ; +/* For now, no runtime updates */ + cdat_header->sequence =0 ; + cdat_header->length +=sizeof (CDATTableHeader); + sum += cdat_header->revision + cdat_header->sequence + + cdat_header->length; +/* Sum of all bytes including checksum must be 0 */ + cdat_header->checksum = ~sum +1 ; + + cdat_st[0 ].base =g_steal_pointer (&cdat_header); + cdat_st[0 ].length =sizeof (*cdat_header); + cdat->entry_len =1 + cdat->built_buf_len; + cdat->entry =g_steal_pointer (&cdat_st); +} + +static void ct3_load_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int num_ent; +int i =0 , ent =1 , file_size =0 ; +CDATSubHeader *hdr; +FILE *fp =NULL ; + +/* Read CDAT file and create its cache */ + fp =fopen (cdat->filename,"r" ); +if (!fp) { +error_setg (errp,"CDAT: Unable to open file" ); +return ; + } + +fseek (fp,0 ,SEEK_END ); + file_size =ftell (fp); +fseek (fp,0 ,SEEK_SET ); + cdat->buf =g_malloc0 (file_size); + +if (fread (cdat->buf, file_size,1 , fp) ==0 ) { +error_setg (errp,"CDAT: File read failed" ); +return ; + } + +fclose (fp); + +if (file_size <sizeof (CDATTableHeader)) { +error_setg (errp,"CDAT: File too short" ); +return ; + } + i =sizeof (CDATTableHeader); + num_ent =1 ; +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + i += hdr->length; + num_ent++; + } +if (i != file_size) { +error_setg (errp,"CDAT: File length missmatch" ); +return ; + } + + cdat_st =g_malloc0 (sizeof (*cdat_st) * num_ent); +if (!cdat_st) { +error_setg (errp,"CDAT: Failed to allocate entry array" ); +return ; + } + +/* Set CDAT header, Entry = 0 */ + cdat_st[0 ].base = cdat->buf; + cdat_st[0 ].length =sizeof (CDATTableHeader); + i =0 ; + +while (i < cdat_st[0 ].length) { + sum += cdat->buf[i++]; + } + +/* Read CDAT structures */ +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + +while (cdat->buf + i < + (uint8_t *)cdat_st[ent].base + cdat_st[ent].length) { +assert (i < file_size); + sum += cdat->buf[i++]; + } + + ent++; + } + +if (sum !=0 ) { +warn_report ("CDAT: Found checksum mismatch in %s" , cdat->filename); + } + cdat->entry_len = num_ent; + cdat->entry =g_steal_pointer (&cdat_st); +} + +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->filename) { +ct3_load_cdat (cdat, errp); + }else { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->to_update) { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +free (cdat->entry); +if (cdat->built_buf) + cdat->free_cdat_table (cdat->built_buf, cdat->built_buf_len, + cdat->private); +if (cdat->buf) +free (cdat->buf); +}hw/cxl/meson.build +1
-0@@ -4,6 +4,7 @@ softmmu_ss.add(when: 'CONFIG_CXL', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c', + 'cxl-cdat.c', ), if_false: files( 'cxl-host-stubs.c',include/hw/cxl/cxl_cdat.h new file +165
-0@@ -0,0 +1,165 @@ +/* +* CXL CDAT Structure +* +* Copyright (C) 2021 Avery Design Systems, Inc. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifndef CXL_CDAT_H +#define CXL_CDAT_H + +#include "hw/cxl/cxl_pci.h" + +/* +* Reference: +* Coherent Device Attribute Table (CDAT) Specification, Rev. 1.02, Oct. 2020 +* Compute Express Link (CXL) Specification, Rev. 2.0, Oct. 2020 +*/ +/* Table Access DOE - CXL 8.1.11 */ +#define CXL_DOE_TABLE_ACCESS 2 +#define CXL_DOE_PROTOCOL_CDAT ((CXL_DOE_TABLE_ACCESS << 16) | CXL_VENDOR_ID) + +/* Read Entry - CXL 8.1.11.1 */ +#define CXL_DOE_TAB_TYPE_CDAT 0 +#define CXL_DOE_TAB_ENT_MAX 0xFFFF + +/* Read Entry Request - CXL 8.1.11.1 Table 134 */ +#define CXL_DOE_TAB_REQ 0 +typedef struct CDATReq { +DOEHeader header; +uint8_t req_code; +uint8_t table_type; +uint16_t entry_handle; +} QEMU_PACKED CDATReq; + +/* Read Entry Response - CXL 8.1.11.1 Table 135 */ +#define CXL_DOE_TAB_RSP 0 +typedef struct CDATRsp { +DOEHeader header; +uint8_t rsp_code; +uint8_t table_type; +uint16_t entry_handle; +} QEMU_PACKED CDATRsp; + +/* CDAT Table Format - CDAT Table 1 */ +#define CXL_CDAT_REV 1 +typedef struct CDATTableHeader { +uint32_t length; +uint8_t revision; +uint8_t checksum; +uint8_t reserved[6 ]; +uint32_t sequence; +} QEMU_PACKED CDATTableHeader; + +/* CDAT Structure Types - CDAT Table 2 */ +typedef enum { +CDAT_TYPE_DSMAS =0 , +CDAT_TYPE_DSLBIS =1 , +CDAT_TYPE_DSMSCIS =2 , +CDAT_TYPE_DSIS =3 , +CDAT_TYPE_DSEMTS =4 , +CDAT_TYPE_SSLBIS =5 , +}CDATType ; + +typedef struct CDATSubHeader { +uint8_t type; +uint8_t reserved; +uint16_t length; +}CDATSubHeader ; + +/* Device Scoped Memory Affinity Structure - CDAT Table 3 */ +typedef struct CDATDsmas { +CDATSubHeader header; +uint8_t DSMADhandle; +uint8_t flags; +#define CDAT_DSMAS_FLAG_NV (1 << 2) +#define CDAT_DSMAS_FLAG_SHAREABLE (1 << 3) +#define CDAT_DSMAS_FLAG_HW_COHERENT (1 << 4) +#define CDAT_DSMAS_FLAG_DYNAMIC_CAP (1 << 5) +uint16_t reserved; +uint64_t DPA_base; +uint64_t DPA_length; +} QEMU_PACKED CDATDsmas; + +/* Device Scoped Latency and Bandwidth Information Structure - CDAT Table 5 */ +typedef struct CDATDslbis { +CDATSubHeader header; +uint8_t handle; +/* Definitions of these fields refer directly to HMAT fields */ +uint8_t flags; +uint8_t data_type; +uint8_t reserved; +uint64_t entry_base_unit; +uint16_t entry[3 ]; +uint16_t reserved2; +} QEMU_PACKED CDATDslbis; + +/* Device Scoped Memory Side Cache Information Structure - CDAT Table 6 */ +typedef struct CDATDsmscis { +CDATSubHeader header; +uint8_t DSMAS_handle; +uint8_t reserved[3 ]; +uint64_t memory_side_cache_size; +uint32_t cache_attributes; +} QEMU_PACKED CDATDsmscis; + +/* Device Scoped Initiator Structure - CDAT Table 7 */ +typedef struct CDATDsis { +CDATSubHeader header; +uint8_t flags; +uint8_t handle; +uint16_t reserved; +} QEMU_PACKED CDATDsis; + +/* Device Scoped EFI Memory Type Structure - CDAT Table 8 */ +typedef struct CDATDsemts { +CDATSubHeader header; +uint8_t DSMAS_handle; +uint8_t EFI_memory_type_attr; +uint16_t reserved; +uint64_t DPA_offset; +uint64_t DPA_length; +} QEMU_PACKED CDATDsemts; + +/* Switch Scoped Latency and Bandwidth Information Structure - CDAT Table 9 */ +typedef struct CDATSslbisHeader { +CDATSubHeader header; +uint8_t data_type; +uint8_t reserved[3 ]; +uint64_t entry_base_unit; +} QEMU_PACKED CDATSslbisHeader; + +/* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ +typedef struct CDATSslbe { +uint16_t port_x_id; +uint16_t port_y_id; +uint16_t latency_bandwidth; +uint16_t reserved; +} QEMU_PACKED CDATSslbe; + +typedef struct CDATSslbis { +CDATSslbisHeader sslbis_header; +CDATSslbe sslbe[]; +}CDATSslbis ; + +typedef struct CDATEntry { +void *base; +uint32_t length; +}CDATEntry ; + +typedef struct CDATObject { +CDATEntry *entry; +int entry_len; + +int (*build_cdat_table)(CDATSubHeader ***cdat_table,void *priv); +void (*free_cdat_table)(CDATSubHeader **,int num,void *priv); +bool to_update; +void *private; +char *filename; +uint8_t *buf; +struct CDATSubHeader **built_buf; +int built_buf_len; +}CDATObject ; +#endif /* CXL_CDAT_H */ include/hw/cxl/cxl_component.h +7
-0@@ -19,6 +19,7 @@#include "qemu/range.h" #include "qemu/typedefs.h" #include "hw/register.h" +#include "qapi/error.h" enum reg_type {CXL2_DEVICE ,@@ -184,6 +185,8 @@ typedef struct cxl_component {struct PCIDevice *pdev; }; }; + +CDATObject cdat; } CXLComponentState; void cxl_component_register_block_init(Object *obj,@@ -220,4 +223,8 @@ static inline hwaddr cxl_decode_ig(int ig)CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp); +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate); +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp); +#endif include/hw/cxl/cxl_device.h +3
-0@@ -243,6 +243,9 @@ struct CXLType3Dev {AddressSpace hostmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate; + +/* DOE */ +DOECap doe_cdat; };#define TYPE_CXL_TYPE3 "cxl-type3"include/hw/cxl/cxl_pci.h +1
-0@@ -13,6 +13,7 @@#include "qemu/compiler.h" #include "hw/pci/pci.h" #include "hw/pci/pcie.h" +#include "hw/cxl/cxl_cdat.h" #define CXL_VENDOR_ID 0x1e98
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Jonathan CameronReading through your notes, everything seems reasonable, though I'm not sure I agree with the two pass notion, though I'll wait to see the patch set.
The enum is a good idea, *forehead slap*, I should have done it. If we have a local enum, why not just make it global (within the file) and allocate the table as I have once we know how many MRs are present?
6 eggs/half dozen though, I'm ultimately fine with either.
Quoted 39 lines · Jonathan Cameron
Quoted 25 lines · On Wed, 12 Oct 2022 14:21:15 -0400 Gregory Price <[email protected]>…
Included in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
functionQuoted 3 lines
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)Thanks, I'm going to roll this stuff into the original patch set for v8. Some of this I already have (like the check patch stuff). Some I may disagree with in which case I'll reply to the patches - note I haven't looked at them in detail yet!
Jonathan
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Quoted 7 lines · On Thu, 13 Oct 2022 07:36:28 -0400 Gregory Price <[email protected]>…
Reading through your notes, everything seems reasonable, though I'm not sure I agree with the two pass notion, though I'll wait to see the patch set.
The enum is a good idea, *forehead slap*, I should have done it. If we have a local enum, why not just make it global (within the file) and allocate the table as I have once we know how many MRs are present?
It's not global as we need the entries to be packed. So if just one mr (which ever one) the entries for that need to be at the beginning of cdat_table. I also don't want to bake into the outer caller that the entries will always be the same size for different MRs.
For the two pass case...
I'll send code in a few mins, but in meantime my thought is that the extended code for volatile + non volatile will looks something like:
(variable names made up)
if (ct3d->volatile_mem) {
volatile_mr = host_memory_backend_get_memory(ct3d->volatile_mem....);
if (!volatile_mr) {
return -ENINVAL;
}
rc = ct3_build_cdat_entries_for_mr(NULL, dsmad++, volatile_mr);
if (rc < 0) {
return rc;
}
volatile_len = rc;
}
if (ct3d->nonvolatile_mem) {
nonvolatile_mr = host_memory_backend_get_memory(ct3d->nonvolatile_mem);
if (!nonvolatile_mr) {
return -ENINVAL;
}
rc = ct3_build_cdat_entries_for_mr(NULL, dmsmad++, nonvolatile_mr....);
if (rc < 0) {
return rc;
}
nonvolatile_len = rc;
}
dsmad = 0;
table = g_malloc(0, (volatile_len + nonvolatile_len) * sizeof(*table));
if (!table) {
return -ENOMEM;
}
if (volatile_len) {
rc = ct3_build_cdat_entries_for_mr(&table[0], dmsad++, volatile_mr....);
if (rc < 0) {
return rc;
}
}
if (nonvolatile_len) {
rc = ct3_build_cdat_entries_for_mr(&table[volatile_len], dsmad++, nonvolatile_mr...);
if (rc < 0) {
/* Only place we need error handling. Could make it more generic of course */
for (i = 0; i < volatile_len; i++) {
g_free(cdat_table[i]);
}
return rc;
}
}
*cdat_table = g_steal_pointer(&table);Jonathan
Quoted 46 lines
6 eggs/half dozen though, I'm ultimately fine with either.
Quoted 39 lines · Jonathan Cameron
Quoted 25 lines · On Wed, 12 Oct 2022 14:21:15 -0400 Gregory Price <[email protected]>…
Included in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
functionQuoted 3 lines
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)Thanks, I'm going to roll this stuff into the original patch set for v8. Some of this I already have (like the check patch stuff). Some I may disagree with in which case I'll reply to the patches - note I haven't looked at them in detail yet!
Jonathan
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
fwiw this is what my function looked like after the prior changes, very similar to yours proposed below
static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,
void *priv)
{
CXLType3Dev *ct3d = priv;
MemoryRegion *vmr = NULL, *pmr = NULL;
uint64_t dpa_base = 0;
int dsmad_handle = 0;
int num_ents = 0;
int cur_ent = 0;
int ret = 0;
if (ct3d->hostvmem) {
vmr = host_memory_backend_get_memory(ct3d->hostvmem);
if (!vmr)
return -EINVAL;
num_ents += CT3_CDAT_SUBTABLE_SIZE;
}
if (ct3d->hostpmem) {
pmr = host_memory_backend_get_memory(ct3d->hostpmem);
if (!pmr)
return -EINVAL;
num_ents += CT3_CDAT_SUBTABLE_SIZE;
}
if (!num_ents) {
return 0;
}
*cdat_table = g_malloc0(num_ents * sizeof(*cdat_table));
if (!*cdat_table) {
return -ENOMEM;
}
/* Volatile aspects are mapped first */
if (vmr) {
ret = ct3_build_cdat_subtable(*cdat_table, vmr, dsmad_handle++,
false, dpa_base);
if (ret < 0) {
goto error_cleanup;
}
dpa_base = vmr->size;
cur_ent += ret;
}
/* Non volatile aspects */
if (pmr) {
/* non-volatile entries follow the volatile entries */
ret = ct3_build_cdat_subtable(&(*cdat_table)[cur_ent], pmr,
dsmad_handle, true, dpa_base);
if (ret < 0) {
goto error_cleanup;
}
cur_ent += ret;
}
assert(cur_ent == num_ents);
return ret;
error_cleanup:
int i;
for (i = 0; i < num_ents; i++) {
g_free(*cdat_table[i]);
}
g_free(*cdat_table);
return ret;
}Quoted 122 lines · Jonathan Cameron
Quoted 7 lines · On Thu, 13 Oct 2022 07:36:28 -0400 Gregory Price <[email protected]>…
Reading through your notes, everything seems reasonable, though I'm not sure I agree with the two pass notion, though I'll wait to see the patch set.
The enum is a good idea, *forehead slap*, I should have done it. If we have a local enum, why not just make it global (within the file) and allocate the table as I have once we know how many MRs are present?
It's not global as we need the entries to be packed. So if just one mr (which ever one) the entries for that need to be at the beginning of
cdat_table. I also don't want to bake into the outer caller that the entries will always be the same size for different MRs.For the two pass case...
I'll send code in a few mins, but in meantime my thought is that the extended code for volatile + non volatile will looks something like:
(variable names made up)if (ct3d->volatile_mem) { volatile_mr = host_memory_backend_get_memory(ct3d->volatile_mem....); if (!volatile_mr) { return -ENINVAL; } rc = ct3_build_cdat_entries_for_mr(NULL, dsmad++, volatile_mr); if (rc < 0) { return rc; } volatile_len = rc; } if (ct3d->nonvolatile_mem) { nonvolatile_mr = host_memory_backend_get_memory(ct3d->nonvolatile_mem); if (!nonvolatile_mr) { return -ENINVAL; } rc = ct3_build_cdat_entries_for_mr(NULL, dmsmad++, nonvolatile_mr....); if (rc < 0) { return rc; } nonvolatile_len = rc; } dsmad = 0; table = g_malloc(0, (volatile_len + nonvolatile_len) * sizeof(*table)); if (!table) { return -ENOMEM; } if (volatile_len) { rc = ct3_build_cdat_entries_for_mr(&table[0], dmsad++, volatile_mr....); if (rc < 0) { return rc; } } if (nonvolatile_len) { rc = ct3_build_cdat_entries_for_mr(&table[volatile_len], dsmad++, nonvolatile_mr...); if (rc < 0) { /* Only place we need error handling. Could make it more generic of course */ for (i = 0; i < volatile_len; i++) { g_free(cdat_table[i]); } return rc; } } *cdat_table = g_steal_pointer(&table);Jonathan
Quoted 46 lines
6 eggs/half dozen though, I'm ultimately fine with either.
Quoted 39 lines · Jonathan Cameron
Quoted 25 lines · On Wed, 12 Oct 2022 14:21:15 -0400 Gregory Price <[email protected]>…
Included in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
functionQuoted 3 lines
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)Thanks, I'm going to roll this stuff into the original patch set for v8. Some of this I already have (like the check patch stuff). Some I may disagree with in which case I'll reply to the patches - note I haven't looked at them in detail yet!
Jonathan
Re: [PATCH v7 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Quoted 2 lines · On Thu, 13 Oct 2022 08:35:13 -0400 Gregory Price <[email protected]>…
fwiw this is what my function looked like after the prior changes, very similar to yours proposed below
Makes sense given only real change is exactly where the size comes from ;)
FYI, I've pushed out latest version on top of qemu/master at gitlab.com/jic23/ as tag doe-v8
Just as soon as I finish bouncing patches to a machine I can push from I'll push out the rest of my queue.
My current thought is to slide your series under the rest of that queue (so directly on top of the DOE set - v8+ depending on reviews).
The other series coming through is Ira's event injection but my guess is that will take a bit more time to stabilize.
Jonathan
Quoted 51 lines
static int ct3_build_cdat_table(CDATSubHeader ***cdat_table,
void *priv){ CXLType3Dev *ct3d = priv; MemoryRegion *vmr = NULL, *pmr = NULL; uint64_t dpa_base = 0; int dsmad_handle = 0; int num_ents = 0; int cur_ent = 0; int ret = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); if (!vmr) return -EINVAL; num_ents += CT3_CDAT_SUBTABLE_SIZE; } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); if (!pmr) return -EINVAL; num_ents += CT3_CDAT_SUBTABLE_SIZE; } if (!num_ents) { return 0; } *cdat_table = g_malloc0(num_ents * sizeof(*cdat_table)); if (!*cdat_table) { return -ENOMEM; } /* Volatile aspects are mapped first */ if (vmr) { ret = ct3_build_cdat_subtable(*cdat_table, vmr, dsmad_handle++, false, dpa_base); if (ret < 0) { goto error_cleanup; } dpa_base = vmr->size; cur_ent += ret; } /* Non volatile aspects */ if (pmr) { /* non-volatile entries follow the volatile entries */ ret = ct3_build_cdat_subtable(&(*cdat_table)[cur_ent], pmr, dsmad_handle, true, dpa_base); if (ret < 0) { goto error_cleanup; }
Quoted 8 lines
cur_ent += ret;
}
assert(cur_ent == num_ents);
return ret;
error_cleanup:
int i;
for (i = 0; i < num_ents; i++) {Might as well loop only to cur_ent as the rest will be NULL.
Quoted 130 lines
g_free(*cdat_table[i]); } g_free(*cdat_table); return ret; }Quoted 122 lines · Jonathan Cameron
Quoted 7 lines · On Thu, 13 Oct 2022 07:36:28 -0400 Gregory Price <[email protected]>…
Reading through your notes, everything seems reasonable, though I'm not sure I agree with the two pass notion, though I'll wait to see the patch set.
The enum is a good idea, *forehead slap*, I should have done it. If we have a local enum, why not just make it global (within the file) and allocate the table as I have once we know how many MRs are present?
It's not global as we need the entries to be packed. So if just one mr (which ever one) the entries for that need to be at the beginning of
cdat_table. I also don't want to bake into the outer caller that the entries will always be the same size for different MRs.For the two pass case...
I'll send code in a few mins, but in meantime my thought is that the extended code for volatile + non volatile will looks something like:
(variable names made up)if (ct3d->volatile_mem) { volatile_mr = host_memory_backend_get_memory(ct3d->volatile_mem....); if (!volatile_mr) { return -ENINVAL; } rc = ct3_build_cdat_entries_for_mr(NULL, dsmad++, volatile_mr); if (rc < 0) { return rc; } volatile_len = rc; } if (ct3d->nonvolatile_mem) { nonvolatile_mr = host_memory_backend_get_memory(ct3d->nonvolatile_mem); if (!nonvolatile_mr) { return -ENINVAL; } rc = ct3_build_cdat_entries_for_mr(NULL, dmsmad++, nonvolatile_mr....); if (rc < 0) { return rc; } nonvolatile_len = rc; } dsmad = 0; table = g_malloc(0, (volatile_len + nonvolatile_len) * sizeof(*table)); if (!table) { return -ENOMEM; } if (volatile_len) { rc = ct3_build_cdat_entries_for_mr(&table[0], dmsad++, volatile_mr....); if (rc < 0) { return rc; } } if (nonvolatile_len) { rc = ct3_build_cdat_entries_for_mr(&table[volatile_len], dsmad++, nonvolatile_mr...); if (rc < 0) { /* Only place we need error handling. Could make it more generic of course */ for (i = 0; i < volatile_len; i++) { g_free(cdat_table[i]); } return rc; } } *cdat_table = g_steal_pointer(&table);Jonathan
Quoted 46 lines
6 eggs/half dozen though, I'm ultimately fine with either.
Quoted 39 lines · Jonathan Cameron
Quoted 25 lines · On Wed, 12 Oct 2022 14:21:15 -0400 Gregory Price <[email protected]>…
Included in this response is a recommended patch set on top of this patch that resolves a number of issues, including style and a heap corruption bug.
The purpose of this patch set is to refactor the CDAT initialization code to support future patch sets that will introduce multi-region support in CXL Type3 devices.
1) Checkpatch errors in the immediately prior patch
2) Flatting of code in cdat initialization
3) Changes in allocation and error checking for cleanliness
4) Change in the allocation/free strategy of CDAT sub-tables to simplify multi-region allocation in the future. Also resolves a heap corruption bug
5) Refactor of CDAT initialization code into a function that initializes sub-tables per memory-region.Gregory Price (5):
hw/mem/cxl_type3: fix checkpatch errors
hw/mem/cxl_type3: Pull validation checks ahead of functional code hw/mem/cxl_type3: CDAT pre-allocate and check resources prior to work hw/mem/cxl_type3: Change the CDAT allocation/free strategy hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a
functionQuoted 3 lines
hw/mem/cxl_type3.c | 240 +++++++++++++++++++++++---------------------- 1 file changed, 122 insertions(+), 118 deletions(-)Thanks, I'm going to roll this stuff into the original patch set for v8. Some of this I already have (like the check patch stuff). Some I may disagree with in which case I'll reply to the patches - note I haven't looked at them in detail yet!
Jonathan
Re: [PATCH 5/5] hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a function
In reply to Jonathan CameronQuoted 15 lines
Quoted 4 lines
/* For now, no memory side cache, plausiblish numbers */hw/mem/cxl_type3.c +1
-1- *dslbis_nonvolatile1 = (CDATDslbis ) { + *dslbis1 = (CDATDslbis ) { .header = {hw/mem/cxl_type3.c line 64
Quoted 10 lines
hw/mem/cxl_type3.c +2
-2.type = CDAT_TYPE_DSLBIS, - .length = sizeof(*dslbis_nonvolatile1), + .length = sizeof(*dslbis1), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags = HMAT_LB_MEM_MEMORY, .data_type = HMAT_LB_DATA_READ_LATENCY, .entry_base_unit = 10000, /* 10ns base */ .entry[0] = 15, /* 150ns */
Quoted 5 lines
If we are going to wrap this up for volatile / non-volatile we probably need to pass in a reasonable value for these. Whilst not technically always true, to test the Linux handling I'd want non-volatile to report as longer latency.
Here's a good question
Do we want the base unit and entry to be adjustable for volatile and nonvolatile regions for the purpose of testing? Or should this simply be a static value for each?
Since we need to pass in (is_pmem/is_nonvolatile) or whatever into the cdat function, we could just use that to do one of a few options:
1) Select from a static value
2) Select a static value and apply a multiplier for nvmem
3) Use a base/value provided by the use and apply a multiplier
4) Make vmem and pmem have separately configurable latencies
Re: [PATCH 5/5] hw/mem/cxl_type3: Refactor CDAT sub-table entry initialization into a function
Quoted 21 lines · On Thu, 13 Oct 2022 15:40:47 -0400 Gregory Price <[email protected]>…
Quoted 15 lines
Quoted 4 lines
/* For now, no memory side cache, plausiblish numbers */hw/mem/cxl_type3.c +1
-1- *dslbis_nonvolatile1 = (CDATDslbis ) { + *dslbis1 = (CDATDslbis ) { .header = {hw/mem/cxl_type3.c line 64
Quoted 10 lines
hw/mem/cxl_type3.c +2
-2.type = CDAT_TYPE_DSLBIS, - .length = sizeof(*dslbis_nonvolatile1), + .length = sizeof(*dslbis1), }, - .handle = nonvolatile_dsmad, + .handle = dsmad_handle, .flags = HMAT_LB_MEM_MEMORY, .data_type = HMAT_LB_DATA_READ_LATENCY, .entry_base_unit = 10000, /* 10ns base */ .entry[0] = 15, /* 150ns */Quoted 5 lines
If we are going to wrap this up for volatile / non-volatile we probably need to pass in a reasonable value for these. Whilst not technically always true, to test the Linux handling I'd want non-volatile to report as longer latency.
Quoted 5 lines
Here's a good question
Do we want the base unit and entry to be adjustable for volatile and nonvolatile regions for the purpose of testing? Or should this simply be a static value for each?
We definitely want a 'default' value if nothing is provided. It might be useful to allow it to be adjusted, but lets add that when we have a use for it (perhaps testing some stuff in kernel where the values matter enough to make them controllable).
Quoted 7 lines
Since we need to pass in (is_pmem/is_nonvolatile) or whatever into the cdat function, we could just use that to do one of a few options:
1) Select from a static value
2) Select a static value and apply a multiplier for nvmem
3) Use a base/value provided by the use and apply a multiplier
4) Make vmem and pmem have separately configurable latencies
For now 1 is fine I think.
I've just pushed out a doe-v9 tag and cxl-2022-10-14 branch to gitlab.com/jic23/qemu Also advanced the base tree to current QEMU mainline.
Note that if anyone is playing with the switch cci device and mainline kernel you'll currently need to revert
https://lore.kernel.org/linux-pci/[email protected]/
Jonathan