QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.
v8 cover QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.
Changes since v7: Details in individual patches:
Thanks to Gregory Price for reviewing!
- Fix heap corruption.
- Check allocations succeed.
- Substantial refactor of type 3 cdat table build to make it simpler and easier to add volatile entry support.
V7 Cover letter - lightly edited.
Whilst I have carried on Huai-Cheng Kuo's series version numbering and naming, there have been very substantial changes since v6 so I would suggest fresh review makes sense for anyone who has looked at this before. In particularly if the Avery design folks could check I haven't broken anything that would be great.
For reference v6: QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
https://lore.kernel.org/qemu-devel/[email protected]/
Summary of changes:
1) Linux headers definitions for DOE are now upstream so drop that patch.
2) Add CDAT for switch upstream port.
3) Generate 'plausible' default CDAT tables when a file is not provided.
4) General refactoring to calculate the correct table sizes and allocate based on that rather than copying from a local static array.
5) Changes from earlier reviews such as matching QEMU type naming style.
6) Moved compliance and SPDM usecases to future patch sets.
Sign-offs on these are complex because the patches were originally developed by Huai-Cheng Kuo, but posted by Chris Browy and then picked up by Jonathan Cameron who made substantial changes.
Huai-Cheng Kuo confirmed they are happy to maintain this updated code.
What's here?
This series brings generic PCI Express Data Object Exchange support (DOE) DOE is defined in the PCIe Base Spec r6.0. It consists of a mailbox in PCI config space via a PCIe Extended Capability Structure.
The PCIe spec defines several protocols (including one to discover what protocols a given DOE instance supports) and other specification such as CXL define additional protocols using their own vendor IDs.
In this series we make use of the DOE to support the CXL spec defined Table Access Protocol, specifically to provide access to CDAT - a table specified in a specification that is hosted by the UEFI forum and is used to provide runtime discoverability of the sort of information that would otherwise be available in firmware tables (memory types, latency and bandwidth information etc).
The Linux kernel gained support for DOE / CDAT on CXL type 3 EPs in 6.0. The version merged did not support interrupts (earlier versions did so that support in the emulation was tested a while back).
This series provides CDAT emulation for CXL switch upstream ports and CXL type 3 memory devices. Note that to exercise the switch support additional Linux kernel patches are needed.
https://lore.kernel.org/linux-cxl/[email protected]/
(I'll post a new version of that support shortly)
Additional protocols will be supported by follow on patch sets:
* CXL compliance protocol.
* CMA / SPDM device attestation.
(Old version at https://gitlab.com/jic23/qemu/-/commits/cxl-next - will refresh that tree next week)
Huai-Cheng Kuo (3):
hw/pci: PCIe Data Object Exchange emulation hw/cxl/cdat: CXL CDAT Data Object Exchange implementation hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Jonathan Cameron (2):
hw/mem/cxl-type3: Add MSIX support hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE
MAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 224 ++++++++++++++++++++ hw/cxl/meson.build | 1 + hw/mem/cxl_type3.c | 276 +++++++++++++++++++++++++ hw/pci-bridge/cxl_upstream.c | 194 ++++++++++++++++- hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_cdat.h | 166 +++++++++++++++ include/hw/cxl/cxl_component.h | 7 + include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++ include/hw/pci/pcie_regs.h | 4 + 15 files changed, 1377 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/pci/pcie_doe.h
Signature · 1 line
2.37.2
PATCH v8 1/5 hw/pci: PCIe Data Object Exchange emulation
From: Huai-Cheng Kuo <[email protected]>
Emulation of PCIe Data Object Exchange (DOE) PCIE Base Specification r6.0 6.3 Data Object Exchange
Supports multiple DOE PCIe Extended Capabilities for a single PCIe device. For each capability, a static array of DOEProtocol should be passed to pcie_doe_init(). The protocols in that array will be registered under the DOE capability structure. For each protocol, vendor ID, type, and corresponding callback function (handle_request()) should be implemented. This callback function represents how the DOE request for corresponding protocol will be handled.
pcie_doe_{read/write}_config() must be appended to corresponding PCI device's config_read/write() handler to enable DOE access. In pcie_doe_read_config(), false will be returned if pci_config_read() offset is not within DOE capability range. In pcie_doe_write_config(), the function will have no affect if the address is not within the related DOE PCIE extended capability.
---
Changes since v7: None.
MAINTAINERS | 7 + hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++++++ include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++++ include/hw/pci/pcie_regs.h | 4 + 7 files changed, 506 insertions(+)
MAINTAINERS +7 -0
@@ -1832,6 +1832,13 @@ F: qapi/pci.json F: docs/pci* F: docs/specs/*pci* +PCIE DOE +M: Huai-Cheng Kuo <[email protected]> +M: Chris Browy <[email protected]> +S: Supported +F: include/hw/pci/pcie_doe.h +F: hw/pci/pcie_doe.c + ACPI/SMBIOS M: Michael S. Tsirkin <[email protected]> M: Igor Mammedov <[email protected]>
hw/pci/meson.build +1 -0
@@ -13,6 +13,7 @@ pci_ss.add(files( # allow plugging PCIe devices into PCI buses, include them even if # CONFIG_PCI_EXPRESS=n. pci_ss.add(files('pcie.c', 'pcie_aer.c')) +pci_ss.add(files('pcie_doe.c')) softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c', 'pcie_host.c')) softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
hw/pci/pcie_doe.c new file +367 -0
367 lines not on this page · raw message
include/hw/pci/pci_ids.h +3 -0
@@ -157,6 +157,9 @@/* Vendors and devices. Sort key: vendor first, device next. */ +/* Ref: PCIe r6.0 Table 6-32 */ +#define PCI_VENDOR_ID_PCI_SIG 0x0001 +#define PCI_VENDOR_ID_LSI_LOGIC 0x1000#define PCI_DEVICE_ID_LSI_53C810 0x0001#define PCI_DEVICE_ID_LSI_53C895A 0x0012
include/hw/pci/pcie.h +1 -0
@@ -26,6 +26,7 @@#include "hw/pci/pcie_aer.h" #include "hw/pci/pcie_sriov.h" #include "hw/hotplug.h" +#include "hw/pci/pcie_doe.h" typedef enum { /* for attention and power indicator */
include/hw/pci/pcie_doe.h new file +123 -0
123 lines not on this page · raw message
include/hw/pci/pcie_regs.h +4 -0
@@ -179,4 +179,8 @@ typedef enum PCIExpLinkWidth {#define PCI_ACS_VER 0x1#define PCI_ACS_SIZEOF 8 +/* DOE Capability Register Fields */ +#define PCI_DOE_VER 0x1 +#define PCI_DOE_SIZEOF 24 +#endif /* QEMU_PCIE_REGS_H */
Signature · 1 line
2.37.2
PATCH v8 2/5 hw/mem/cxl-type3: Add MSIX support
In reply to Jonathan CameronThis will be used by several upcoming patch sets so break it out such that it doesn't matter which one lands first.
--- hw/mem/cxl_type3.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/mem/cxl_type3.c +9 -0
@@ -13,6 +13,7 @@#include "qemu/rcu.h" #include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" +#include "hw/pci/msix.h" /* * Null value of all Fs suggested by IEEE RA guidelines for use of
@@ -146,6 +147,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)ComponentRegisters *regs = &cxl_cstate->crb;MemoryRegion *mr = ®s->component_registers;uint8_t *pci_conf = pci_dev->config; +unsigned short msix_num =1 ; +int i;if (!cxl_setup_memory (ct3d, errp)) {return ;
@@ -180,6 +183,12 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , &ct3d->cxl_dstate.device_registers); + +/* MSI(-X) Initailization */ +msix_init_exclusive_bar (pci_dev, msix_num,4 ,NULL ); +for (i =0 ; i < msix_num; i++) { +msix_vector_use (pci_dev, i); + } }static void ct3_exit (PCIDevice *pci_dev)
Signature · 1 line
2.37.2
PATCH v8 3/5 hw/cxl/cdat: CXL CDAT Data Object Exchange implementation
In reply to Jonathan CameronFrom: Huai-Cheng Kuo <[email protected]>
The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.03, July. 2022" and "Compute Express Link Specification, Rev. 3.0, July. 2022"
This patch adds core support that will be shared by both end-points and switch port emulation.
---
Changes since v7:
- Up rev spec references and version number.
Changes since RFC:
- Split out libary code from specific device.
--- hw/cxl/cxl-cdat.c | 224 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_cdat.h | 165 ++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 7 ++ include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + 6 files changed, 401 insertions(+)
hw/cxl/cxl-cdat.c new file +224 -0
@@ -0,0 +1,224 @@ +/* +* CXL CDAT Structure +* +* Copyright (C) 2021 Avery Design Systems, Inc. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" +#include "qapi/error.h" +#include "qemu/error-report.h" + +static void cdat_len_check (CDATSubHeader *hdr,Error **errp) +{ +assert (hdr->length); +assert (hdr->reserved ==0 ); + +switch (hdr->type) { +case CDAT_TYPE_DSMAS : +assert (hdr->length ==sizeof (CDATDsmas)); +break ; +case CDAT_TYPE_DSLBIS : +assert (hdr->length ==sizeof (CDATDslbis)); +break ; +case CDAT_TYPE_DSMSCIS : +assert (hdr->length ==sizeof (CDATDsmscis)); +break ; +case CDAT_TYPE_DSIS : +assert (hdr->length ==sizeof (CDATDsis)); +break ; +case CDAT_TYPE_DSEMTS : +assert (hdr->length ==sizeof (CDATDsemts)); +break ; +case CDAT_TYPE_SSLBIS : +assert (hdr->length >=sizeof (CDATSslbisHeader)); +assert ((hdr->length -sizeof (CDATSslbisHeader)) % +sizeof (CDATSslbe) ==0 ); +break ; +default : +error_setg (errp,"Type %d is reserved" , hdr->type); + } +} + +static void ct3_build_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATTableHeader *cdat_header = NULL; + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int ent, i; + +/* Use default table if fopen == NULL */ +assert (cdat->build_cdat_table); + + cdat_header =g_malloc0 (sizeof (*cdat_header)); +if (!cdat_header) { +error_setg (errp,"Failed to allocate CDAT header" ); +return ; + } + + cdat->built_buf_len = cdat->build_cdat_table (&cdat->built_buf, cdat->private); + +if (!cdat->built_buf_len) { +/* Build later as not all data available yet */ + cdat->to_update = true; +return ; + } + cdat->to_update = false; + + cdat_st =g_malloc0 (sizeof (*cdat_st) * (cdat->built_buf_len +1 )); +if (!cdat_st) { +error_setg (errp,"Failed to allocate CDAT entry array" ); +return ; + } + +/* Entry 0 for CDAT header, starts with Entry 1 */ +for (ent =1 ; ent < cdat->built_buf_len +1 ; ent++) { +CDATSubHeader *hdr = cdat->built_buf[ent -1 ]; +uint8_t *buf = (uint8_t *)cdat->built_buf[ent -1 ]; + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + + cdat_header->length += hdr->length; +for (i =0 ; i < hdr->length; i++) { + sum += buf[i]; + } + } + +/* CDAT header */ + cdat_header->revision =CXL_CDAT_REV ; +/* For now, no runtime updates */ + cdat_header->sequence =0 ; + cdat_header->length +=sizeof (CDATTableHeader); + sum += cdat_header->revision + cdat_header->sequence + + cdat_header->length; +/* Sum of all bytes including checksum must be 0 */ + cdat_header->checksum = ~sum +1 ; + + cdat_st[0 ].base =g_steal_pointer (&cdat_header); + cdat_st[0 ].length =sizeof (*cdat_header); + cdat->entry_len =1 + cdat->built_buf_len; + cdat->entry =g_steal_pointer (&cdat_st); +} + +static void ct3_load_cdat (CDATObject *cdat,Error **errp) +{ + g_autofree CDATEntry *cdat_st = NULL; +uint8_t sum =0 ; +int num_ent; +int i =0 , ent =1 , file_size =0 ; +CDATSubHeader *hdr; +FILE *fp =NULL ; + +/* Read CDAT file and create its cache */ + fp =fopen (cdat->filename,"r" ); +if (!fp) { +error_setg (errp,"CDAT: Unable to open file" ); +return ; + } + +fseek (fp,0 ,SEEK_END ); + file_size =ftell (fp); +fseek (fp,0 ,SEEK_SET ); + cdat->buf =g_malloc0 (file_size); + +if (fread (cdat->buf, file_size,1 , fp) ==0 ) { +error_setg (errp,"CDAT: File read failed" ); +return ; + } + +fclose (fp); + +if (file_size <sizeof (CDATTableHeader)) { +error_setg (errp,"CDAT: File too short" ); +return ; + } + i =sizeof (CDATTableHeader); + num_ent =1 ; +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + i += hdr->length; + num_ent++; + } +if (i != file_size) { +error_setg (errp,"CDAT: File length missmatch" ); +return ; + } + + cdat_st =g_malloc0 (sizeof (*cdat_st) * num_ent); +if (!cdat_st) { +error_setg (errp,"CDAT: Failed to allocate entry array" ); +return ; + } + +/* Set CDAT header, Entry = 0 */ + cdat_st[0 ].base = cdat->buf; + cdat_st[0 ].length =sizeof (CDATTableHeader); + i =0 ; + +while (i < cdat_st[0 ].length) { + sum += cdat->buf[i++]; + } + +/* Read CDAT structures */ +while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); +cdat_len_check (hdr, errp); + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + +while (cdat->buf + i < + (uint8_t *)cdat_st[ent].base + cdat_st[ent].length) { +assert (i < file_size); + sum += cdat->buf[i++]; + } + + ent++; + } + +if (sum !=0 ) { +warn_report ("CDAT: Found checksum mismatch in %s" , cdat->filename); + } + cdat->entry_len = num_ent; + cdat->entry =g_steal_pointer (&cdat_st); +} + +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->filename) { +ct3_load_cdat (cdat, errp); + }else { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +if (cdat->to_update) { +ct3_build_cdat (cdat, errp); + } +} + +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate) +{ +CDATObject *cdat = &cxl_cstate->cdat; + +free (cdat->entry); +if (cdat->built_buf) { + cdat->free_cdat_table (cdat->built_buf, cdat->built_buf_len, + cdat->private); + } +if (cdat->buf) { +free (cdat->buf); + } +}
hw/cxl/meson.build +1 -0
@@ -4,6 +4,7 @@ softmmu_ss.add(when: 'CONFIG_CXL', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c', + 'cxl-cdat.c', ), if_false: files( 'cxl-host-stubs.c',
include/hw/cxl/cxl_cdat.h new file +165 -0
165 lines not on this page · raw message
include/hw/cxl/cxl_component.h +7 -0
@@ -19,6 +19,7 @@#include "qemu/range.h" #include "qemu/typedefs.h" #include "hw/register.h" +#include "qapi/error.h" enum reg_type {CXL2_DEVICE ,
@@ -184,6 +185,8 @@ typedef struct cxl_component {struct PCIDevice *pdev; }; }; + +CDATObject cdat; } CXLComponentState; void cxl_component_register_block_init(Object *obj,
@@ -220,4 +223,8 @@ static inline hwaddr cxl_decode_ig(int ig)CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +void cxl_doe_cdat_init (CXLComponentState *cxl_cstate,Error **errp); +void cxl_doe_cdat_release (CXLComponentState *cxl_cstate); +void cxl_doe_cdat_update (CXLComponentState *cxl_cstate,Error **errp); +#endif
include/hw/cxl/cxl_device.h +3 -0
@@ -243,6 +243,9 @@ struct CXLType3Dev {AddressSpace hostmem_as;CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate; + +/* DOE */ +DOECap doe_cdat; };#define TYPE_CXL_TYPE3 "cxl-type3"
include/hw/cxl/cxl_pci.h +1 -0
@@ -13,6 +13,7 @@#include "qemu/compiler.h" #include "hw/pci/pci.h" #include "hw/pci/pcie.h" +#include "hw/cxl/cxl_cdat.h" #define CXL_VENDOR_ID 0x1e98
Signature · 1 line
2.37.2
PATCH v8 4/5 hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Jonathan CameronFrom: Huai-Cheng Kuo <[email protected]>
The CDAT can be specified in two ways. One is to add ",cdat=<filename>" in "-device cxl-type3"'s command option. The file is required to provide the whole CDAT table in binary mode. The other is to use the default that provides some 'reasonable' numbers based on type of memory and size.
The DOE capability supporting CDAT is added to hw/mem/cxl_type3.c with capability offset 0x190. The config read/write to this capability range can be generated in the OS to request the CDAT data.
---
Changes since v7: Thanks to Gregory Price for review + patches.
- Fix a heap corruption
- Factor out the entry buildling to a separate function that will soon be useful for volatile case.
- Switch to enum of entries so NUM_ENTRIES is automatically kept in sync with any additional elements.
Changes since RFC:
- Break out type 3 user of library as separate patch.
- Change reported data for default to be based on the options provided for the type 3 device.
--- hw/mem/cxl_type3.c | 267 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 267 insertions(+)
hw/mem/cxl_type3.c +267 -0
@@ -12,9 +12,258 @@#include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" +#include "sysemu/numa.h" #include "hw/cxl/cxl.h" #include "hw/pci/msix.h" +#define DWORD_BYTE 4 + +/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + }; + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis0 = NULL; + g_autofree CDATDslbis *dslbis1 = NULL; + g_autofree CDATDslbis *dslbis2 = NULL; + g_autofree CDATDslbis *dslbis3 = NULL; + g_autofree CDATDsemts *dsemts = NULL; + +if (!cdat_table) { +return NUM_ENTRIES ; + } + + dsmas =g_malloc (sizeof (*dsmas)); +if (!dsmas) { +return -ENOMEM ; + } + *dsmas = (CDATDsmas ) { + .header = { + .type =CDAT_TYPE_DSMAS , + .length =sizeof (*dsmas), + }, + .DSMADhandle = dsmad_handle, + .flags =CDAT_DSMAS_FLAG_NV , + .DPA_base =0 , + .DPA_length =int128_get64 (mr->size), + }; + +/* For now, no memory side cache, plausiblish numbers */ + dslbis0 =g_malloc (sizeof (*dslbis0)); +if (!dslbis0) { +return -ENOMEM ; + } + *dslbis0 = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis0), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_LATENCY , + .entry_base_unit =10000 ,/* 10ns base */ + .entry[0 ] =15 ,/* 150ns */ + }; + + dslbis1 =g_malloc (sizeof (*dslbis1)); +if (!dslbis1) { +return -ENOMEM ; + } + *dslbis1 = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis1), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_LATENCY , + .entry_base_unit =10000 , + .entry[0 ] =25 ,/* 250ns */ + }; + + dslbis2 =g_malloc (sizeof (*dslbis2)); +if (!dslbis2) { +return -ENOMEM ; + } + *dslbis2 = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis2), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_READ_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + + dslbis3 =g_malloc (sizeof (*dslbis3)); +if (!dslbis3) { +return -ENOMEM ; + } + *dslbis3 = (CDATDslbis ) { + .header = { + .type =CDAT_TYPE_DSLBIS , + .length =sizeof (*dslbis3), + }, + .handle = dsmad_handle, + .flags =HMAT_LB_MEM_MEMORY , + .data_type =HMAT_LB_DATA_WRITE_BANDWIDTH , + .entry_base_unit =1000 ,/* GB/s */ + .entry[0 ] =16 , + }; + + dsemts =g_malloc (sizeof (*dsemts)); +if (!dsemts) { +return -ENOMEM ; + } + *dsemts = (CDATDsemts ) { + .header = { + .type =CDAT_TYPE_DSEMTS , + .length =sizeof (*dsemts), + }, + .DSMAS_handle = dsmad_handle, +/* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr =2 , + .DPA_offset =0 , + .DPA_length =int128_get64 (mr->size), + }; + +/* Header always at start of structure */ + cdat_table[DSMAS ] =g_steal_pointer (&dsmas); + cdat_table[DSLBIS0 ] =g_steal_pointer (&dslbis0); + cdat_table[DSLBIS1 ] =g_steal_pointer (&dslbis1); + cdat_table[DSLBIS2 ] =g_steal_pointer (&dslbis2); + cdat_table[DSLBIS3 ] =g_steal_pointer (&dslbis3); + cdat_table[DSEMTS ] =g_steal_pointer (&dsemts); + +return NUM_ENTRIES ; +} + +static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +int dsmad_handle =0 ; +int len =0 ; +int rc; + +if (!ct3d->hostmem) { +return 0 ; + } + + volatile_mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!volatile_mr) { +return -EINVAL ; + } + +/* How many entries needed for non volatile mr */ + rc =ct3_build_cdat_entries_for_mr (NULL , dsmad_handle, volatile_mr); +if (rc <0 ) { +return rc; + } + len = rc; + + table =g_malloc0 (len *sizeof (*table)); +if (!table) { +return -ENOMEM ; + } + +/* Now fill them in */ + rc =ct3_build_cdat_entries_for_mr (table, dsmad_handle++, volatile_mr); +if (rc <0 ) { +return rc; + } + + *cdat_table =g_steal_pointer (&table); + +return len; +} + +static void ct3_free_cdat_table (CDATSubHeader **cdat_table,int num,void *priv) +{ +int i; + +for (i =0 ; i < num; i++) { +g_free (cdat_table[i]); + } +g_free (cdat_table); +} + +static bool cxl_doe_cdat_rsp (DOECap *doe_cap) +{ +CDATObject *cdat = &CXL_TYPE3 (doe_cap->pdev)->cxl_cstate.cdat; +uint16_t ent; +void *base; +uint32_t len; +CDATReq *req =pcie_doe_get_write_mbox_ptr (doe_cap); +CDATRsp rsp; + +assert (cdat->entry_len); + +/* Discard if request length mismatched */ +if (pcie_doe_get_obj_len (req) < +DIV_ROUND_UP (sizeof (CDATReq),DWORD_BYTE )) { +return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp ) { + .header = { + .vendor_id =CXL_VENDOR_ID , + .data_obj_type =CXL_DOE_TABLE_ACCESS , + .reserved =0x0 , + .length =DIV_ROUND_UP ((sizeof (rsp) + len),DWORD_BYTE ), + }, + .rsp_code =CXL_DOE_TAB_RSP , + .table_type =CXL_DOE_TAB_TYPE_CDAT , + .entry_handle = (ent < cdat->entry_len -1 ) ? + ent +1 :CXL_DOE_TAB_ENT_MAX , + }; + +memcpy (doe_cap->read_mbox, &rsp,sizeof (rsp)); +memcpy (doe_cap->read_mbox +DIV_ROUND_UP (sizeof (rsp),DWORD_BYTE ), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + +return true; +} + +static uint32_t ct3d_config_read (PCIDevice *pci_dev,uint32_t addr,int size) +{ +CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev); +uint32_t val; + +if (pcie_doe_read_config (&ct3d->doe_cdat, addr, size, &val)) { +return val; + } + +return pci_default_read_config (pci_dev, addr, size); +} + +static void ct3d_config_write (PCIDevice *pci_dev,uint32_t addr,uint32_t val, +int size) +{ +CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev); + +pcie_doe_write_config (&ct3d->doe_cdat, addr, val, size); +pci_default_write_config (pci_dev, addr, val, size); +} +/* * Null value of all Fs suggested by IEEE RA guidelines for use of * EU, OUI and CID
@@ -140,6 +389,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)return true; } +static DOEProtocol doe_cdat_prot[] = { + {CXL_VENDOR_ID ,CXL_DOE_TABLE_ACCESS , cxl_doe_cdat_rsp }, + { } +}; +static void ct3_realize (PCIDevice *pci_dev,Error **errp) {CXLType3Dev *ct3d =CXL_TYPE3 (pci_dev);
@@ -189,6 +443,14 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)for (i =0 ; i < msix_num; i++) {msix_vector_use (pci_dev, i); } + +/* DOE Initailization */ +pcie_doe_init (pci_dev, &ct3d->doe_cdat,0x190 , doe_cdat_prot, true,0 ); + + cxl_cstate->cdat.build_cdat_table = ct3_build_cdat_table; + cxl_cstate->cdat.free_cdat_table = ct3_free_cdat_table; + cxl_cstate->cdat.private = ct3d; +cxl_doe_cdat_init (cxl_cstate, errp); }static void ct3_exit (PCIDevice *pci_dev)
@@ -197,6 +459,7 @@ static void ct3_exit(PCIDevice *pci_dev)CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;ComponentRegisters *regs = &cxl_cstate->crb; +cxl_doe_cdat_release (cxl_cstate);g_free (regs->special_ops);address_space_destroy (&ct3d->hostmem_as); }
@@ -296,6 +559,7 @@ static Property ct3_props[] = {DEFINE_PROP_LINK ("lsa" , CXLType3Dev, lsa,TYPE_MEMORY_BACKEND , HostMemoryBackend *),DEFINE_PROP_UINT64 ("sn" , CXLType3Dev, sn,UI64_NULL ), +DEFINE_PROP_STRING ("cdat" , CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_END_OF_LIST(), };
@@ -361,6 +625,9 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->device_id =0xd93 ;/* LVF for now */ pc->revision =1 ; + pc->config_write = ct3d_config_write; + pc->config_read = ct3d_config_read; +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;
Signature · 1 line
2.37.2
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };
// ...
hw/mem/cxl_type3.c line 43
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }
the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using CT3_CDAT_NUM_ENTRIES directly in the function that allocates the cdat buffer itself. I do understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.
Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)
to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}
s/volatile/nonvolatile
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Quoted 19 lines · Gregory Price
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
Signed-off-by Gregory Price <[email protected]>hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };// ...
Quoted 7 lines
hw/mem/cxl_type3.c line 43
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using
CT3_CDAT_NUM_ENTRIESdirectly in the function that allocates the cdat buffer itself.
Yes I think I agree here.
Quoted 24 lines
I do
understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}s/volatile/nonvolatile
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Quoted 24 lines · On Thu, 13 Oct 2022 13:09:26 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 23 lines · Gregory Price
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
Signed-off-by Gregory Price <[email protected]>hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };// ...
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }
Quoted 6 lines
Quoted 3 lines
the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using
CT3_CDAT_NUM_ENTRIESdirectly in the function that allocates the cdat buffer itself.Yes I think I agree here.
Ok, seems a consensus against having this local.
I can do this for now and then revisit if / when things become more complex and this becomes not global. I guess a potential case of premature flexibility.
Jonathan
Quoted 26 lines
Quoted 25 lines
I do
understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}s/volatile/nonvolatile
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
If you're expecting this to be dynamic in size in the future, then maybe what we really want here is a function
static int ct3_get_cdat_size(CXLType3Dev ct3d) {
/* TODO: Dynamic sizing calculations */
return CT3_CDAT_NUM_ENTRIES;
}Quoted 68 lines · Jonathan Cameron
Quoted 24 lines · On Thu, 13 Oct 2022 13:09:26 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 23 lines · Gregory Price
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
Signed-off-by Gregory Price <[email protected]>hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };// ...
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }Quoted 6 lines
Quoted 3 lines
the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using
CT3_CDAT_NUM_ENTRIESdirectly in the function that allocates the cdat buffer itself.Yes I think I agree here.
Ok, seems a consensus against having this local.
I can do this for now and then revisit if / when things become more complex and this becomes not global. I guess a potential case of premature flexibility.
Jonathan
Quoted 26 lines
Quoted 25 lines
I do
understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}s/volatile/nonvolatile
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
Quoted 7 lines · On Thu, 13 Oct 2022 13:32:26 -0400 Gregory Price <[email protected]>…
If you're expecting this to be dynamic in size in the future, then maybe what we really want here is a function
static int ct3_get_cdat_size(CXLType3Dev ct3d) { /* TODO: Dynamic sizing calculations */ return CT3_CDAT_NUM_ENTRIES; }
Perhaps, though that suffers from lack of locality and clearly flagging that the enum is definitely not global.
Anyhow, lets argue that mechanism when it's needed :)
Jonathan
Quoted 71 lines · Jonathan Cameron
Quoted 68 lines
Quoted 24 lines · On Thu, 13 Oct 2022 13:09:26 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 23 lines · Gregory Price
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
Signed-off-by Gregory Price <[email protected]>hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };// ...
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }Quoted 6 lines
Quoted 3 lines
the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using
CT3_CDAT_NUM_ENTRIESdirectly in the function that allocates the cdat buffer itself.Yes I think I agree here.
Ok, seems a consensus against having this local.
I can do this for now and then revisit if / when things become more complex and this becomes not global. I guess a potential case of premature flexibility.
Jonathan
Quoted 26 lines
Quoted 25 lines
I do
understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}s/volatile/nonvolatile
Re: [PATCH v8 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange
In reply to Gregory PriceQuoted 29 lines · On Thu, 13 Oct 2022 12:26:58 -0400 Gregory Price <[email protected]>…
Other than the nitpicks below, lgtm. Not sure if you need a sign off from me given the contributions:
Signed-off-by Gregory Price <[email protected]>hw/mem/cxl_type3.c line 33
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+/* If no cdat_table == NULL returns number of entries */ +static int ct3_build_cdat_entries_for_mr (CDATSubHeader **cdat_table, +int dsmad_handle,MemoryRegion *mr) +{ +enum { +DSMAS , +DSLBIS0 , +DSLBIS1 , +DSLBIS2 , +DSLBIS3 , +DSEMTS , +NUM_ENTRIES + };// ...
hw/mem/cxl_type3.c line 43
Quoted 3 lines
hw/mem/cxl_type3.c +3
-0+if (!cdat_table) { +return NUM_ENTRIES ; + }the only thing that i would recommend is making this enum global (maybe renaming them CT3_CDAT_[ENTRY_NAME]) and using
CT3_CDAT_NUM_ENTRIESdirectly in the function that allocates the cdat buffer itself. I do understand the want to keep the enum and the code creating the entries co-located though, so i'm just nitpicking here i guess.
Quoted 6 lines
Generally I dislike the pattern of passing a NULL into a function to get configuration data, and then calling that function again. This function wants to be named...
ct3_build_cdat_entries_for_mr_or_get_table_size_if_cdat_is_null(...)
I agree it's a messy pattern, but as things become dynamic I'd expect we'll have a bunch of checks that really need to be in that function and would be replicated at the caller to calculate the size of the array.
I'm expecting this code to get more complex over time.
For example adding memory-side cache support based on commandline parameters. Once we do that, the enum will almost certainly no longer be global as we'll have a bunch of other entries (potentially different numbers for each region). Whether that is done with calls to a separate function, or by changing this one isn't clear to me yet.
Quoted 7 lines
to accurately describe what it does. Just kinda feels like an extra function call for no reason.
But either way, it works, this is just a nitpick on my side.
Quoted 9 lines
hw/mem/cxl_type3.c line 163
Quoted 7 lines
hw/mem/cxl_type3.c +7
-0+static int ct3_build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ + g_autofree CDATSubHeader **table = NULL; +CXLType3Dev *ct3d = priv; +MemoryRegion *volatile_mr; +/* ... snip ... */ +}s/volatile/nonvolatile
Doh. I'll update the gitlab tree in a minute for this, but leave sending out the updated version for a few days to let others take a look if they wish.
In meantime, regression in mainline kernel against CXL qemu emulation. Bisection with 8 steps to go. *sigh*
Jonathan
PATCH v8 5/5 hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE
In reply to Jonathan CameronThis Data Object Exchange Mailbox allows software to query the latency and bandwidth between ports on the switch. For now only provide information on routes between the upstream port and each downstream port (not p2p).
--
Changes since v7:
- Moved to enum for cdat_table elements to make sizing explicit.
- Handle memory allocation failures that would have been ignored previously.
--- hw/pci-bridge/cxl_upstream.c | 194 ++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_cdat.h | 1 + 2 files changed, 194 insertions(+), 1 deletion(-)
hw/pci-bridge/cxl_upstream.c +193 -1
@@ -10,11 +10,12 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "hw/qdev-properties.h" #include "hw/pci/msi.h" #include "hw/pci/pcie.h" #include "hw/pci/pcie_port.h" -#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 1 +#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 2#define CXL_UPSTREAM_PORT_MSI_OFFSET 0x70#define CXL_UPSTREAM_PORT_PCIE_CAP_OFFSET 0x90
@@ -28,6 +29,7 @@ typedef struct CXLUpstreamPort {/*< public >*/ CXLComponentState cxl_cstate; +DOECap doe_cdat; } CXLUpstreamPort;CXLComponentState *cxl_usp_to_cstate (CXLUpstreamPort *usp)
@@ -60,6 +62,9 @@ static void cxl_usp_dvsec_write_config(PCIDevice *dev, uint32_t addr,static void cxl_usp_write_config (PCIDevice *d,uint32_t address,uint32_t val,int len) { +CXLUpstreamPort *usp =CXL_USP (d); + +pcie_doe_write_config (&usp->doe_cdat, address, val, len);pci_bridge_write_config (d, address, val, len);pcie_cap_flr_write_config (d, address, val, len);pcie_aer_write_config (d, address, val, len);
@@ -67,6 +72,18 @@ static void cxl_usp_write_config(PCIDevice *d, uint32_t address,cxl_usp_dvsec_write_config (d, address, val, len); } +static uint32_t cxl_usp_read_config (PCIDevice *d,uint32_t address,int len) +{ +CXLUpstreamPort *usp =CXL_USP (d); +uint32_t val; + +if (pcie_doe_read_config (&usp->doe_cdat, address, len, &val)) { +return val; + } + +return pci_default_read_config (d, address, len); +} +static void latch_registers (CXLUpstreamPort *usp) {uint32_t *reg_state = usp->cxl_cstate.crb.cache_mem_registers;
@@ -119,6 +136,166 @@ static void build_dvsecs(CXLComponentState *cxl) REG_LOC_DVSEC_REVID, dvsec); } +static bool cxl_doe_cdat_rsp (DOECap *doe_cap) +{ +CDATObject *cdat = &CXL_USP (doe_cap->pdev)->cxl_cstate.cdat; +uint16_t ent; +void *base; +uint32_t len; +CDATReq *req =pcie_doe_get_write_mbox_ptr (doe_cap); +CDATRsp rsp; + +cxl_doe_cdat_update (&CXL_USP (doe_cap->pdev)->cxl_cstate, &error_fatal); +assert (cdat->entry_len); + +/* Discard if request length mismatched */ +if (pcie_doe_get_obj_len (req) < +DIV_ROUND_UP (sizeof (CDATReq),sizeof (uint32_t ))) { +return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp ) { + .header = { + .vendor_id =CXL_VENDOR_ID , + .data_obj_type =CXL_DOE_TABLE_ACCESS , + .reserved =0x0 , + .length =DIV_ROUND_UP ((sizeof (rsp) + len),sizeof (uint32_t )), + }, + .rsp_code =CXL_DOE_TAB_RSP , + .table_type =CXL_DOE_TAB_TYPE_CDAT , + .entry_handle = (ent < cdat->entry_len -1 ) ? + ent +1 :CXL_DOE_TAB_ENT_MAX , + }; + +memcpy (doe_cap->read_mbox, &rsp,sizeof (rsp)); +memcpy (doe_cap->read_mbox +DIV_ROUND_UP (sizeof (rsp),sizeof (uint32_t )), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + +return true; +} + +static DOEProtocol doe_cdat_prot[] = { + {CXL_VENDOR_ID ,CXL_DOE_TABLE_ACCESS , cxl_doe_cdat_rsp }, + { } +}; + +static int build_cdat_table (CDATSubHeader ***cdat_table,void *priv) +{ +enum { +SSLBIS_LATENCY , +SSLBIS_BANDWIDTH , +NUM_ENTRIES + }; + g_autofree CDATSslbis *sslbis_latency = NULL; + g_autofree CDATSslbis *sslbis_bandwidth = NULL; +CXLUpstreamPort *us =CXL_USP (priv); +PCIBus *bus = &PCI_BRIDGE (us)->sec_bus; +int devfn, sslbis_size, i; +int count =0 ; +uint16_t port_ids[256 ]; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; +PCIEPort *port; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } + +/* +* Whilst the PCI express spec doesn't allow anything other than +* downstream ports on this bus, let us be a little paranoid +*/ +if (!object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { +continue ; + } + + port =PCIE_PORT (d); + port_ids[count] = port->port; + count++; + } + +/* May not yet have any ports - try again later */ +if (count ==0 ) { +return 0 ; + } + + sslbis_size =sizeof (CDATSslbis) +sizeof (*sslbis_latency->sslbe) * count; + sslbis_latency =g_malloc (sslbis_size); +if (!sslbis_latency) { +return -ENOMEM ; + } + *sslbis_latency = (CDATSslbis ) { + .sslbis_header = { + .header = { + .type =CDAT_TYPE_SSLBIS , + .length = sslbis_size, + }, + .data_type =HMATLB_DATA_TYPE_ACCESS_LATENCY , + .entry_base_unit =10000 , + }, + }; + +for (i =0 ; i < count; i++) { + sslbis_latency->sslbe[i] = (CDATSslbe ) { + .port_x_id =CDAT_PORT_ID_USP , + .port_y_id = port_ids[i], + .latency_bandwidth =15 ,/* 150ns */ + }; + } + + sslbis_bandwidth =g_malloc (sslbis_size); +if (!sslbis_bandwidth) { +return 0 ; + } + *sslbis_bandwidth = (CDATSslbis ) { + .sslbis_header = { + .header = { + .type =CDAT_TYPE_SSLBIS , + .length = sslbis_size, + }, + .data_type =HMATLB_DATA_TYPE_ACCESS_BANDWIDTH , + .entry_base_unit =1000 , + }, + }; + +for (i =0 ; i < count; i++) { + sslbis_bandwidth->sslbe[i] = (CDATSslbe ) { + .port_x_id =CDAT_PORT_ID_USP , + .port_y_id = port_ids[i], + .latency_bandwidth =16 ,/* 16 GB/s */ + }; + } + + *cdat_table =g_malloc0 (sizeof (*cdat_table) *NUM_ENTRIES ); +if (!*cdat_table) { +return -ENOMEM ; + } + +/* Header always at start of structure */ + (*cdat_table)[SSLBIS_LATENCY ] =g_steal_pointer (&sslbis_latency); + (*cdat_table)[SSLBIS_BANDWIDTH ] =g_steal_pointer (&sslbis_bandwidth); + +return NUM_ENTRIES ; +} + +static void free_default_cdat_table (CDATSubHeader **cdat_table,int num, +void *priv) +{ +int i; + +for (i =0 ; i < num; i++) { +g_free (cdat_table[i]); + } +g_free (cdat_table); +} +static void cxl_usp_realize (PCIDevice *d,Error **errp) {PCIEPort *p =PCIE_PORT (d);
@@ -161,6 +338,14 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp)PCI_BASE_ADDRESS_MEM_TYPE_64 , component_bar); +pcie_doe_init (d, &usp->doe_cdat, cxl_cstate->dvsec_offset, doe_cdat_prot, + true,1 ); + + cxl_cstate->cdat.build_cdat_table = build_cdat_table; + cxl_cstate->cdat.free_cdat_table = free_default_cdat_table; + cxl_cstate->cdat.private = d; +cxl_doe_cdat_init (cxl_cstate, errp); +return ; err_cap:
@@ -179,6 +364,11 @@ static void cxl_usp_exitfn(PCIDevice *d)pci_bridge_exitfn (d); } +static Property cxl_upstream_props[] = { +DEFINE_PROP_STRING ("cdat" , CXLUpstreamPort, cxl_cstate.cdat.filename), +DEFINE_PROP_END_OF_LIST () +}; +static void cxl_upstream_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);
@@ -186,6 +376,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data) k->is_bridge = true; k->config_write = cxl_usp_write_config; + k->config_read = cxl_usp_read_config; k->realize = cxl_usp_realize; k->exit = cxl_usp_exitfn; k->vendor_id =0x19e5 ;/* Huawei */
@@ -194,6 +385,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data)set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); dc->desc ="CXL Switch Upstream Port" ; dc->reset = cxl_usp_reset; +device_class_set_props (dc, cxl_upstream_props); }static const TypeInfo cxl_usp_info = {
include/hw/cxl/cxl_cdat.h +1 -0
@@ -131,6 +131,7 @@ typedef struct CDATSslbisHeader {uint64_t entry_base_unit; }QEMU_PACKED CDATSslbisHeader; +#define CDAT_PORT_ID_USP 0x100/* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ typedef struct CDATSslbe { uint16_t port_x_id;
Signature · 1 line
2.37.2