hw/cxl: Move CXL emulation options and state to machines.
v2 cover hw/cxl: Move CXL emulation options and state to machines.
Changes since v1 (thanks to Paolo Bonzini)
* Update 'description' of cxl-fmw as suggested to mention it's an array.
* Add a wrapper cxl_hook_up_pxb_registers() to cxl-host.c as it'll be common for all machines using CXL with PXB.
Run through the CI at:
https://gitlab.com/jic23/qemu/-/pipelines/553257456
V1 Cover letter:
Currently only machine with CXL support upstream is i386/pc but arm/virt patches have been posted and once this is merged an updated series will follow. Switch support is queued behind this as well because they both include documentation updates.
Paolo Bonzini highlighted a couple of issues with the current CXL emulation code.
* Top level parameter rather than machine for fixed memory windows
The --cxl-fixed-memory-window top level command line parameters won't play well with efforts to make it possible to instantiate entire machines via RPC. Better to move these to be machine configuration. This change is relatively straight forward, but does result in very long command lines (cannot break fixed window setup into multiple -M entries).
* Move all CXL stuff to machine specific code and helpers
To simplify the various interactions between machine setup and host bridges etc, currently various CXL steps are called from the generic core/machine.c and softmmu/vl.c + there are CXL elements in MachineState.
Much of this is straight forward to do with one exception:
The CXL pci_expander_bridge host bridges require MMIO register space. This series does this by walking the bus and filling the register space in via the machine_done callback. This is similar to the walk done for identifying host bridges in the ACPI building code but it is rather ugly and postpones rejection of PXB_CXL instances where cxl=off (default).
All comments welcome, but the first patch at least changes the command-line so to avoid have to add backwards compatibility code, it would be great to merge that before 7.1 is released.
Thanks,
Jonathan
Jonathan Cameron (8):
hw/cxl: Make the CXL fixed memory window setup a machine parameter. hw/acpi/cxl: Pass in the CXLState directly rather than MachineState hw/cxl: Push linking of CXL targets into i386/pc rather than in
machine.c
tests/acpi: Allow modification of q35 CXL CEDT table. pci/pci_expander_bridge: For CXL HB delay the HB register memory
region setup.
tests/acpi: Update q35/CEDT.cxl for new memory addresses. hw/cxl: Move the CXLState from MachineState to machine type specific
state.
hw/machine: Drop cxl_supported flag as no longer useful
docs/system/devices/cxl.rst | 4 +- hw/acpi/cxl.c | 9 +- hw/core/machine.c | 28 ------ hw/cxl/cxl-host-stubs.c | 9 +- hw/cxl/cxl-host.c | 100 ++++++++++++++++++-- hw/i386/acpi-build.c | 8 +- hw/i386/pc.c | 31 +++--- hw/pci-bridge/meson.build | 5 +- hw/pci-bridge/pci_expander_bridge.c | 32 ++++--- hw/pci-bridge/pci_expander_bridge_stubs.c | 14 +++ include/hw/acpi/cxl.h | 5 +- include/hw/boards.h | 3 +- include/hw/cxl/cxl.h | 9 +- include/hw/cxl/cxl_host.h | 23 +++++ include/hw/i386/pc.h | 2 + include/hw/pci-bridge/pci_expander_bridge.h | 12 +++ qapi/machine.json | 13 +++ softmmu/vl.c | 46 --------- tests/data/acpi/q35/CEDT.cxl | Bin 184 -> 184 bytes tests/qtest/bios-tables-test.c | 4 +- tests/qtest/cxl-test.c | 4 +- 21 files changed, 222 insertions(+), 139 deletions(-) create mode 100644 hw/pci-bridge/pci_expander_bridge_stubs.c create mode 100644 include/hw/cxl/cxl_host.h create mode 100644 include/hw/pci-bridge/pci_expander_bridge.h
Signature · 1 line
2.32.0
PATCH v2 1/8 hw/cxl: Make the CXL fixed memory window setup a machine parameter.
Paolo Bonzini requested this change to simplify the ongoing effort to allow machine setup entirely via RPC.
Includes shortening the command line form cxl-fixed-memory-window to cxl-fmw as the command lines are extremely long even with this change.
The json change is needed to ensure that there is a CXLFixedMemoryWindowOptionsList even though the actual element in the json is never used. Similar to existing SgxEpcProperties.
Update cxl-test and bios-tables-test to reflect new parameters.
--- docs/system/devices/cxl.rst | 4 +- hw/core/machine.c | 22 --------- hw/cxl/cxl-host-stubs.c | 6 +-- hw/cxl/cxl-host.c | 72 +++++++++++++++++++++++++++-- hw/i386/pc.c | 3 ++ hw/pci-bridge/pci_expander_bridge.c | 2 +- include/hw/boards.h | 1 + include/hw/cxl/cxl.h | 7 +-- include/hw/cxl/cxl_host.h | 21 +++++++++ qapi/machine.json | 13 ++++++ softmmu/vl.c | 44 ------------------ tests/qtest/bios-tables-test.c | 4 +- tests/qtest/cxl-test.c | 4 +- 13 files changed, 116 insertions(+), 87 deletions(-)
docs/system/devices/cxl.rst +2 -2
@@ -251,7 +251,7 @@ A very simple setup with just one directly attached CXL Type 3 device:: -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ - -cxl-fixed-memory-window targets.0=cxl.1,size=4G + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way interleave across 2 CXL host bridges. Each host bridge has 2 CXL Root Ports, with
@@ -277,7 +277,7 @@ the CXL Type3 device directly attached (no switches).:: -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ - -cxl-fixed-memory-window targets.0=cxl.1,targets.1=cxl.2,size=4G,interleave-granularity=8k + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k Kernel Configuration Options ----------------------------
hw/core/machine.c +0 -22
@@ -628,20 +628,6 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value, nvdimms_state->persistence_string =g_strdup (value); } -static bool machine_get_cxl (Object *obj,Error **errp) -{ -MachineState *ms =MACHINE (obj); - -return ms->cxl_devices_state->is_enabled; -} - -static void machine_set_cxl (Object *obj,bool value,Error **errp) -{ -MachineState *ms =MACHINE (obj); - - ms->cxl_devices_state->is_enabled = value; -} -void machine_class_allow_dynamic_sysbus_dev (MachineClass *mc,const char *type) {QAPI_LIST_PREPEND (mc->allowed_dynamic_sysbus_devices,g_strdup (type));
@@ -928,8 +914,6 @@ static void machine_class_init(ObjectClass *oc, void *data) mc->default_ram_size =128 * MiB; mc->rom_file_has_mr = true; -/* Few machines support CXL, so default to off */ - mc->cxl_supported = false;/* numa node memory size aligned on 8MB by default. * On Linux, each node's border has to be 8MB aligned */
@@ -1091,13 +1075,7 @@ static void machine_initfn(Object *obj) }if (mc->cxl_supported) { -Object *obj =OBJECT (ms); - ms->cxl_devices_state =g_new0 (CXLState,1 ); -object_property_add_bool (obj,"cxl" , machine_get_cxl, machine_set_cxl); -object_property_set_description (obj,"cxl" , -"Set on/off to enable/disable " -"CXL instantiation" ); }if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) {
hw/cxl/cxl-host-stubs.c +2 -4
@@ -6,11 +6,9 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "hw/cxl/cxl.h" - -void cxl_fixed_memory_window_config (MachineState *ms, -CXLFixedMemoryWindowOptions *object, -Error **errp) {}; +#include "hw/cxl/cxl_host.h" void cxl_fixed_memory_window_link_targets (Error **errp) {}; +void cxl_machine_init (Object *obj,CXLState *state) {};const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c +67 -5
92 lines not on this page · raw message
hw/i386/pc.c +3 -0
@@ -76,6 +76,7 @@#include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" #include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/qapi-visit-machine.h"
@@ -1682,6 +1683,7 @@ static void pc_machine_set_max_fw_size(Object *obj, Visitor *v,static void pc_machine_initfn (Object *obj) {PCMachineState *pcms =PC_MACHINE (obj); +MachineState *ms =MACHINE (obj);#ifdef CONFIG_VMPORT pcms->vmport =ON_OFF_AUTO_AUTO ;
@@ -1706,6 +1708,7 @@ static void pc_machine_initfn(Object *obj) pcms->pcspk =isa_new (TYPE_PC_SPEAKER );object_property_add_alias (OBJECT (pcms),"pcspk-audiodev" ,OBJECT (pcms->pcspk),"audiodev" ); +cxl_machine_init (obj, ms->cxl_devices_state); }static void pc_machine_reset (MachineState *machine)
hw/pci-bridge/pci_expander_bridge.c +1 -1
@@ -468,7 +468,7 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" );return ; } -if (!ms->cxl_devices_state->is_enabled) { +if (!ms->cxl_devices_state || !ms->cxl_devices_state->is_enabled) {error_setg (errp,"Machine does not have cxl=on" );return ; }
include/hw/boards.h +1 -0
@@ -362,6 +362,7 @@ struct MachineState {struct NVDIMMState *nvdimms_state;struct CXLState *cxl_devices_state;struct NumaState *numa_state; +CXLFixedMemoryWindowOptionsList *cfmws_list; };#define DEFINE_MACHINE (namestr, machine_initfn) \
include/hw/cxl/cxl.h +2 -5
@@ -12,6 +12,7 @@#include "qapi/qapi-types-machine.h" +#include "qapi/qapi-visit-machine.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pci_host.h" #include "cxl_pci.h"
@@ -40,6 +41,7 @@ typedef struct CXLState {MemoryRegion host_mr;unsigned int next_mr_idx;GList *fixed_windows; +CXLFixedMemoryWindowOptionsList *cfmw_list; } CXLState;struct CXLHost {
@@ -51,11 +53,6 @@ struct CXLHost {#define TYPE_PXB_CXL_HOST "pxb-cxl-host" OBJECT_DECLARE_SIMPLE_TYPE(CXLHost, PXB_CXL_HOST) -void cxl_fixed_memory_window_config (MachineState *ms, -CXLFixedMemoryWindowOptions *object, -Error **errp);void cxl_fixed_memory_window_link_targets (Error **errp); -extern const MemoryRegionOps cfmws_ops; -#endif
include/hw/cxl/cxl_host.h new file +21 -0
@@ -0,0 +1,21 @@ +/* +* QEMU CXL Host Setup +* +* Copyright (c) 2022 Huawei +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" + +#ifndef CXL_HOST_H +#define CXL_HOST_H + +void cxl_machine_init (Object *obj,CXLState *state); + +extern const MemoryRegionOps cfmws_ops; + +#endif
qapi/machine.json +13 -0
@@ -523,6 +523,19 @@ '*interleave-granularity': 'size', 'targets': ['str'] }} +## +# @CXLFMWProperties: +# +# List of CXL Fixed Memory Windows. +# +# @cxl-fmw: List of CXLFixedMemoryWindowOptions +# +# Since 7.1 +## +{ 'struct' : 'CXLFMWProperties', + 'data': { 'cxl-fmw': ['CXLFixedMemoryWindowOptions'] } +} + ## # @X86CPURegister32: #
softmmu/vl.c +0 -44
80 lines not on this page · raw message
tests/qtest/bios-tables-test.c +2 -2
@@ -1567,8 +1567,8 @@ static void test_acpi_q35_cxl(void)" -device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3" " -device cxl-rp,port=1,bus=cxl.2,id=rp4,chassis=0,slot=6" " -device cxl-type3,bus=rp4,memdev=cxl-mem4,lsa=lsa4" -" -cxl-fixed-memory-window targets.0=cxl.1,size=4G,interleave-granularity=8k" -" -cxl-fixed-memory-window targets.0=cxl.1,targets.1=cxl.2,size=4G,interleave-granularity=8k" , +" -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k," +"cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.targets.1=cxl.2,cxl-fmw.1.size=4G,cxl-fmw.1.interleave-granularity=8k" , tmp_path, tmp_path, tmp_path, tmp_path, tmp_path, tmp_path, tmp_path, tmp_path);test_acpi_one (params, &data);
tests/qtest/cxl-test.c +2 -2
@@ -10,12 +10,12 @@#define QEMU_PXB_CMD "-machine q35,cxl=on " \ "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-cxl-fixed-memory-window targets.0=cxl.0,size=4G " + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G "#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ - "-cxl-fixed-memory-window targets.0=cxl.0,targets.1=cxl.1,size=4G " + "-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=4G "#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
Signature · 1 line
2.32.0
Re: [PATCH v2 1/8] hw/cxl: Make the CXL fixed memory window setup a machine parameter.
Quoted 7 lines · Jonathan Cameron
Paolo Bonzini requested this change to simplify the ongoing effort to allow machine setup entirely via RPC.
Includes shortening the command line form cxl-fixed-memory-window to cxl-fmw as the command lines are extremely long even with this change.
Quoted 8 lines
The json change is needed to ensure that there is a CXLFixedMemoryWindowOptionsList even though the actual element in the json is never used. Similar to existing SgxEpcProperties.
Update cxl-test and bios-tables-test to reflect new parameters.
Signed-off-by Jonathan Cameron <[email protected]>
One thing missing however is updating qemu-options.hx - maybe fold in the below?
Thanks!
----8<-------
qemu-options.hx +33 -38
@@ -127,10 +127,43 @@ SRST ERST DEF("M", HAS_ARG, QEMU_OPTION_M, + " cxl-fmw.0.targets.0=firsttarget,cxl-fmw.0.targets.1=secondtarget,cxl-fmw.0.size=size[,cxl-fmw.0.interleave-granularity=granularity]\n" " sgx-epc.0.memdev=memid,sgx-epc.0.node=numaid\n", QEMU_ARCH_ALL) SRST +``cxl-fmw.0.targets.0=firsttarget,cxl-fmw.0.targets.1=secondtarget,cxl-fmw.0.size=size[,cxl-fmw.0.interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets.X=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. X is the + target index from 0. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=128G,cxl-fmw.0.interleave-granularity=512k + ``sgx-epc.0.memdev=@var{memid},sgx-epc.0.node=@var{numaid}`` Define an SGX EPC section. ERST
@@ -467,44 +500,6 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST -DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, - "-cxl-fixed-memory-window targets.0=firsttarget,targets.1=secondtarget,size=size[,interleave-granularity=granularity]\n", - QEMU_ARCH_ALL) -SRST -``-cxl-fixed-memory-window targets.0=firsttarget,targets.1=secondtarget,size=size[,interleave-granularity=granularity]`` - Define a CXL Fixed Memory Window (CFMW). - - Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. - - They are regions of Host Physical Addresses (HPA) on a system which - may be interleaved across one or more CXL host bridges. The system - software will assign particular devices into these windows and - configure the downstream Host-managed Device Memory (HDM) decoders - in root ports, switch ports and devices appropriately to meet the - interleave requirements before enabling the memory devices. - - ``targets.X=firsttarget`` provides the mapping to CXL host bridges - which may be identified by the id provied in the -device entry. - Multiple entries are needed to specify all the targets when - the fixed memory window represents interleaved memory. X is the - target index from 0. - - ``size=size`` sets the size of the CFMW. This must be a multiple of - 256MiB. The region will be aligned to 256MiB but the location is - platform and configuration dependent. - - ``interleave-granularity=granularity`` sets the granularity of - interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB - 4096KiB, 8192KiB and 16384KiB granularities supported. - - Example: - - :: - - -cxl-fixed-memory-window targets.0=cxl.0,targets.1=cxl.1,size=128G,interleave-granularity=512k - -ERST - DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)
Re: [PATCH v2 1/8] hw/cxl: Make the CXL fixed memory window setup a machine parameter.
Quoted 14 lines · On Tue, 7 Jun 2022 15:37:02 -0700 Davidlohr Bueso <[email protected]> wrote:
Quoted 12 lines · Jonathan Cameron
Paolo Bonzini requested this change to simplify the ongoing effort to allow machine setup entirely via RPC.
Includes shortening the command line form cxl-fixed-memory-window to cxl-fmw as the command lines are extremely long even with this change.
The json change is needed to ensure that there is a CXLFixedMemoryWindowOptionsList even though the actual element in the json is never used. Similar to existing SgxEpcProperties.
Quoted 5 lines
Quoted 3 lines
Update cxl-test and bios-tables-test to reflect new parameters.
Signed-off-by Jonathan Cameron <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>
Thanks.
Quoted 3 lines
One thing missing however is updating qemu-options.hx - maybe fold in the below?
Excellent point.
The original patch set has been rolling so long I'd forgotten we had documentation in qemu-options.hx.
One comment inline though...
Quoted 10 lines
Thanks!
----8<-------
oqemu-options.hx +0
-0
qemu-options.hx SRST · line 130
Quoted 4 lines
qemu-options.hx +1
-0@@ -127,10 +127,43 @@ SRST ERST DEF("M", HAS_ARG, QEMU_OPTION_M, + " cxl-fmw.0.targets.0=firsttarget,cxl-fmw.0.targets.1=secondtarget,cxl-fmw.0.size=size[,cxl-fmw.0.interleave-granularity=granularity]\n"
This _M entry is odd, given it's just short hand for _machine. So I'm thinking it makes more sense to document this lot in the "machine" block further up the doc. That particular entry has a different style to this one, so I'll modify this inline with existing entries there.
Possibly someone should move the sgx entry over to machine as well. +Cc Sean Chistopherson and Yan Zhong as that sgx-epc entry is from their patch set. A quick glance through discussion of that patch didn't throw up a reason for doing it as a separate entry, but I haven't dug deep.
Quoted 84 lines
qemu-options.hx +32
-38" sgx-epc.0.memdev=memid,sgx-epc.0.node=numaid\n", QEMU_ARCH_ALL) SRST +``cxl-fmw.0.targets.0=firsttarget,cxl-fmw.0.targets.1=secondtarget,cxl-fmw.0.size=size[,cxl-fmw.0.interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets.X=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. X is the + target index from 0. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.targets.1=cxl.1,cxl-fmw.0.size=128G,cxl-fmw.0.interleave-granularity=512k + ``sgx-epc.0.memdev=@var{memid},sgx-epc.0.node=@var{numaid}`` Define an SGX EPC section. ERST@@ -467,44 +500,6 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST -DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, - "-cxl-fixed-memory-window targets.0=firsttarget,targets.1=secondtarget,size=size[,interleave-granularity=granularity]\n", - QEMU_ARCH_ALL) -SRST -``-cxl-fixed-memory-window targets.0=firsttarget,targets.1=secondtarget,size=size[,interleave-granularity=granularity]`` - Define a CXL Fixed Memory Window (CFMW). - - Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. - - They are regions of Host Physical Addresses (HPA) on a system which - may be interleaved across one or more CXL host bridges. The system - software will assign particular devices into these windows and - configure the downstream Host-managed Device Memory (HDM) decoders - in root ports, switch ports and devices appropriately to meet the - interleave requirements before enabling the memory devices. - - ``targets.X=firsttarget`` provides the mapping to CXL host bridges - which may be identified by the id provied in the -device entry. - Multiple entries are needed to specify all the targets when - the fixed memory window represents interleaved memory. X is the - target index from 0. - - ``size=size`` sets the size of the CFMW. This must be a multiple of - 256MiB. The region will be aligned to 256MiB but the location is - platform and configuration dependent. - - ``interleave-granularity=granularity`` sets the granularity of - interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB - 4096KiB, 8192KiB and 16384KiB granularities supported. - - Example: - - :: - - -cxl-fixed-memory-window targets.0=cxl.0,targets.1=cxl.1,size=128G,interleave-granularity=512k - -ERST - DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)
PATCH v2 2/8 hw/acpi/cxl: Pass in the CXLState directly rather than MachineState
In reply to Jonathan CameronRefactoring step on path to moving all CXL state out of MachineState.
--- hw/acpi/cxl.c | 9 ++++----- hw/i386/acpi-build.c | 4 ++-- include/hw/acpi/cxl.h | 5 +++-- 3 files changed, 9 insertions(+), 9 deletions(-)
hw/acpi/cxl.c +4 -5
@@ -65,9 +65,8 @@ static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)* Interleave ways encoding in CXL 2.0 ECN: 3, 6, 12 and 16-way memory * interleaving. */ -static void cedt_build_cfmws (GArray *table_data,MachineState *ms) +static void cedt_build_cfmws (GArray *table_data,CXLState *cxls) { -CXLState *cxls = ms->cxl_devices_state;GList *it;for (it = cxls->fixed_windows; it; it = it->next) {
@@ -129,9 +128,9 @@ static int cxl_foreach_pxb_hb(Object *obj, void *opaque)return 0 ; } -void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +void cxl_build_cedt (GArray *table_offsets,GArray *table_data,BIOSLinker *linker,const char *oem_id, -const char *oem_table_id) +const char *oem_table_id,CXLState *cxl_state) {Aml *cedt;AcpiTable table = { .sig ="CEDT" , .rev =1 , .oem_id = oem_id,
@@ -144,7 +143,7 @@ void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,/* reserve space for CEDT header */ object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); -cedt_build_cfmws (cedt->buf, ms); +cedt_build_cfmws (cedt->buf, cxl_state);/* copy AML table into ACPI tables blob and patch header there */ g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len);
hw/i386/acpi-build.c +2 -2
@@ -2712,8 +2712,8 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine) x86ms->oem_id, x86ms->oem_table_id); }if (machine->cxl_devices_state->is_enabled) { -cxl_build_cedt (machine, table_offsets, tables_blob, tables->linker, - x86ms->oem_id, x86ms->oem_table_id); +cxl_build_cedt (table_offsets, tables_blob, tables->linker, + x86ms->oem_id, x86ms->oem_table_id, machine->cxl_devices_state); }acpi_add_table (table_offsets, tables_blob);
include/hw/acpi/cxl.h +3 -2
@@ -19,10 +19,11 @@#define HW_ACPI_CXL_H #include "hw/acpi/bios-linker-loader.h" +#include "hw/cxl/cxl.h" -void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +void cxl_build_cedt (GArray *table_offsets,GArray *table_data,BIOSLinker *linker,const char *oem_id, -const char *oem_table_id); +const char *oem_table_id,CXLState *cxl_state);void build_cxl_osc_method (Aml *dev);#endif
Signature · 1 line
2.32.0
PATCH v2 3/8 hw/cxl: Push linking of CXL targets into i386/pc rather than in machine.c
In reply to Jonathan CameronWhilst here take the oportunity to shorten the function name.
--- hw/cxl/cxl-host-stubs.c | 2 +- hw/cxl/cxl-host.c | 8 +++----- hw/i386/pc.c | 5 +++++ include/hw/cxl/cxl.h | 2 -- include/hw/cxl/cxl_host.h | 1 + softmmu/vl.c | 2 -- 6 files changed, 10 insertions(+), 10 deletions(-)
hw/cxl/cxl-host-stubs.c +1 -1
@@ -8,7 +8,7 @@#include "hw/cxl/cxl.h" #include "hw/cxl/cxl_host.h" -void cxl_fixed_memory_window_link_targets (Error **errp) {}; +void cxl_fmws_link_targets (CXLState *stat,Error **errp) {};void cxl_machine_init (Object *obj,CXLState *state) {};const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c +3 -5
@@ -68,14 +68,12 @@ static void cxl_fixed_memory_window_config(CXLState *cxl_state,return ; } -void cxl_fixed_memory_window_link_targets (Error **errp) +void cxl_fmws_link_targets (CXLState *cxl_state,Error **errp) { -MachineState *ms =MACHINE (qdev_get_machine ()); - -if (ms->cxl_devices_state && ms->cxl_devices_state->fixed_windows) { +if (cxl_state && cxl_state->fixed_windows) {GList *it; -for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { +for (it = cxl_state->fixed_windows; it; it = it->next) {CXLFixedWindow *fw = it->data;int i;
hw/i386/pc.c +5 -0
@@ -732,6 +732,11 @@ void pc_machine_done(Notifier *notifier, void *data)PCMachineState *pcms =container_of (notifier, PCMachineState, machine_done);X86MachineState *x86ms =X86_MACHINE (pcms); +MachineState *ms =MACHINE (pcms); + +if (ms->cxl_devices_state) { +cxl_fmws_link_targets (ms->cxl_devices_state, &error_fatal); + }/* set the number of CPUs */ x86_rtc_set_cpus_count (x86ms->rtc, x86ms->boot_cpus);
include/hw/cxl/cxl.h +0 -2
@@ -53,6 +53,4 @@ struct CXLHost {#define TYPE_PXB_CXL_HOST "pxb-cxl-host" OBJECT_DECLARE_SIMPLE_TYPE(CXLHost, PXB_CXL_HOST) -void cxl_fixed_memory_window_link_targets (Error **errp); -#endif
include/hw/cxl/cxl_host.h +1 -0
@@ -15,6 +15,7 @@#define CXL_HOST_H void cxl_machine_init (Object *obj,CXLState *state); +void cxl_fmws_link_targets (CXLState *stat,Error **errp);extern const MemoryRegionOps cfmws_ops;
softmmu/vl.c +0 -2
@@ -93,7 +93,6 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" -#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif
@@ -2665,7 +2664,6 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); -cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {
Signature · 1 line
2.32.0
PATCH v2 4/8 tests/acpi: Allow modification of q35 CXL CEDT table.
In reply to Jonathan CameronNeeded to allow memory address changes as a result of next patch.
--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)
tests/qtest/bios-tables-test-allowed-diff.h +1 -0
@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/CEDT.cxl",
Signature · 1 line
2.32.0
PATCH v2 5/8 pci/pci_expander_bridge: For CXL HB delay the HB register memory region setup.
In reply to Jonathan CameronAs the CXLState will no long be accessible via MachineState at time of PXB_CXL realization, come back later from the machine specific code to fill in the missing memory region setup. Only at this stage is it possible to check if cxl=on, so that check is moved to this later point.
Note that for multiple host bridges, the allocation order of the register spaces is changed. This will be reflected in ACPI CEDT.
Stubs are added to handle case of CONFIG_PXB=n for machines that call these functions.
The bus walking logic is common to all machines so add a utility function + stub to cxl-host*.
--- hw/cxl/cxl-host-stubs.c | 1 + hw/cxl/cxl-host.c | 20 +++++++++++++ hw/i386/pc.c | 3 ++ hw/pci-bridge/meson.build | 5 +++- hw/pci-bridge/pci_expander_bridge.c | 32 +++++++++++++-------- hw/pci-bridge/pci_expander_bridge_stubs.c | 14 +++++++++ include/hw/cxl/cxl_host.h | 1 + include/hw/pci-bridge/pci_expander_bridge.h | 12 ++++++++ 8 files changed, 75 insertions(+), 13 deletions(-)
hw/cxl/cxl-host-stubs.c +1 -0
@@ -10,5 +10,6 @@void cxl_fmws_link_targets (CXLState *stat,Error **errp) {};void cxl_machine_init (Object *obj,CXLState *state) {}; +void cxl_hook_up_pxb_registers (PCIBus *bus,CXLState *state,Error **errp) {};const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c +20 -0
@@ -20,6 +20,7 @@#include "hw/pci/pci_bridge.h" #include "hw/pci/pci_host.h" #include "hw/pci/pcie_port.h" +#include "hw/pci-bridge/pci_expander_bridge.h" static void cxl_fixed_memory_window_config(CXLState *cxl_state, CXLFixedMemoryWindowOptions *object,
@@ -280,3 +281,22 @@ void cxl_machine_init(Object *obj, CXLState *state)object_property_set_description (obj,"cxl-fmw" ,"CXL Fixed Memory Windows (array)" ); } + +void cxl_hook_up_pxb_registers (PCIBus *bus,CXLState *state,Error **errp) +{ +/* Walk the pci busses looking for pxb busses to hook up */ +if (bus) { + QLIST_FOREACH(bus, &bus->child, sibling) { +if (!pci_bus_is_root (bus)) { +continue ; + } +if (pci_bus_is_cxl (bus)) { +if (!state->is_enabled) { +error_setg (errp,"CXL host bridges present, but cxl=off" ); +return ; + } +pxb_cxl_hook_up_registers (state, bus, errp); + } + } + } +}
hw/i386/pc.c +3 -0
@@ -37,6 +37,7 @@#include "hw/ide.h" #include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" +#include "hw/pci-bridge/pci_expander_bridge.h" #include "hw/nvram/fw_cfg.h" #include "hw/timer/hpet.h" #include "hw/firmware/smbios.h"
@@ -735,6 +736,8 @@ void pc_machine_done(Notifier *notifier, void *data)MachineState *ms =MACHINE (pcms);if (ms->cxl_devices_state) { +cxl_hook_up_pxb_registers (pcms->bus, ms->cxl_devices_state, + &error_fatal);cxl_fmws_link_targets (ms->cxl_devices_state, &error_fatal); }
hw/pci-bridge/meson.build +4 -1
@@ -3,7 +3,8 @@ pci_ss.add(files('pci_bridge_dev.c')) pci_ss.add(when: 'CONFIG_I82801B11', if_true: files('i82801b11.c')) pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c')) pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c')) -pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c')) +pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c'), + if_false: files('pci_expander_bridge_stubs.c')) pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c')) pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c'))
@@ -13,3 +14,5 @@ pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c')) pci_ss.add(when: 'CONFIG_SIMBA', if_true: files('simba.c')) softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss) + +softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('pci_expander_bridge_stubs.c'))
hw/pci-bridge/pci_expander_bridge.c +20 -12
@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/pci-bridge/pci_expander_bridge.h" #include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h"
@@ -186,25 +187,38 @@ static const TypeInfo pxb_host_info = {static void pxb_cxl_realize (DeviceState *dev,Error **errp) { -MachineState *ms =MACHINE (qdev_get_machine ());SysBusDevice *sbd =SYS_BUS_DEVICE (dev);CXLHost *cxl =PXB_CXL_HOST (dev);CXLComponentState *cxl_cstate = &cxl->cxl_cstate;struct MemoryRegion *mr = &cxl_cstate->crb.component_registers; -hwaddr offset;cxl_component_register_block_init (OBJECT (dev), cxl_cstate,TYPE_PXB_CXL_HOST );sysbus_init_mmio (sbd, mr); +} + +/* +* Host bridge realization has no means of knowning state associated +* with a particular machine. As such, it is nececssary to delay +* final setup of the host bridge register space until later in the +* machine bring up. +*/ +void pxb_cxl_hook_up_registers (CXLState *cxl_state,PCIBus *bus,Error **errp) +{ +PXBDev *pxb =PXB_CXL_DEV (pci_bridge_get_device (bus)); +CXLHost *cxl = pxb->cxl.cxl_host_bridge; +CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +struct MemoryRegion *mr = &cxl_cstate->crb.component_registers; +hwaddr offset; - offset =memory_region_size (mr) * ms->cxl_devices_state->next_mr_idx; -if (offset >memory_region_size (&ms->cxl_devices_state->host_mr)) { + offset =memory_region_size (mr) * cxl_state->next_mr_idx; +if (offset >memory_region_size (&cxl_state->host_mr)) {error_setg (errp,"Insufficient space for pxb cxl host register space" );return ; } -memory_region_add_subregion (&ms->cxl_devices_state->host_mr, offset, mr); - ms->cxl_devices_state->next_mr_idx++; +memory_region_add_subregion (&cxl_state->host_mr, offset, mr); + cxl_state->next_mr_idx++; }static void pxb_cxl_host_class_init (ObjectClass *class,void *data)
@@ -461,17 +475,11 @@ static const TypeInfo pxb_pcie_dev_info = {static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) { -MachineState *ms =MACHINE (qdev_get_machine ()); -/* A CXL PXB's parent bus is still PCIe */ if (!pci_bus_is_express (pci_get_bus (dev))) {error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" );return ; } -if (!ms->cxl_devices_state || !ms->cxl_devices_state->is_enabled) { -error_setg (errp,"Machine does not have cxl=on" ); -return ; - }pxb_dev_realize_common (dev,CXL , errp);pxb_dev_reset (DEVICE (dev));
hw/pci-bridge/pci_expander_bridge_stubs.c new file +14 -0
@@ -0,0 +1,14 @@ +/* +* SPDX-License-Identifier: GPL-2.0-or-later +* +* Stubs for calls made from machines to handle the case where CONFIG_PXB +* is not enabled. +*/ +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "hw/pci/pci.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci-bridge/pci_expander_bridge.h" +#include "hw/cxl/cxl.h" + +void pxb_cxl_hook_up_registers (CXLState *state,PCIBus *bus,Error **errp) {};
include/hw/cxl/cxl_host.h +1 -0
@@ -16,6 +16,7 @@void cxl_machine_init (Object *obj,CXLState *state);void cxl_fmws_link_targets (CXLState *stat,Error **errp); +void cxl_hook_up_pxb_registers (PCIBus *bus,CXLState *state,Error **errp);extern const MemoryRegionOps cfmws_ops;
include/hw/pci-bridge/pci_expander_bridge.h new file +12 -0
@@ -0,0 +1,12 @@ +/* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef PCI_EXPANDER_BRIDGE_H +#define PCI_EXPANDER_BRIDGE_H + +#include "hw/cxl/cxl.h" + +void pxb_cxl_hook_up_registers (CXLState *state,PCIBus *bus,Error **errp); + +#endif /* PCI_EXPANDER_BRIDGE_H */
Signature · 1 line
2.32.0
PATCH v2 6/8 tests/acpi: Update q35/CEDT.cxl for new memory addresses.
In reply to Jonathan CameronThe CEDT table includes addreses of host bridge registers. There are allocated in a different order due to the previous patch, so update to the table is needed.
--- tests/data/acpi/q35/CEDT.cxl | Bin 184 -> 184 bytes tests/qtest/bios-tables-test-allowed-diff.h | 1 - 2 files changed, 1 deletion(-)
tests/data/acpi/q35/CEDT.cxl binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -1
@@ -1,2 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/q35/CEDT.cxl",
Signature · 1 line
2.32.0
PATCH v2 7/8 hw/cxl: Move the CXLState from MachineState to machine type specific state.
In reply to Jonathan CameronThis removes the last of the CXL code from the MachineState where it is visible to all Machines to only those that support CXL (currently i386/pc) As i386/pc always support CXL now, stop allocating the state independently.
Note the pxb register hookup code runs even if cxl=off in order to detect pxb_cxl host bridges and fail to start if any are present as they won't have the control registers available.
--- hw/core/machine.c | 6 ------ hw/i386/acpi-build.c | 6 +++--- hw/i386/pc.c | 33 ++++++++++++++++----------------- include/hw/boards.h | 1 - include/hw/i386/pc.h | 2 ++ 5 files changed, 21 insertions(+), 27 deletions(-)
hw/core/machine.c +0 -6
@@ -33,7 +33,6 @@#include "sysemu/qtest.h" #include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" -#include "hw/cxl/cxl.h" #include "migration/global_state.h" #include "migration/vmstate.h" #include "exec/confidential-guest-support.h"
@@ -1074,10 +1073,6 @@ static void machine_initfn(Object *obj) "Valid values are cpu, mem-ctrl"); } -if (mc->cxl_supported) { - ms->cxl_devices_state =g_new0 (CXLState,1 ); - } -if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) { ms->numa_state =g_new0 (NumaState,1 ); object_property_add_bool(obj, "hmat",
@@ -1115,7 +1110,6 @@ static void machine_finalize(Object *obj)g_free (ms->device_memory);g_free (ms->nvdimms_state);g_free (ms->numa_state); -g_free (ms->cxl_devices_state); }bool machine_usb (MachineState *machine)
hw/i386/acpi-build.c +3 -3
@@ -1633,7 +1633,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,/* Handle the ranges for the PXB expanders */ if (pci_bus_is_cxl (bus)) { -MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +MemoryRegion *mr = &pcms->cxl_devices_state.host_mr;uint64_t base = mr->addr; cxl_present = true;
@@ -2711,9 +2711,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine) machine->nvdimms_state, machine->ram_slots, x86ms->oem_id, x86ms->oem_table_id); } -if (machine->cxl_devices_state->is_enabled) { +if (pcms->cxl_devices_state.is_enabled) {cxl_build_cedt (table_offsets, tables_blob, tables->linker, - x86ms->oem_id, x86ms->oem_table_id, machine->cxl_devices_state); + x86ms->oem_id, x86ms->oem_table_id, &pcms->cxl_devices_state); }acpi_add_table (table_offsets, tables_blob);
hw/i386/pc.c +16 -17
@@ -733,12 +733,12 @@ void pc_machine_done(Notifier *notifier, void *data)PCMachineState *pcms =container_of (notifier, PCMachineState, machine_done);X86MachineState *x86ms =X86_MACHINE (pcms); -MachineState *ms =MACHINE (pcms); -if (ms->cxl_devices_state) { -cxl_hook_up_pxb_registers (pcms->bus, ms->cxl_devices_state, - &error_fatal); -cxl_fmws_link_targets (ms->cxl_devices_state, &error_fatal); +cxl_hook_up_pxb_registers (pcms->bus, &pcms->cxl_devices_state, + &error_fatal); + +if (pcms->cxl_devices_state.is_enabled) { +cxl_fmws_link_targets (&pcms->cxl_devices_state, &error_fatal); }/* set the number of CPUs */
@@ -908,8 +908,8 @@ void pc_memory_init(PCMachineState *pcms, &machine->device_memory->mr); } -if (machine->cxl_devices_state->is_enabled) { -MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +if (pcms->cxl_devices_state.is_enabled) { +MemoryRegion *mr = &pcms->cxl_devices_state.host_mr;hwaddr cxl_size = MiB;if (pcmc->has_reserved_memory && machine->device_memory->base) {
@@ -927,12 +927,12 @@ void pc_memory_init(PCMachineState *pcms,memory_region_init (mr,OBJECT (machine),"cxl_host_reg" , cxl_size);memory_region_add_subregion (system_memory, cxl_base, mr); cxl_resv_end = cxl_base + cxl_size; -if (machine->cxl_devices_state->fixed_windows) { +if (pcms->cxl_devices_state.fixed_windows) {hwaddr cxl_fmw_base;GList *it; cxl_fmw_base =ROUND_UP (cxl_base + cxl_size,256 * MiB); -for (it = machine->cxl_devices_state->fixed_windows; it; it = it->next) { +for (it = pcms->cxl_devices_state.fixed_windows; it; it = it->next) {CXLFixedWindow *fw = it->data; fw->base = cxl_fmw_base;
@@ -974,7 +974,7 @@ void pc_memory_init(PCMachineState *pcms, res_mem_end +=memory_region_size (&machine->device_memory->mr); } -if (machine->cxl_devices_state->is_enabled) { +if (pcms->cxl_devices_state.is_enabled) { res_mem_end = cxl_resv_end; } *val =cpu_to_le64 (ROUND_UP (res_mem_end,1 * GiB));
@@ -1010,12 +1010,12 @@ uint64_t pc_pci_hole64_start(void)X86MachineState *x86ms =X86_MACHINE (pcms);uint64_t hole64_start =0 ; -if (ms->cxl_devices_state->host_mr.addr) { - hole64_start = ms->cxl_devices_state->host_mr.addr + -memory_region_size (&ms->cxl_devices_state->host_mr); -if (ms->cxl_devices_state->fixed_windows) { +if (pcms->cxl_devices_state.host_mr.addr) { + hole64_start = pcms->cxl_devices_state.host_mr.addr + +memory_region_size (&pcms->cxl_devices_state.host_mr); +if (pcms->cxl_devices_state.fixed_windows) {GList *it; -for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { +for (it = pcms->cxl_devices_state.fixed_windows; it; it = it->next) {CXLFixedWindow *fw = it->data; hole64_start = fw->mr.addr +memory_region_size (&fw->mr); }
@@ -1691,7 +1691,6 @@ static void pc_machine_set_max_fw_size(Object *obj, Visitor *v,static void pc_machine_initfn (Object *obj) {PCMachineState *pcms =PC_MACHINE (obj); -MachineState *ms =MACHINE (obj);#ifdef CONFIG_VMPORT pcms->vmport =ON_OFF_AUTO_AUTO ;
@@ -1716,7 +1715,7 @@ static void pc_machine_initfn(Object *obj) pcms->pcspk =isa_new (TYPE_PC_SPEAKER );object_property_add_alias (OBJECT (pcms),"pcspk-audiodev" ,OBJECT (pcms->pcspk),"audiodev" ); -cxl_machine_init (obj, ms->cxl_devices_state); +cxl_machine_init (obj, &pcms->cxl_devices_state); }static void pc_machine_reset (MachineState *machine)
include/hw/boards.h +0 -1
@@ -360,7 +360,6 @@ struct MachineState {CPUArchIdList *possible_cpus;CpuTopology smp;struct NVDIMMState *nvdimms_state; -struct CXLState *cxl_devices_state;struct NumaState *numa_state;CXLFixedMemoryWindowOptionsList *cfmws_list; };
include/hw/i386/pc.h +2 -0
@@ -14,6 +14,7 @@#include "qom/object.h" #include "hw/i386/sgx-epc.h" #include "hw/firmware/smbios.h" +#include "hw/cxl/cxl.h" #define HPET_INTCAP "hpet-intcap"
@@ -55,6 +56,7 @@ typedef struct PCMachineState {hwaddr memhp_io_base;SGXEPCState sgx_epc; +CXLState cxl_devices_state; } PCMachineState;#define PC_MACHINE_ACPI_DEVICE_PROP "acpi-device"
Signature · 1 line
2.32.0
PATCH v2 8/8 hw/machine: Drop cxl_supported flag as no longer useful
In reply to Jonathan CameronAs all the CXL elements have moved to boards that support CXL, there is no need to maintain a top level flag.
--- hw/i386/pc.c | 1 - include/hw/boards.h | 1 - 2 files changed, 2 deletions(-)
hw/i386/pc.c +0 -1
@@ -1804,7 +1804,6 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = true; mc->smp_props.dies_supported = true; - mc->cxl_supported = true; mc->default_ram_id ="pc.ram" ; object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
include/hw/boards.h +0 -1
@@ -269,7 +269,6 @@ struct MachineClass {bool ignore_boot_device_suffixes;bool smbus_no_migration_support;bool nvdimm_supported; -bool cxl_supported;bool numa_mem_supported;bool auto_enable_numa;SMPCompatProps smp_props;
Signature · 1 line
2.32.0
Quoted 39 lines · Jonathan Cameron
Changes since v1 (thanks to Paolo Bonzini)
* Update 'description' of cxl-fmw as suggested to mention it's an array.
* Add a wrappercxl_hook_up_pxb_registers()tocxl-host.cas it'll be common for all machines using CXL with PXB.Run through the CI at:
https://gitlab.com/jic23/qemu/-/pipelines/553257456V1 Cover letter:
Currently only machine with CXL support upstream is i386/pc but arm/virt patches have been posted and once this is merged an updated series will follow. Switch support is queued behind this as well because they both include documentation updates.
Paolo Bonzini highlighted a couple of issues with the current CXL emulation code.
* Top level parameter rather than machine for fixed memory windows
The --cxl-fixed-memory-window top level command line parameters won't play well with efforts to make it possible to instantiate entire machines via RPC. Better to move these to be machine configuration. This change is relatively straight forward, but does result in very long command lines (cannot break fixed window setup into multiple -M entries).
* Move all CXL stuff to machine specific code and helpers
To simplify the various interactions between machine setup and host bridges etc, currently various CXL steps are called from the generic
core/machine.candsoftmmu/vl.c+ there are CXL elements in MachineState.Much of this is straight forward to do with one exception:
The CXLpci_expander_bridgehost bridges require MMIO register space. This series does this by walking the bus and filling the register space in via themachine_donecallback. This is similar to the walk done for identifying host bridges in the ACPI building code but it is rather ugly and postpones rejection ofPXB_CXLinstances where cxl=off (default).
Quoted 4 lines
All comments welcome, but the first patch at least changes the command-line so to avoid have to add backwards compatibility code, it would be great to merge that before 7.1 is released.
LGTM overall. I'm not thrilled with introducing another [sub]scronym "fmw", but otherwise, no complaints.
Series is:
Quoted 46 lines
Thanks,
Jonathan
Jonathan Cameron (8):
hw/cxl: Make the CXL fixed memory window setup a machine parameter. hw/acpi/cxl: Pass in the CXLState directly rather than MachineState hw/cxl: Push linking of CXL targets into i386/pc rather than in
machine.c
tests/acpi: Allow modification of q35 CXL CEDT table. pci/pci_expander_bridge: For CXL HB delay the HB register memory
region setup.
tests/acpi: Update q35/CEDT.cxl for new memory addresses. hw/cxl: Move the CXLState from MachineState to machine type specific
state.
hw/machine: Dropcxl_supportedflag as no longer usefuldocs/system/devices/cxl.rst | 4 +- hw/acpi/cxl.c | 9 +- hw/core/machine.c | 28 ------ hw/cxl/cxl-host-stubs.c | 9 +- hw/cxl/cxl-host.c | 100 ++++++++++++++++++-- hw/i386/acpi-build.c | 8 +- hw/i386/pc.c | 31 +++--- hw/pci-bridge/meson.build | 5 +- hw/pci-bridge/pci_expander_bridge.c | 32 ++++--- hw/pci-bridge/pci_expander_bridge_stubs.c | 14 +++ include/hw/acpi/cxl.h | 5 +- include/hw/boards.h | 3 +- include/hw/cxl/cxl.h | 9 +- include/hw/cxl/cxl_host.h | 23 +++++ include/hw/i386/pc.h | 2 + include/hw/pci-bridge/pci_expander_bridge.h | 12 +++ qapi/machine.json | 13 +++ softmmu/vl.c | 46 --------- tests/data/acpi/q35/CEDT.cxl | Bin 184 -> 184 bytes tests/qtest/bios-tables-test.c | 4 +- tests/qtest/cxl-test.c | 4 +- 21 files changed, 222 insertions(+), 139 deletions(-) create mode 100644 hw/pci-bridge/pci_expander_bridge_stubs.c create mode 100644 include/hw/cxl/cxl_host.h create mode 100644 include/hw/pci-bridge/pci_expander_bridge.hSignature · 1 line
2.32.0
Quoted 40 lines · On Mon, 6 Jun 2022 10:33:38 -0700 Ben Widawsky <[email protected]> wrote:
Quoted 39 lines · Jonathan Cameron
Changes since v1 (thanks to Paolo Bonzini)
* Update 'description' of cxl-fmw as suggested to mention it's an array.
* Add a wrappercxl_hook_up_pxb_registers()tocxl-host.cas it'll be common for all machines using CXL with PXB.Run through the CI at:
https://gitlab.com/jic23/qemu/-/pipelines/553257456V1 Cover letter:
Currently only machine with CXL support upstream is i386/pc but arm/virt patches have been posted and once this is merged an updated series will follow. Switch support is queued behind this as well because they both include documentation updates.
Paolo Bonzini highlighted a couple of issues with the current CXL emulation code.
* Top level parameter rather than machine for fixed memory windows
The --cxl-fixed-memory-window top level command line parameters won't play well with efforts to make it possible to instantiate entire machines via RPC. Better to move these to be machine configuration. This change is relatively straight forward, but does result in very long command lines (cannot break fixed window setup into multiple -M entries).
* Move all CXL stuff to machine specific code and helpers
To simplify the various interactions between machine setup and host bridges etc, currently various CXL steps are called from the generic
core/machine.candsoftmmu/vl.c+ there are CXL elements in MachineState.Much of this is straight forward to do with one exception:
The CXLpci_expander_bridgehost bridges require MMIO register space. This series does this by walking the bus and filling the register space in via themachine_donecallback. This is similar to the walk done for identifying host bridges in the ACPI building code but it is rather ugly and postpones rejection ofPXB_CXLinstances where cxl=off (default).
Quoted 7 lines
Quoted 4 lines
All comments welcome, but the first patch at least changes the command-line so to avoid have to add backwards compatibility code, it would be great to merge that before 7.1 is released.
LGTM overall. I'm not thrilled with introducing another [sub]scronym "fmw", but otherwise, no complaints.
Agreed, it's a less than ideal compromise :(
Spelling it out now seems too long as it has to be repeated a lot in a given commandline. I think cfmw is too vague for a QEMU parameter and for CXL only the ACPI "structure" (hence the s) has defined acronym of CFMWS. The underlying actual routing hardware is impdef so never given an acronym (there is only one direct reference to the window itself in the CXL spec and that's spelled out as "fixed-memory window").
Jonathan
Quoted 50 lines
Series is:
Reviewed-by Ben Widawsky <[email protected]>Quoted 46 lines
Thanks,
Jonathan
Jonathan Cameron (8):
hw/cxl: Make the CXL fixed memory window setup a machine parameter. hw/acpi/cxl: Pass in the CXLState directly rather than MachineState hw/cxl: Push linking of CXL targets into i386/pc rather than in
machine.c
tests/acpi: Allow modification of q35 CXL CEDT table. pci/pci_expander_bridge: For CXL HB delay the HB register memory
region setup.
tests/acpi: Update q35/CEDT.cxl for new memory addresses. hw/cxl: Move the CXLState from MachineState to machine type specific
state.
hw/machine: Dropcxl_supportedflag as no longer usefuldocs/system/devices/cxl.rst | 4 +- hw/acpi/cxl.c | 9 +- hw/core/machine.c | 28 ------ hw/cxl/cxl-host-stubs.c | 9 +- hw/cxl/cxl-host.c | 100 ++++++++++++++++++-- hw/i386/acpi-build.c | 8 +- hw/i386/pc.c | 31 +++--- hw/pci-bridge/meson.build | 5 +- hw/pci-bridge/pci_expander_bridge.c | 32 ++++--- hw/pci-bridge/pci_expander_bridge_stubs.c | 14 +++ include/hw/acpi/cxl.h | 5 +- include/hw/boards.h | 3 +- include/hw/cxl/cxl.h | 9 +- include/hw/cxl/cxl_host.h | 23 +++++ include/hw/i386/pc.h | 2 + include/hw/pci-bridge/pci_expander_bridge.h | 12 +++ qapi/machine.json | 13 +++ softmmu/vl.c | 46 --------- tests/data/acpi/q35/CEDT.cxl | Bin 184 -> 184 bytes tests/qtest/bios-tables-test.c | 4 +- tests/qtest/cxl-test.c | 4 +- 21 files changed, 222 insertions(+), 139 deletions(-) create mode 100644 hw/pci-bridge/pci_expander_bridge_stubs.c create mode 100644 include/hw/cxl/cxl_host.h create mode 100644 include/hw/pci-bridge/pci_expander_bridge.hSignature · 1 line
2.32.0