CXl 2.0 emulation Support
v6 cover CXl 2.0 emulation Support
This version is mainly a reorganization of v5 patch ordering to make for more sensible stepwise review and merging. I took the opportunity to add a few more tests and introduce an overview document.
Changes since v5: Thanks to Michael and Igor for comments.
- Reorganize series to make it more sensible to take in several chunks. Includes splitting qtest/cxl-test up so that we have basic testing for each stage. Suggested chunks as follows:
1-15: Infrastructure + host bridge
At this stage the pxb-cxl host bridge it is presented to the OS as a legacy / pcie host bridge.
This set is close to those Michael Tsirkin suggested picking up in v5, but the cxl=on machine parameter was pulled forwards to avoid breaking backwards compatibility. It could logically be delayed until HB MMIO is introduced but only at cost of then failing to instantiate pxb-cxl in cases that would previously have 'worked'.
16-22: Root port, plus CXL type 3 device.
Will allow Linux cxl-pci driver to bind and present some basic features of the type 3 devices.
23-40: x86 PC support including HB MMIO and CFMWS
At this region creation and use is possible on x86 platforms. Includes a new bios-table-test and more complex qtests/cxl-test cases. Includes the interleaving code and an RFC about adding ops to memory_region_ram_init_from_file().
41-42: ARM virt enabling equivalent to that done for x86. 43: Documentation
I can break this down if helpful as it includes command line examples only possible after support is in place for all elements.
I've had gitlab CI pipelines testing at patch 15, 22, and 43 and other than the build-oss-fuzz which is timing out and a false positive (I think) in checkpatch they are clean.
http://gitlab.com/jic23/qemu/~/pipelines
(trivial commit message update followed pushing those out, but
the code is the same).
- Add a documentation patch as requested by Alex in v4 review.
- Drop the code unification around PCI host bridge setup.
It will need extensive testing and was only of marginal benefit for this series so get it off the critical path.
- Don't build CEDT unless machine parameter cxl=on is set. Tests added to exercise the cxl=on case and generate meaningful tables (not empty).
- Move handling of pxb-cxl reset to a later patch where the required infrastructure has been created (avoids a crash).
- Split up qtests/cxl-test to build up step by step. Introduced an aarch64 test case covering most complex case only as this probably sufficient to catch any arch specific problems.
- Introduce test plus bios tables CEDT.cxl and DSDT.cxl.
Updated background info from v5:
Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.
Big TODOs:
* Lack of unaligned accesses to interleaved memory. Are there any side effects of this restriction?
* Volatile memory devices (easy but it's more code so left for now).
* Switch support. Linux kernel support is under review currently, so there is now something to test against.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
* CFMWS flags as requested by Ben.
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches. The kernel CXL mocking code allows some forms of testing, but QEMU provides a more versatile and exensible platform.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/
What we have here is about what you need for it to be useful for testing currently kernel code. Note the kernel code is moving fast so since v4, some features have been introduced we don't yet support in QEMU (e.g. use of the PCIe serial number extended capability).
All comments welcome.
Additional info that was here in v5 is now in the documentation patch.
Thanks,
Jonathan
Ben Widawsky (24):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory device utilities hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge) hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) hw/cxl/device: Add some trivial commands hw/cxl/device: Plumb real Label Storage Area (LSA) sizing hw/cxl/device: Implement get/set Label Storage Area (LSA) hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/cxl: Add _OSC implementation (9.14.2)
acpi/cxl: Create the CEDT (9.14.1)
acpi/cxl: Introduce CFMWS structures in CEDT hw/cxl/component Add a dumb HDM decoder handler qtest/cxl: Add more complex test cases with CFMWs
Jonathan Cameron (19):
MAINTAINERS: Add entry for Compute Express Link Emulation
cxl: Machine level control on whether CXL support is enabled
qtest/cxl: Introduce initial test for pxb-cxl only.
qtests/cxl: Add initial root port and CXL type3 tests
hw/cxl/component: Add utils for interleave parameter encoding/decoding
hw/cxl/host: Add support for CXL Fixed Memory Windows.
hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
pci/pcie_port: Add pci_find_port_by_pn()
CXL/cxl_component: Add cxl_get_hb_cstate()
mem/cxl_type3: Add read and write functions for associated hostmem.
cxl/cxl-host: Add memops for CFMWS region.
RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
i386/pc: Enable CXL fixed memory windows
tests/acpi: q35: Allow addition of a CXL test.
qtests/bios-tables-test: Add a test for CXL emulation.
tests/acpi: Add tables for CXL emulation.
hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
pxb-cxl
qtest/cxl: Add aarch64 virt test for CXL docs/cxl: Add initial Compute eXpress Link (CXL) documentation.
MAINTAINERS | 7 + docs/system/device-emulation.rst | 1 + docs/system/devices/cxl.rst | 302 +++++++++++++++++ hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl-stub.c | 12 + hw/acpi/cxl.c | 231 +++++++++++++ hw/acpi/meson.build | 4 +- hw/arm/Kconfig | 1 + hw/arm/virt-acpi-build.c | 33 ++ hw/arm/virt.c | 40 ++- hw/core/machine.c | 28 ++ hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 284 ++++++++++++++++ hw/cxl/cxl-device-utils.c | 271 ++++++++++++++++ hw/cxl/cxl-host-stubs.c | 22 ++ hw/cxl/cxl-host.c | 262 +++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 483 ++++++++++++++++++++++++++++ hw/cxl/meson.build | 12 + hw/i386/acpi-build.c | 57 +++- hw/i386/pc.c | 57 +++- hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 353 ++++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 172 +++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci-host/gpex-acpi.c | 22 +- hw/pci/pci.c | 21 +- hw/pci/pcie_port.c | 25 ++ include/hw/acpi/cxl.h | 28 ++ include/hw/arm/virt.h | 1 + include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 51 +++ include/hw/cxl/cxl_component.h | 206 ++++++++++++ include/hw/cxl/cxl_device.h | 272 ++++++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 14 + include/hw/pci/pci_bridge.h | 20 ++ include/hw/pci/pci_bus.h | 7 + include/hw/pci/pci_ids.h | 1 + include/hw/pci/pcie_port.h | 2 + qapi/machine.json | 15 + qemu-options.hx | 37 +++ scripts/device-crash-test | 1 + softmmu/memory.c | 9 + softmmu/vl.c | 11 + tests/data/acpi/q35/CEDT.cxl | Bin 0 -> 184 bytes tests/data/acpi/q35/DSDT.cxl | Bin 0 -> 9627 bytes tests/qtest/bios-tables-test.c | 39 +++ tests/qtest/cxl-test.c | 181 +++++++++++ tests/qtest/meson.build | 5 + 55 files changed, 3996 insertions(+), 25 deletions(-) create mode 100644 docs/system/devices/cxl.rst create mode 100644 hw/acpi/cxl-stub.c create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-host-stubs.c create mode 100644 hw/cxl/cxl-host.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/q35/CEDT.cxl create mode 100644 tests/data/acpi/q35/DSDT.cxl create mode 100644 tests/qtest/cxl-test.c
Signature · 1 line
2.32.0
PATCH v6 1/43 hw/pci/cxl: Add a CXL component type (interface)
From: Ben Widawsky <[email protected]>
A CXL component is a hardware entity that implements CXL component registers from the CXL 2.0 spec (8.2.3). Currently these represent 3 general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)
A CXL component can be conceptually thought of as a PCIe device with extra functionality when enumerated and enabled. For this reason, CXL does here, and will continue to add on to existing PCI code paths.
Host bridges will typically need to be handled specially and so they can implement this newly introduced interface or not. All other components should implement this interface. Implementing this interface allows the core PCI code to treat these devices as special where appropriate.
--- hw/pci/pci.c | 10 ++++++++++ include/hw/pci/pci.h | 8 ++++++++ 2 files changed, 18 insertions(+)
hw/pci/pci.c +10 -0
@@ -201,6 +201,11 @@ static const TypeInfo pci_bus_info = { .class_init = pci_bus_class_init, }; +static const TypeInfo cxl_interface_info = { + .name =INTERFACE_CXL_DEVICE , + .parent =TYPE_INTERFACE , +}; +static const TypeInfo pcie_interface_info = { .name =INTERFACE_PCIE_DEVICE , .parent =TYPE_INTERFACE ,
@@ -2128,6 +2133,10 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp) pci_dev->cap_present |=QEMU_PCI_CAP_EXPRESS ; } +if (object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE )) { + pci_dev->cap_present |=QEMU_PCIE_CAP_CXL ; + } + pci_dev =do_pci_register_device (pci_dev,object_get_typename (OBJECT (qdev)), pci_dev->devfn, errp);
@@ -2884,6 +2893,7 @@ static void pci_register_types(void)type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info);type_register_static (&conventional_pci_interface_info); +type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);type_register_static (&pci_device_type_info); }
include/hw/pci/pci.h +8 -0
@@ -196,6 +196,8 @@ enum { QEMU_PCIE_LNKSTA_DLLLA = (1 << QEMU_PCIE_LNKSTA_DLLLA_BITNR),#define QEMU_PCIE_EXTCAP_INIT_BITNR 9 QEMU_PCIE_EXTCAP_INIT = (1 << QEMU_PCIE_EXTCAP_INIT_BITNR), +#define QEMU_PCIE_CXL_BITNR 10 + QEMU_PCIE_CAP_CXL = (1 << QEMU_PCIE_CXL_BITNR), };#define TYPE_PCI_DEVICE "pci-device"
@@ -203,6 +205,12 @@ typedef struct PCIDeviceClass PCIDeviceClass;DECLARE_OBJ_CHECKERS (PCIDevice, PCIDeviceClass,PCI_DEVICE ,TYPE_PCI_DEVICE ) +/* +* Implemented by devices that can be plugged on CXL buses. In the spec, this is +* actually a "CXL Component, but we name it device to match the PCI naming. + */ +#define INTERFACE_CXL_DEVICE "cxl-device" +/* Implemented by devices that can be plugged on PCI Express buses */ #define INTERFACE_PCIE_DEVICE "pci-express-device"
Signature · 1 line
2.32.0
PATCH v6 2/43 hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The cxl_pci header manages bits associated with PCI, specifically the DVSEC and related fields. The cxl_component.h variant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] registers. Per spec, the registers are little endian.
None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
--- hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 219 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 4 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 16 +++ include/hw/cxl/cxl_component.h | 196 +++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++ 8 files changed, 578 insertions(+)
hw/Kconfig +1 -0
@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfig
hw/cxl/Kconfig new file +3 -0
@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESS
hw/cxl/cxl-component-utils.c new file +219 -0
219 lines not on this page · raw message
hw/cxl/meson.build new file +4 -0
@@ -0,0 +1,4 @@ +softmmu_ss.add(when: 'CONFIG_CXL', + if_true: files( + 'cxl-component-utils.c', + ))
hw/meson.build +1 -0
@@ -6,6 +6,7 @@ subdir('block') subdir('char') subdir('core') subdir('cpu') +subdir('cxl') subdir('display') subdir('dma') subdir('gpio')
include/hw/cxl/cxl.h new file +16 -0
@@ -0,0 +1,16 @@ +/* +* QEMU CXL Support +* +* Copyright (c) 2020 Intel +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#ifndef CXL_H +#define CXL_H + +#include "cxl_pci.h" +#include "cxl_component.h" + +#endif
include/hw/cxl/cxl_component.h new file +196 -0
196 lines not on this page · raw message
include/hw/cxl/cxl_pci.h new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v6 3/43 MAINTAINERS: Add entry for Compute Express Link Emulation
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The CXL emulation will be jointly maintained by Ben Widawsky and Jonathan Cameron. Broken out as a separate patch to improve visibility.
--- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
@@ -2524,6 +2524,13 @@ F: qapi/block*.json F: qapi/transaction.json T: git https://repo.or.cz/qemu/armbru.git block-next +Compute Express Link +M: Ben Widawsky <[email protected]> +M: Jonathan Cameron <[email protected]> +S: Supported +F: hw/cxl/ +F: include/hw/cxl/ + Dirty Bitmaps M: Eric Blake <[email protected]> M: Vladimir Sementsov-Ogievskiy <[email protected]>
Signature · 1 line
2.32.0
PATCH v6 4/43 hw/cxl/device: Introduce a CXL device (8.2.8)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.
This new device type is specifically catered towards the eventual implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.
--- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_device.h | 165 ++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+)
include/hw/cxl/cxl.h +1 -0
@@ -12,5 +12,6 @@#include "cxl_pci.h" #include "cxl_component.h" +#include "cxl_device.h" #endif
include/hw/cxl/cxl_device.h new file +165 -0
165 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v6 5/43 hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This implements all device MMIO up to the first capability. That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
--- hw/cxl/cxl-device-utils.c | 109 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_device.h | 31 +++++++++- 3 files changed, 140 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c new file +109 -0
@@ -0,0 +1,109 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +/* +* Device registers have no restrictions per the spec, and so fall back to the +* default memory mapped register rules in 8.2: +* Software shall use CXL.io Memory Read and Write to access memory mapped +* register defined in this section. Unless otherwise specified, software +* shall restrict the accesses width based on the following: +* • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes +* quantity. +* • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 +* Bytes +* • The address shall be a multiple of the access width, e.g. when +* accessing a register as a 4 Byte quantity, the address shall be +* multiple of 4. +* • The accesses shall map to contiguous bytes.If these rules are not +* followed, the behavior is undefined +*/ + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (size ==4 ) { +return cxl_dstate->caps_reg_state32[offset /4 ]; + }else { +return cxl_dstate->caps_reg_state64[offset /8 ]; + } +} + +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL ,/* status register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL ,/* caps registers are read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ +memory_region_init (&cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MMIO_SIZE )); + +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_CAPS_SIZE ); +memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate, +"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->device); +} + +static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; +const int cap_count =1 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_COUNT , cap_count); + +cxl_device_cap_init (cxl_dstate,DEVICE ,1 ); +device_reg_init_common (cxl_dstate); +}
hw/cxl/meson.build +1 -0
@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
include/hw/cxl/cxl_device.h +30 -1
@@ -63,6 +63,8 @@#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10/* Figure 138 */ #define CXL_DEVICE_CAP_REG_SIZE 0x10/* 8.2.8.2 */ #define CXL_DEVICE_CAPS_MAX 4/* 8.2.8.2.1 + 8.2.8.5 */ +#define CXL_CAPS_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE * (CXL_DEVICE_CAPS_MAX + 1))/* +1 for header */ #define CXL_DEVICE_REGISTERS_OFFSET 0x80/* Read comment above */ #define CXL_DEVICE_REGISTERS_LENGTH 0x8/* 8.2.8.3.1 */
@@ -75,12 +77,22 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) +#define CXL_MMIO_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ + CXL_MAILBOX_REGISTERS_LENGTH) + typedef struct cxl_device_state { MemoryRegion device_registers; /* mmio for device capabilities array - 8.2.8.2 */ MemoryRegion device; - MemoryRegion caps; + struct { + MemoryRegion caps; + union { + uint32_t caps_reg_state32[CXL_CAPS_SIZE / 4]; + uint64_t caps_reg_state64[CXL_CAPS_SIZE / 8]; + }; + }; /* mmio for the mailbox registers 8.2.8.4 */ MemoryRegion mailbox;
@@ -133,6 +145,23 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +#define cxl_device_cap_init (dstate, reg, cap_id) \ + do { \ + uint32_t *cap_hdrs = dstate->caps_reg_state32; \ + int which = R_CXL_DEV_##reg##_CAP_HDR0; \ + cap_hdrs[which] = \ + FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, \ + CAP_ID, cap_id); \ + cap_hdrs[which] = FIELD_DP32( \ + cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1); \ + cap_hdrs[which + 1] = \ + FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1, \ + CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET); \ + cap_hdrs[which + 2] = \ + FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2, \ + CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH); \ + } while (0) + REG32(CXL_DEV_MAILBOX_CAP, 0) FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)FIELD (CXL_DEV_MAILBOX_CAP ,INT_CAP ,5 ,1 )
Signature · 1 line
2.32.0
PATCH v6 6/43 hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in cxl-mailbox-utils.c, and the hardware.
No commands are implemented yet.
--- hw/cxl/cxl-device-utils.c | 128 ++++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 171 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 19 +++- 5 files changed, 320 insertions(+), 2 deletions(-)
hw/cxl/cxl-device-utils.c +127 -1
@@ -44,6 +44,114 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return 0 ; } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +switch (size) { +case 1 : +return cxl_dstate->mbox_reg_state[offset]; +case 2 : +return cxl_dstate->mbox_reg_state16[offset /2 ]; +case 4 : +return cxl_dstate->mbox_reg_state32[offset /4 ]; +case 8 : +return cxl_dstate->mbox_reg_state64[offset /8 ]; +default : +g_assert_not_reached (); + } +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + reg_state[offset /4 ] = value; +} + +static void mailbox_mem_writeq (uint64_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + + reg_state[offset /8 ] = value; +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) { +memcpy (cxl_dstate->mbox_reg_state + offset, &value, size); +return ; + } + +/* +* Lock is needed to prevent concurrent writes as well as to +* prevent writes coming in while the firmware is processing. +* Until background commands or the second mailbox are implemented +* memory access is synchronized at a higher level (per memory region). +*/ + +switch (size) { +case 4 : +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +mailbox_mem_writeq (cxl_dstate->mbox_reg_state64, offset, value); +break ; +default : +g_assert_not_reached (); + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,/* status register is read only */
@@ -84,20 +192,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_CAPS_SIZE);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -106,4 +227,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
hw/cxl/cxl-mailbox-utils.c new file +171 -0
171 lines not on this page · raw message
hw/cxl/meson.build +1 -0
@@ -2,4 +2,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))
include/hw/cxl/cxl.h +3 -0
@@ -14,4 +14,7 @@#include "cxl_component.h" #include "cxl_device.h" +#define CXL_COMPONENT_REG_BAR_IDX 0 +#define CXL_DEVICE_REG_BAR_IDX 2 +#endif
include/hw/cxl/cxl_device.h +18 -1
@@ -95,7 +95,21 @@ typedef struct cxl_device_state { };/* mmio for the mailbox registers 8.2.8.4 */ -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH /2 ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH /4 ]; +uint64_t mbox_reg_state64[CXL_MAILBOX_REGISTERS_LENGTH /8 ]; + }; +struct cel_log { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };/* memory region for persistent memory, HDM */ uint64_t pmem_size;
@@ -145,6 +159,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
Signature · 1 line
2.32.0
Re: [PATCH v6 06/43] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 98 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.No commands are implemented yet.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 128 ++++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 171 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 19 +++- 5 files changed, 320 insertions(+), 2 deletions(-)hw/cxl/cxl-device-utils.c +66
-0@@ -44,6 +44,114 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return 0 ; } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +switch (size) { +case 1 : +return cxl_dstate->mbox_reg_state[offset]; +case 2 : +return cxl_dstate->mbox_reg_state16[offset /2 ]; +case 4 : +return cxl_dstate->mbox_reg_state32[offset /4 ]; +case 8 : +return cxl_dstate->mbox_reg_state64[offset /8 ]; +default : +g_assert_not_reached (); + } +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + reg_state[offset /4 ] = value; +} + +static void mailbox_mem_writeq (uint64_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + + reg_state[offset /8 ] = value; +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) {
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 122
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+memcpy (cxl_dstate->mbox_reg_state + offset, &value, size); +return ; + } + +/* +* Lock is needed to prevent concurrent writes as well as to +* prevent writes coming in while the firmware is processing. +* Until background commands or the second mailbox are implemented +* memory access is synchronized at a higher level (per memory region). +*/
What lock?
That said you probably don't need one as all access to IO space should already be serialised by the BQL so even multiple vCPUs will serialise their access.
Quoted 5 lines
hw/cxl/cxl-device-utils.c +5
-0+ +switch (size) { +case 4 : +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ;
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 137
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+case 8 : +mailbox_mem_writeq (cxl_dstate->mbox_reg_state64, offset, value); +break ; +default : +g_assert_not_reached (); + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate);
You want some braces in there to meet coding standards.
Quoted 183 lines
hw/cxl/cxl-device-utils.c +36
-1+} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,/* status register is read only */ @@ -84,20 +192,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_CAPS_SIZE);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -106,4 +227,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +112
-0@@ -0,0 +1,171 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Implement the handler +* static ret_code cmd_foo_bar(struct cxl_cmd *cmd, +* CXLDeviceState *cxl_dstate, uint16_t *len) +* 3. Add the command to the cxl_cmd_set[][] +* [FOO][BAR] = { "FOO_BAR", cmd_foo_bar, x, y }, +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define DEFINE_MAILBOX_HANDLER_ZEROED (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define DEFINE_MAILBOX_HANDLER_NOP (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \ + } + +static QemuUUID cel_uuid; + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/
hw/cxl/cxl-mailbox-utils.c line 122
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c new file +10
-0+uint64_t command_reg = cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD ]; + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { +qemu_log_mask (LOG_UNIMP ,"Command %04xh not implemented\n" , + set <<8 | cmd);
is ret of CXL_MBOX_SUCCESS still ok for an unimplemented command?
hw/cxl/cxl-mailbox-utils.c line 133
Quoted 11 lines
hw/cxl/cxl-mailbox-utils.c new file +11
-0+ goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); +
Not super keen on the goto, it seems to me the you could trivially re-arrange this to avoid it as it is not a super deep implementation.
if (h) {
if (len == cxl_cmd->in) {
/* do the thing */
} else {
ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
} else {
qemu_log_mask(LOG_UNIMP, "Command %04xh not implemented\n",
set << 8 | cmd);
}/* process the result */
Quoted 88 lines
hw/cxl/cxl-mailbox-utils.c new file +38
-0+handled: +/* Set the return code */ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* Set the return length */ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + + cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD ] = command_reg; + cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_STS ] = status_reg; + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +/* CXL 2.0: Table 169 Get Supported Logs Log Entry */ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ; + +for (int set =0 ; set <256 ; set++) { +for (int cmd =0 ; cmd <256 ; cmd++) { +if (cxl_cmd_set[set][cmd].handler) { +struct cxl_cmd *c = &cxl_cmd_set[set][cmd]; +struct cel_log *log = + &cxl_dstate->cel_log[cxl_dstate->cel_size]; + + log->opcode = (set <<8 ) | cmd; + log->effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -2,4 +2,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,4 +14,7 @@#include "cxl_component.h" #include "cxl_device.h" +#define CXL_COMPONENT_REG_BAR_IDX 0 +#define CXL_DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +15
-1@@ -95,7 +95,21 @@ typedef struct cxl_device_state { };/* mmio for the mailbox registers 8.2.8.4 */ -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH /2 ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH /4 ]; +uint64_t mbox_reg_state64[CXL_MAILBOX_REGISTERS_LENGTH /8 ]; + }; +struct cel_log { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };/* memory region for persistent memory, HDM */ uint64_t pmem_size;
include/hw/cxl/cxl_device.h CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET) · line 167
Quoted 9 lines
include/hw/cxl/cxl_device.h +3
-0@@ -145,6 +159,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v6 06/43] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 113 lines · On Tue, 01 Mar 2022 15:32:02 +0000 Alex Bennée <[email protected]> wrote:
Quoted 98 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.No commands are implemented yet.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 128 ++++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 171 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 19 +++- 5 files changed, 320 insertions(+), 2 deletions(-)hw/cxl/cxl-device-utils.c +66
-0@@ -44,6 +44,114 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return 0 ; } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +switch (size) { +case 1 : +return cxl_dstate->mbox_reg_state[offset]; +case 2 : +return cxl_dstate->mbox_reg_state16[offset /2 ]; +case 4 : +return cxl_dstate->mbox_reg_state32[offset /4 ]; +case 8 : +return cxl_dstate->mbox_reg_state64[offset /8 ]; +default : +g_assert_not_reached (); + } +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + reg_state[offset /4 ] = value; +} + +static void mailbox_mem_writeq (uint64_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + + reg_state[offset /8 ] = value; +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) {hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 122
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+memcpy (cxl_dstate->mbox_reg_state + offset, &value, size); +return ; + } + +/* +* Lock is needed to prevent concurrent writes as well as to +* prevent writes coming in while the firmware is processing. +* Until background commands or the second mailbox are implemented +* memory access is synchronized at a higher level (per memory region). +*/ What lock?
Quoted 3 lines
That said you probably don't need one as all access to IO space should already be serialised by the BQL so even multiple vCPUs will serialise their access.
oops. I removed the lock for exactly this reason but failed to remove the comment. Now gone ;)
Quoted 187 lines
Quoted 187 lines
hw/cxl/cxl-device-utils.c +36
-1+} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,/* status register is read only */ @@ -84,20 +192,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_CAPS_SIZE);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -106,4 +227,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +116
-0@@ -0,0 +1,171 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Implement the handler +* static ret_code cmd_foo_bar(struct cxl_cmd *cmd, +* CXLDeviceState *cxl_dstate, uint16_t *len) +* 3. Add the command to the cxl_cmd_set[][] +* [FOO][BAR] = { "FOO_BAR", cmd_foo_bar, x, y }, +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define DEFINE_MAILBOX_HANDLER_ZEROED (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define DEFINE_MAILBOX_HANDLER_NOP (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \ + } + +static QemuUUID cel_uuid; + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD ]; + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND );
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c line 122
Quoted 6 lines
hw/cxl/cxl-mailbox-utils.c new file +6
-0+uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { +qemu_log_mask (LOG_UNIMP ,"Command %04xh not implemented\n" , + set <<8 | cmd);is ret of
CXL_MBOX_SUCCESSstill ok for an unimplemented command?
Good spot. CXL_MBOX_UNSUPPORTED is a more useful response.
Quoted 17 lines
hw/cxl/cxl-mailbox-utils.c line 133
Quoted 12 lines
hw/cxl/cxl-mailbox-utils.c new file +11
-0+ goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); +Not super keen on the goto, it seems to me the you could trivially re-arrange this to avoid it as it is not a super deep implementation.
if (h) {
Quoted 8 lines
if (len == cxl_cmd->in) {
/* do the thing */
} else {
ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
} else {
qemu_log_mask(LOG_UNIMP, "Command %04xh not implemented\n",
set << 8 | cmd);
}Indeed much cleaner.
..
Quoted 5 lines
Otherwise:
Reviewed-by Alex Bennée <[email protected]>
Everything else changed as suggested.
Thanks,
Jonathan
PATCH v6 7/43 hw/cxl/device: Add memory device utilities
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Memory devices implement extra capabilities on top of CXL devices. This adds support for that.
A large part of memory devices is the mailbox/command interface. All of the mailbox handling is done in the mailbox-utils library. Longer term, new CXL devices that are being emulated may want to handle commands differently, and therefore would need a mechanism to opt in/out of the specific generic handlers. As such, this is considered sufficient for now, but may need more depth in the future.
--- hw/cxl/cxl-device-utils.c | 38 ++++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 22 ++++++++++++++++++--- 2 files changed, 56 insertions(+), 4 deletions(-)
hw/cxl/cxl-device-utils.c +37 -1
@@ -137,6 +137,31 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,cxl_process_mailbox (cxl_dstate); } +static uint64_t mdev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +uint64_t retval =0 ; + + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MEDIA_STATUS ,1 ); + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MBOX_READY ,1 ); + +return retval; +} + +static const MemoryRegionOps mdev_ops = { + .read = mdev_reg_read, + .write =NULL ,/* memory device register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =8 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps mailbox_ops = { .read = mailbox_reg_read, .write = mailbox_reg_write,
@@ -194,6 +219,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "device-status", CXL_DEVICE_REGISTERS_LENGTH);memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->memory_device, obj, &mdev_ops, + cxl_dstate,"memory device caps" , +CXL_MEMORY_DEVICE_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);
@@ -203,6 +231,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)memory_region_add_subregion (&cxl_dstate->device_registers,CXL_MAILBOX_REGISTERS_OFFSET , &cxl_dstate->mailbox); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MEMORY_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->memory_device); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { }
@@ -215,10 +246,12 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; } +static void memdev_reg_init_common (CXLDeviceState *cxl_dstate) { } +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64; -const int cap_count =2 ; +const int cap_count =3 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP64 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -231,5 +264,8 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,MAILBOX ,2 );mailbox_reg_init_common (cxl_dstate); +cxl_device_cap_init (cxl_dstate,MEMORY_DEVICE ,0x4000 ); +memdev_reg_init_common (cxl_dstate); +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
include/hw/cxl/cxl_device.h +19 -3
@@ -77,15 +77,21 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) -#define CXL_MMIO_SIZE \ - (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ - CXL_MAILBOX_REGISTERS_LENGTH) + +#define CXL_MEMORY_DEVICE_REGISTERS_OFFSET \ + (CXL_MAILBOX_REGISTERS_OFFSET + CXL_MAILBOX_REGISTERS_LENGTH) +#define CXL_MEMORY_DEVICE_REGISTERS_LENGTH 0x8 + +#define CXL_MMIO_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ + CXL_MAILBOX_REGISTERS_LENGTH + CXL_MEMORY_DEVICE_REGISTERS_LENGTH) typedef struct cxl_device_state { MemoryRegion device_registers; /* mmio for device capabilities array - 8.2.8.2 */ MemoryRegion device; + MemoryRegion memory_device; struct { MemoryRegion caps; union {
@@ -158,6 +164,9 @@ REG64(CXL_DEV_CAP_ARRAY, 0) /* Documented as 128 bit register but 64 byte access CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MEMORY_DEVICE , +CXL_DEVICE_CAP_HDR1_OFFSET + + CXL_DEVICE_CAP_REG_SIZE * 2)int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate);void cxl_process_mailbox (CXLDeviceState *cxl_dstate);
@@ -208,4 +217,11 @@ REG64(CXL_DEV_BG_CMD_STS, 0x18) REG32(CXL_DEV_CMD_PAYLOAD, 0x20) +REG64(CXL_MEM_DEV_STS, 0) + FIELD(CXL_MEM_DEV_STS, FATAL, 0, 1) + FIELD(CXL_MEM_DEV_STS, FW_HALT, 1, 1) + FIELD(CXL_MEM_DEV_STS, MEDIA_STATUS, 2, 2) + FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) + FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +#endif
Signature · 1 line
2.32.0
PATCH v6 8/43 hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Using the previously implemented stubbed helpers, it is now possible to easily add the missing, required commands to the implementation.
--- hw/cxl/cxl-mailbox-utils.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +26 -1
@@ -38,6 +38,14 @@* a register interface that already deals with it. */ +enum { + EVENTS = 0x01, + #define GET_RECORDS 0x0 +#define CLEAR_RECORDS 0x1 +#define GET_INTERRUPT_POLICY 0x2 +#define SET_INTERRUPT_POLICY 0x3 +}; +/* 8.2.8.4.5.1 Command Return Codes */ typedef enum { CXL_MBOX_SUCCESS = 0x0,
@@ -93,9 +101,26 @@ struct cxl_cmd {return CXL_MBOX_SUCCESS ; \ } +DEFINE_MAILBOX_HANDLER_ZEROED (events_get_records,0x20 ); +DEFINE_MAILBOX_HANDLER_NOP (events_clear_records); +DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 ); +DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +static QemuUUID cel_uuid; -static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_LOG_CHANGE (1 << 4) + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = { + [EVENTS ][GET_RECORDS ] = {"EVENTS_GET_RECORDS" , + cmd_events_get_records,1 ,0 }, + [EVENTS ][CLEAR_RECORDS ] = {"EVENTS_CLEAR_RECORDS" , + cmd_events_clear_records, ~0 ,IMMEDIATE_LOG_CHANGE }, + [EVENTS ][GET_INTERRUPT_POLICY ] = {"EVENTS_GET_INTERRUPT_POLICY" , + cmd_events_get_interrupt_policy,0 ,0 }, + [EVENTS ][SET_INTERRUPT_POLICY ] = {"EVENTS_SET_INTERRUPT_POLICY" , + cmd_events_set_interrupt_policy,4 ,IMMEDIATE_CONFIG_CHANGE }, +};void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {
Signature · 1 line
2.32.0
PATCH v6 9/43 hw/cxl/device: Timestamp implementation (8.2.9.3)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Errata F4 to CXL 2.0 clarified the meaning of the timer as the sum of the value set with the timestamp set command and the number of nano seconds since it was last set.
--- hw/cxl/cxl-mailbox-utils.c | 44 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 50 insertions(+)
hw/cxl/cxl-mailbox-utils.c +44 -0
@@ -44,6 +44,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -106,9 +109,48 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 );DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +/* 8.2.9.3.1 */ +static ret_code cmd_timestamp_get (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +uint64_t time, delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done; + } + +/* First find the delta from the last time the host set the time. */ + time =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); + delta = time - cxl_dstate->timestamp.last_set; + +/* Then adjust the actual time */ +stq_le_p (cmd->payload, cxl_dstate->timestamp.host_set + delta); + +done: + *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.3.2 */ +static ret_code cmd_timestamp_set (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); + + cxl_dstate->timestamp.host_set =le64_to_cpu (*(uint64_t *)cmd->payload); + + *len =0 ; +return CXL_MBOX_SUCCESS ; +} +static QemuUUID cel_uuid;#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {
@@ -120,6 +162,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, + [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, + [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)
include/hw/cxl/cxl_device.h +6 -0
@@ -117,6 +117,12 @@ typedef struct cxl_device_state {size_t cel_size; }; +struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +/* memory region for persistent memory, HDM */ uint64_t pmem_size; } CXLDeviceState;
Signature · 1 line
2.32.0
Re: [PATCH v6 09/43] hw/cxl/device: Timestamp implementation (8.2.9.3)
Quoted 41 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Errata F4 to CXL 2.0 clarified the meaning of the timer as the sum of the value set with the timestamp set command and the number of nano seconds since it was last set.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 44 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 50 insertions(+)hw/cxl/cxl-mailbox-utils.c +13
-0@@ -44,6 +44,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -106,9 +109,48 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 );DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +/* 8.2.9.3.1 */ +static ret_code cmd_timestamp_get (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +uint64_t time, delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done;
hw/cxl/cxl-mailbox-utils.c DEFINE_MAILBOX_HANDLER_NOP(events_clear_records); · line 131
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+ } + +/* First find the delta from the last time the host set the time. */ + time =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); + delta = time - cxl_dstate->timestamp.last_set; + +/* Then adjust the actual time */ +stq_le_p (cmd->payload, cxl_dstate->timestamp.host_set + delta); + +done:
Again I think an unnecessary goto for want of:
uin64_t final_time = 0;
if (cxl_dstate->timestamp.set) {
/* First find the delta from the last time the host set the time. */
time = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
delta = time - cxl_dstate->timestamp.last_set;
final_time = cxl_dstate->timestamp.host_set + delta;}
/* Then adjust the actual time */
stq_le_p(cmd->payload, final_time);
Quoted 41 lines
hw/cxl/cxl-mailbox-utils.c +21
-0+ *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.3.2 */ +static ret_code cmd_timestamp_set (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set =qemu_clock_get_ns (QEMU_CLOCK_VIRTUAL ); + + cxl_dstate->timestamp.host_set =le64_to_cpu (*(uint64_t *)cmd->payload); + + *len =0 ; +return CXL_MBOX_SUCCESS ; +} +static QemuUUID cel_uuid;#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {@@ -120,6 +162,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, + [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, + [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)include/hw/cxl/cxl_device.h +0
-0@@ -117,6 +117,12 @@ typedef struct cxl_device_state {size_t cel_size; };
include/hw/cxl/cxl_device.h typedef struct cxl_device_state { · line 128
Quoted 10 lines
include/hw/cxl/cxl_device.h +6
-0+struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +/* memory region for persistent memory, HDM */ uint64_t pmem_size; } CXLDeviceState;
Otherwise:
Signature · 1 line
Alex Bennée
PATCH v6 10/43 hw/cxl/device: Add log commands (8.2.9.4) + CEL
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
--- hw/cxl/cxl-mailbox-utils.c | 69 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)
hw/cxl/cxl-mailbox-utils.c +69 -0
@@ -47,6 +47,9 @@ enum { TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1 + LOGS = 0x04, +#define GET_SUPPORTED 0x0 +#define GET_LOG 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -149,6 +152,70 @@ static ret_code cmd_timestamp_set(struct cxl_cmd *cmd,static QemuUUID cel_uuid; +/* 8.2.9.4.1 */ +static ret_code cmd_logs_get_supported (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint16_t entries; +uint8_t rsvd[6 ]; +struct { +QemuUUID uuid; +uint32_t size; + } log_entries[1 ]; + } __attribute__((packed)) *supported_logs = (void *)cmd->payload; +_Static_assert (sizeof (*supported_logs) ==0x1c ,"Bad supported log size" ); + + supported_logs->entries =1 ; + supported_logs->log_entries[0 ].uuid = cel_uuid; + supported_logs->log_entries[0 ].size =4 * cxl_dstate->cel_size; + + *len =sizeof (*supported_logs); +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.4.2 */ +static ret_code cmd_logs_get_log (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +QemuUUID uuid; +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (16 ))) *get_log = (void *)cmd->payload; + +/* +* 8.2.9.4.2 +* The device shall return Invalid Parameter if the Offset or Length +* fields attempt to access beyond the size of the log as reported by Get +* Supported Logs. +* +* XXX: Spec is wrong, "Invalid Parameter" isn't a thing. +* XXX: Spec doesn't address incorrect UUID incorrectness. +* +* The CEL buffer is large enough to fit all commands in the emulation, so +* the only possible failure would be if the mailbox itself isn't big +* enough. +*/ +if (get_log->offset + get_log->length > cxl_dstate->payload_size) { +return CXL_MBOX_INVALID_INPUT ; + } + +if (!qemu_uuid_is_equal (&get_log->uuid, &cel_uuid)) { +return CXL_MBOX_UNSUPPORTED ; + } + +/* Store off everything to local variables so we can wipe out the payload */ + *len = get_log->length; + +memmove (cmd->payload, cxl_dstate->cel_log + get_log->offset, + get_log->length); + +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -164,6 +231,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, + [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, + [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)
Signature · 1 line
2.32.0
Re: [PATCH v6 10/43] hw/cxl/device: Add log commands (8.2.9.4) + CEL
Quoted 13 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
Quoted 2 lines
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 11/43 hw/pxb: Use a type for realizing expanders
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This opens up the possibility for more types of expanders (other than PCI and PCIe). We'll need this to create a CXL expander.
--- hw/pci-bridge/pci_expander_bridge.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +7 -4
@@ -24,6 +24,8 @@#include "hw/boards.h" #include "qom/object.h" +enum BusType {PCI ,PCIE }; +#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ; DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
@@ -221,7 +223,8 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)0 ; } -static void pxb_dev_realize_common (PCIDevice *dev,bool pcie,Error **errp) +static void pxb_dev_realize_common (PCIDevice *dev,enum BusType type, +Error **errp) {PXBDev *pxb =convert_to_pxb (dev);DeviceState *ds, *bds =NULL ;
@@ -246,7 +249,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp) } ds =qdev_new (TYPE_PXB_HOST ); -if (pcie) { +if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS );
@@ -295,7 +298,7 @@ static void pxb_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, false, errp); +pxb_dev_realize_common (dev,PCI , errp); }static void pxb_dev_exitfn (PCIDevice *pci_dev)
@@ -348,7 +351,7 @@ static void pxb_pcie_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, true, errp); +pxb_dev_realize_common (dev,PCIE , errp); }static void pxb_pcie_dev_class_init (ObjectClass *klass,void *data)
Signature · 1 line
2.32.0
PATCH v6 12/43 hw/pci/cxl: Create a CXL bus type
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The easiest way to differentiate a CXL bus, and a PCIE bus is using a flag. A CXL bus, in hardware, is backward compatible with PCIE, and therefore the code tries pretty hard to keep them in sync as much as possible.
The other way to implement this would be to try to cast the bus to the correct type. This is less code and useful for debugging via simply looking at the flags.
--- hw/pci-bridge/pci_expander_bridge.c | 9 ++++++++- include/hw/pci/pci_bus.h | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +8 -1
@@ -24,7 +24,7 @@#include "hw/boards.h" #include "qom/object.h" -enum BusType {PCI ,PCIE }; +enum BusType {PCI ,PCIE ,CXL };#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ;
@@ -35,6 +35,10 @@ DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,DECLARE_INSTANCE_CHECKER (PXBBus,PXB_PCIE_BUS , TYPE_PXB_PCIE_BUS) +#define TYPE_PXB_CXL_BUS "pxb-cxl-bus" +DECLARE_INSTANCE_CHECKER (PXBBus,PXB_CXL_BUS , + TYPE_PXB_CXL_BUS) +struct PXBBus {/*< private >*/ PCIBus parent_obj;
@@ -251,6 +255,9 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, ds =qdev_new (TYPE_PXB_HOST );if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); + }else if (type ==CXL ) { + bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); + bus->flags |=PCI_BUS_CXL ; }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
include/hw/pci/pci_bus.h +7 -0
@@ -24,6 +24,8 @@ enum PCIBusFlags {PCI_BUS_IS_ROOT =0x0001 ,/* PCIe extended configuration space is accessible on this bus */ PCI_BUS_EXTENDED_CONFIG_SPACE =0x0002 , +/* This is a CXL Type BUS */ + PCI_BUS_CXL = 0x0004, }; struct PCIBus {
@@ -53,6 +55,11 @@ struct PCIBus {Notifier machine_done; }; +static inline bool pci_bus_is_cxl (PCIBus *bus) +{ +return !!(bus->flags &PCI_BUS_CXL ); +} +static inline bool pci_bus_is_root (PCIBus *bus) {return !!(bus->flags &PCI_BUS_IS_ROOT );
Signature · 1 line
2.32.0
PATCH v6 13/43 cxl: Machine level control on whether CXL support is enabled
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
There are going to be some potential overheads to CXL enablement, for example the host bridge region reserved in memory maps. Add a machine level control so that CXL is disabled by default.
--- hw/core/machine.c | 28 ++++++++++++++++++++++++++++ hw/i386/pc.c | 1 + include/hw/boards.h | 2 ++ include/hw/cxl/cxl.h | 4 ++++ 4 files changed, 35 insertions(+)
hw/core/machine.c +28 -0
@@ -31,6 +31,7 @@#include "sysemu/qtest.h" #include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "migration/global_state.h" #include "migration/vmstate.h" #include "exec/confidential-guest-support.h"
@@ -545,6 +546,20 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value, nvdimms_state->persistence_string =g_strdup (value); } +static bool machine_get_cxl (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->cxl_devices_state->is_enabled; +} + +static void machine_set_cxl (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->cxl_devices_state->is_enabled = value; +} +void machine_class_allow_dynamic_sysbus_dev (MachineClass *mc,const char *type) {QAPI_LIST_PREPEND (mc->allowed_dynamic_sysbus_devices,g_strdup (type));
@@ -777,6 +792,8 @@ static void machine_class_init(ObjectClass *oc, void *data) mc->default_ram_size =128 * MiB; mc->rom_file_has_mr = true; +/* Few machines support CXL, so default to off */ + mc->cxl_supported = false;/* numa node memory size aligned on 8MB by default. * On Linux, each node's border has to be 8MB aligned */
@@ -922,6 +939,16 @@ static void machine_initfn(Object *obj) "Valid values are cpu, mem-ctrl"); } +if (mc->cxl_supported) { +Object *obj =OBJECT (ms); + + ms->cxl_devices_state =g_new0 (CXLState,1 ); +object_property_add_bool (obj,"cxl" , machine_get_cxl, machine_set_cxl); +object_property_set_description (obj,"cxl" , +"Set on/off to enable/disable " +"CXL instantiation" ); + } +if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) { ms->numa_state =g_new0 (NumaState,1 ); object_property_add_bool(obj, "hmat",
@@ -956,6 +983,7 @@ static void machine_finalize(Object *obj)g_free (ms->device_memory);g_free (ms->nvdimms_state);g_free (ms->numa_state); +g_free (ms->cxl_devices_state); }bool machine_usb (MachineState *machine)
hw/i386/pc.c +1 -0
@@ -1739,6 +1739,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = true; mc->smp_props.dies_supported = true; + mc->cxl_supported = true; mc->default_ram_id ="pc.ram" ; object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
include/hw/boards.h +2 -0
@@ -269,6 +269,7 @@ struct MachineClass {bool ignore_boot_device_suffixes;bool smbus_no_migration_support;bool nvdimm_supported; +bool cxl_supported;bool numa_mem_supported;bool auto_enable_numa;SMPCompatProps smp_props;
@@ -360,6 +361,7 @@ struct MachineState {CPUArchIdList *possible_cpus;CpuTopology smp;struct NVDIMMState *nvdimms_state; +struct CXLState *cxl_devices_state;struct NumaState *numa_state; };
include/hw/cxl/cxl.h +4 -0
@@ -17,4 +17,8 @@#define CXL_COMPONENT_REG_BAR_IDX 0#define CXL_DEVICE_REG_BAR_IDX 2 +typedef struct CXLState { +bool is_enabled; +}CXLState ; +#endif
Signature · 1 line
2.32.0
PATCH v6 14/43 hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1
A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in _SB named ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.
Also necessary is to add an exception to scripts/device-crash-test similar to that for exiting pxb as both must created on a PCIexpress host bus.
--- hw/pci-bridge/pci_expander_bridge.c | 86 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 ++ scripts/device-crash-test | 1 + 4 files changed, 98 insertions(+), 2 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +84 -2
@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h"
@@ -56,6 +57,16 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) + +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +struct PXBDev {/*< private >*/ PCIDevice parent_obj;
@@ -68,6 +79,11 @@ struct PXBDev {static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }
@@ -112,11 +128,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;
@@ -218,6 +243,10 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); } +static void pxb_dev_reset (DeviceState *dev) +{ +} +static gint pxb_compare (gconstpointer a,gconstpointer b) {const PXBDev *pxb_a = a, *pxb_b = b;
@@ -389,13 +418,66 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +MachineState *ms =MACHINE (qdev_get_machine ()); + +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } +if (!ms->cxl_devices_state->is_enabled) { +error_setg (errp,"Machine does not have cxl=on" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +pxb_dev_reset (DEVICE (dev)); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; + dc->reset = pxb_dev_reset; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)
hw/pci/pci.c +7 -0
@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);
@@ -2892,6 +2898,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);
include/hw/pci/pci.h +6 -0
@@ -410,6 +410,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"typedef void (*pci_bus_dev_fn )(PCIBus *b,PCIDevice *d,void *opaque);typedef void (*pci_bus_fn )(PCIBus *b,void *opaque);
@@ -769,6 +770,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
scripts/device-crash-test +1 -0
@@ -93,6 +93,7 @@ ERROR_RULE_LIST = [ {'device':'pci-bridge', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0. {'device':'pci-bridge-seat', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0. {'device':'pxb', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0. + {'device':'pxb-cxl', 'expected':True}, # pxb-cxl devices cannot reside on a PCI bus. {'device':'scsi-block', 'expected':True}, # drive property not set {'device':'scsi-generic', 'expected':True}, # drive property not set {'device':'scsi-hd', 'expected':True}, # drive property not set
Signature · 1 line
2.32.0
Re: [PATCH v6 14/43] hw/pxb: Allow creation of a CXL PXB (host bridge)
Quoted 20 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in
_SBnamed ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.
Quoted 6 lines
Also necessary is to add an exception to
scripts/device-crash-testsimilar to that for exiting pxb as both must created on a PCIexpress host bus.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan.Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 15/43 qtest/cxl: Introduce initial test for pxb-cxl only.
In reply to Jonathan CameronInitial test with just pxb-cxl. Other tests will be added alongside functionality.
--- tests/qtest/cxl-test.c | 23 +++++++++++++++++++++++ tests/qtest/meson.build | 4 ++++ 2 files changed, 27 insertions(+)
tests/qtest/cxl-test.c new file +23 -0
@@ -0,0 +1,23 @@ +/* +* QTest testcase for CXL +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "libqtest-single.h" + + +static void cxl_basic_pxb (void ) +{ +qtest_start ("-machine q35,cxl=on -device pxb-cxl,bus=pcie.0" ); +qtest_end (); +} + +int main (int argc,char **argv) +{ +g_test_init (&argc, &argv,NULL ); +qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +return g_test_run (); +}
tests/qtest/meson.build +4 -0
@@ -41,6 +41,9 @@ qtests_pci = \ (config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : []) + \ (config_all_devices.has_key('CONFIG_IVSHMEM_DEVICE') ? ['ivshmem-test'] : []) +qtests_cxl = \ + (config_all_devices.has_key('CONFIG_CXL') ? ['cxl-test'] : []) + qtests_i386 = \ (slirp.found() ? ['pxe-test', 'test-netfilter'] : []) + \ (config_host.has_key('CONFIG_POSIX') ? ['test-filter-mirror'] : []) + \
@@ -75,6 +78,7 @@ qtests_i386 = \ slirp.found() ? ['virtio-net-failover'] : []) + \ (unpack_edk2_blobs ? ['bios-tables-test'] : []) + \ qtests_pci + \ + qtests_cxl + \ ['fdc-test', 'ide-test', 'hd-geo-test',
Signature · 1 line
2.32.0
Re: [PATCH v6 15/43] qtest/cxl: Introduce initial test for pxb-cxl only.
Quoted 4 lines · Jonathan Cameron
Initial test with just pxb-cxl. Other tests will be added alongside functionality.
Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 16/43 hw/cxl/rp: Add a root port
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4
Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.
--- hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 4 +- 5 files changed, 245 insertions(+), 2 deletions(-)
hw/pci-bridge/Kconfig +5 -0
@@ -27,3 +27,8 @@ config DEC_PCI config SIMBA bool + +config CXL + bool + default y if PCI_EXPRESS && PXB + depends on PCI_EXPRESS && MSI_NONBROKEN && PXB
hw/pci-bridge/cxl_root_port.c new file +231 -0
231 lines not on this page · raw message
hw/pci-bridge/meson.build +1 -0
@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c')) pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c')) pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c')) pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c')) +pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c')) # NewWorld PowerMac pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))
hw/pci-bridge/pcie_root_port.c +5 -1
@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)int rc;pci_config_set_interrupt_pin (d->config,1 ); -pci_bridge_initfn (d,TYPE_PCIE_BUS ); +if (d->cap_present &QEMU_PCIE_CAP_CXL ) { +pci_bridge_initfn (d,TYPE_CXL_BUS ); + }else { +pci_bridge_initfn (d,TYPE_PCIE_BUS ); + }pcie_port_init_reg (d); rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,
hw/pci/pci.c +3 -1
@@ -2708,7 +2708,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)object_class_dynamic_cast (klass,INTERFACE_CONVENTIONAL_PCI_DEVICE );ObjectClass *pcie =object_class_dynamic_cast (klass,INTERFACE_PCIE_DEVICE ); -assert (conventional || pcie); +ObjectClass *cxl = +object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE ); +assert (conventional || pcie || cxl); } }
Signature · 1 line
2.32.0
Re: [PATCH v6 16/43] hw/cxl/rp: Add a root port
Quoted 203 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 4 +- 5 files changed, 245 insertions(+), 2 deletions(-)hw/pci-bridge/Kconfig +5
-0@@ -27,3 +27,8 @@ config DEC_PCI config SIMBA bool + +config CXL + bool + default y if PCI_EXPRESS && PXB + depends on PCI_EXPRESS && MSI_NONBROKEN && PXBhw/pci-bridge/cxl_root_port.c new file +161
-0@@ -0,0 +1,231 @@ +/* +* CXL 2.0 Root Port Implementation +* +* Copyright(C) 2020 Intel Corporation. +* +* This library is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; either +* version 2 of the License, or (at your option) any later version. +* +* This library is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +* +* You should have received a copy of the GNU Lesser General Public +* License along with this library; if not, see <http://www.gnu.org/licenses/> +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "qemu/range.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pcie_port.h" +#include "hw/qdev-properties.h" +#include "hw/sysbus.h" +#include "qapi/error.h" +#include "hw/cxl/cxl.h" + +#define CXL_ROOT_PORT_DID 0x7075 + +/* Copied from the gen root port which we derive */ +#define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100 +#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \ + (GEN_PCIE_ROOT_PORT_AER_OFFSET + PCI_ERR_SIZEOF) +#define CXL_ROOT_PORT_DVSEC_OFFSET \ + (GEN_PCIE_ROOT_PORT_ACS_OFFSET + PCI_ACS_SIZEOF) + +typedef struct CXLRootPort { +/*< private >*/ +PCIESlot parent_obj; + +CXLComponentState cxl_cstate; +PCIResReserve res_reserve; +}CXLRootPort ; + +#define TYPE_CXL_ROOT_PORT "cxl-rp" +DECLARE_INSTANCE_CHECKER(CXLRootPort, CXL_ROOT_PORT, TYPE_CXL_ROOT_PORT) + +static void latch_registers (CXLRootPort *crp) +{ +uint32_t *reg_state = crp->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +} + +static void build_dvsecs (CXLComponentState *cxl) +{ +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_extensions ){0 }; +cxl_component_create_dvsec (cxl,EXTENSIONS_PORT_DVSEC_LENGTH , +EXTENSIONS_PORT_DVSEC , +EXTENSIONS_PORT_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_gpf ){ + .rsvd =0 , + .phase1_ctrl =1 ,/* 1μs timeout */ + .phase2_ctrl =1 ,/* 1μs timeout */ + }; +cxl_component_create_dvsec (cxl,GPF_PORT_DVSEC_LENGTH ,GPF_PORT_DVSEC , +GPF_PORT_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_flexbus ){ + .cap =0x26 ,/* IO, Mem, non-MLD */ + .ctrl =0 , + .status =0x26 ,/* same */ + .rcvd_mod_ts_data_phase1 =0xef ,/* WTF? */ + }; +cxl_component_create_dvsec (cxl,PCIE_FLEXBUS_PORT_DVSEC_LENGTH_2_0 , +PCIE_FLEXBUS_PORT_DVSEC , +PCIE_FLEXBUS_PORT_DVSEC_REVID_2_0 , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |CXL_COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + }; +cxl_component_create_dvsec (cxl,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void cxl_rp_realize (DeviceState *dev,Error **errp) +{ +PCIDevice *pci_dev =PCI_DEVICE (dev); +PCIERootPortClass *rpc =PCIE_ROOT_PORT_GET_CLASS (dev); +CXLRootPort *crp =CXL_ROOT_PORT (dev); +CXLComponentState *cxl_cstate = &crp->cxl_cstate; +ComponentRegisters *cregs = &cxl_cstate->crb; +MemoryRegion *component_bar = &cregs->component_registers; +Error *local_err =NULL ; + + rpc->parent_realize (dev, &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } + +int rc = +pci_bridge_qemu_reserve_cap_init (pci_dev,0 , crp->res_reserve, errp); +if (rc <0 ) { + rpc->parent_class.exit (pci_dev); +return ; + } + +if (!crp->res_reserve.io || crp->res_reserve.io ==-1 ) { +pci_word_test_and_clear_mask (pci_dev->wmask +PCI_COMMAND , +PCI_COMMAND_IO ); + pci_dev->wmask[PCI_IO_BASE ] =0 ; + pci_dev->wmask[PCI_IO_LIMIT ] =0 ; + } + + cxl_cstate->dvsec_offset =CXL_ROOT_PORT_DVSEC_OFFSET ; + cxl_cstate->pdev = pci_dev; +build_dvsecs (&crp->cxl_cstate); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_ROOT_PORT ); + +pci_register_bar (pci_dev,CXL_COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + component_bar); +} + +static void cxl_rp_reset (DeviceState *dev) +{ +PCIERootPortClass *rpc =PCIE_ROOT_PORT_GET_CLASS (dev); +CXLRootPort *crp =CXL_ROOT_PORT (dev); + + rpc->parent_reset (dev); + +latch_registers (crp); +} + +static Property gen_rp_props[] = { +DEFINE_PROP_UINT32 ("bus-reserve" , CXLRootPort, res_reserve.bus,-1 ), +DEFINE_PROP_SIZE ("io-reserve" , CXLRootPort, res_reserve.io,-1 ), +DEFINE_PROP_SIZE ("mem-reserve" , CXLRootPort, res_reserve.mem_non_pref,-1 ), +DEFINE_PROP_SIZE ("pref32-reserve" , CXLRootPort, res_reserve.mem_pref_32, +-1 ), +DEFINE_PROP_SIZE ("pref64-reserve" , CXLRootPort, res_reserve.mem_pref_64, +-1 ), +DEFINE_PROP_END_OF_LIST () +}; + +static void cxl_rp_dvsec_write_config (PCIDevice *dev,uint32_t addr, +uint32_t val,int len) +{ +CXLRootPort *crp =CXL_ROOT_PORT (dev); +
hw/pci-bridge/cxl_root_port.c line 171
Quoted 10 lines
hw/pci-bridge/cxl_root_port.c new file +10
-0+if (range_contains (&crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC ], addr)) { +uint8_t *reg = &dev->config[addr]; + addr -= crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC ].lob; +if (addr ==PORT_CONTROL_OFFSET ) { +if (pci_get_word (reg) &PORT_CONTROL_UNMASK_SBR ) { +/* unmask SBR */ + } +if (pci_get_word (reg) &PORT_CONTROL_ALT_MEMID_EN ) { +/* Alt Memory & ID Space Enable */ + }
Can we have LOG_UNIMPs for these null implementations please.
Quoted 30 lines
hw/pci-bridge/cxl_root_port.c new file +30
-0+ } + } +} + +static void cxl_rp_write_config (PCIDevice *d,uint32_t address,uint32_t val, +int len) +{ +uint16_t slt_ctl, slt_sta; + +pcie_cap_slot_get (d, &slt_ctl, &slt_sta); +pci_bridge_write_config (d, address, val, len); +pcie_cap_flr_write_config (d, address, val, len); +pcie_cap_slot_write_config (d, slt_ctl, slt_sta, address, val, len); +pcie_aer_write_config (d, address, val, len); + +cxl_rp_dvsec_write_config (d, address, val, len); +} + +static void cxl_root_port_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *k =PCI_DEVICE_CLASS (oc); +PCIERootPortClass *rpc =PCIE_ROOT_PORT_CLASS (oc); + + k->vendor_id =PCI_VENDOR_ID_INTEL ; + k->device_id =CXL_ROOT_PORT_DID ; + dc->desc ="CXL Root Port" ; + k->revision =0 ; +device_class_set_props (dc, gen_rp_props); + k->config_write = cxl_rp_write_config;
hw/pci-bridge/cxl_root_port.c line 211
Quoted 10 lines
hw/pci-bridge/cxl_root_port.c new file +10
-0+ +device_class_set_parent_realize (dc, cxl_rp_realize, &rpc->parent_realize); +device_class_set_parent_reset (dc, cxl_rp_reset, &rpc->parent_reset); + + rpc->aer_offset =GEN_PCIE_ROOT_PORT_AER_OFFSET ; + rpc->acs_offset =GEN_PCIE_ROOT_PORT_ACS_OFFSET ; + +/* +* Explain +*/
You might as well either expand the comment or drop it, the code makes it clear it is not hot pluggable (although an explanation would be better - given it's PCIe like roots isn't hotplug a thing you see on CXL?)
Quoted 54 lines
hw/pci-bridge/cxl_root_port.c new file +20
-0+ dc->hotpluggable = false; +} + +static const TypeInfo cxl_root_port_info = { + .name =TYPE_CXL_ROOT_PORT , + .parent =TYPE_PCIE_ROOT_PORT , + .instance_size =sizeof (CXLRootPort), + .class_init = cxl_root_port_class_init, + .interfaces = (InterfaceInfo []) { + {INTERFACE_CXL_DEVICE }, + { } + }, +}; + +static void cxl_register (void ) +{ +type_register_static (&cxl_root_port_info); +} + +type_init (cxl_register);hw/pci-bridge/meson.build +1
-0@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c')) pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c')) pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c')) pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c')) +pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c')) # NewWorld PowerMac pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))hw/pci-bridge/pcie_root_port.c +5
-1@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)int rc;pci_config_set_interrupt_pin (d->config,1 ); -pci_bridge_initfn (d,TYPE_PCIE_BUS ); +if (d->cap_present &QEMU_PCIE_CAP_CXL ) { +pci_bridge_initfn (d,TYPE_CXL_BUS ); + }else { +pci_bridge_initfn (d,TYPE_PCIE_BUS ); + }pcie_port_init_reg (d); rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,hw/pci/pci.c +0
-0
hw/pci/pci.c static void pci_device_class_base_init(ObjectClass *klass, void *data) · line 2715
Quoted 9 lines
hw/pci/pci.c +3
-1@@ -2708,7 +2708,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)object_class_dynamic_cast (klass,INTERFACE_CONVENTIONAL_PCI_DEVICE );ObjectClass *pcie =object_class_dynamic_cast (klass,INTERFACE_PCIE_DEVICE ); -assert (conventional || pcie); +ObjectClass *cxl = +object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE ); +assert (conventional || pcie || cxl); } }
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v6 16/43] hw/cxl/rp: Add a root port
Quoted 209 lines · On Tue, 01 Mar 2022 18:08:31 +0000 Alex Bennée <[email protected]> wrote:
Quoted 207 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 4 +- 5 files changed, 245 insertions(+), 2 deletions(-)hw/pci-bridge/Kconfig +5
-0@@ -27,3 +27,8 @@ config DEC_PCI config SIMBA bool + +config CXL + bool + default y if PCI_EXPRESS && PXB + depends on PCI_EXPRESS && MSI_NONBROKEN && PXBhw/pci-bridge/cxl_root_port.c new file +165
-0@@ -0,0 +1,231 @@ +/* +* CXL 2.0 Root Port Implementation +* +* Copyright(C) 2020 Intel Corporation. +* +* This library is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; either +* version 2 of the License, or (at your option) any later version. +* +* This library is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +* +* You should have received a copy of the GNU Lesser General Public +* License along with this library; if not, see <http://www.gnu.org/licenses/> +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "qemu/range.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pcie_port.h" +#include "hw/qdev-properties.h" +#include "hw/sysbus.h" +#include "qapi/error.h" +#include "hw/cxl/cxl.h" + +#define CXL_ROOT_PORT_DID 0x7075 + +/* Copied from the gen root port which we derive */ +#define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100 +#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \ + (GEN_PCIE_ROOT_PORT_AER_OFFSET + PCI_ERR_SIZEOF) +#define CXL_ROOT_PORT_DVSEC_OFFSET \ + (GEN_PCIE_ROOT_PORT_ACS_OFFSET + PCI_ACS_SIZEOF) + +typedef struct CXLRootPort { +/*< private >*/ +PCIESlot parent_obj; + +CXLComponentState cxl_cstate; +PCIResReserve res_reserve; +}CXLRootPort ; + +#define TYPE_CXL_ROOT_PORT "cxl-rp" +DECLARE_INSTANCE_CHECKER(CXLRootPort, CXL_ROOT_PORT, TYPE_CXL_ROOT_PORT) + +static void latch_registers (CXLRootPort *crp) +{ +uint32_t *reg_state = crp->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +} + +static void build_dvsecs (CXLComponentState *cxl) +{ +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_extensions ){0 }; +cxl_component_create_dvsec (cxl,EXTENSIONS_PORT_DVSEC_LENGTH , +EXTENSIONS_PORT_DVSEC , +EXTENSIONS_PORT_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_gpf ){ + .rsvd =0 , + .phase1_ctrl =1 ,/* 1μs timeout */ + .phase2_ctrl =1 ,/* 1μs timeout */ + }; +cxl_component_create_dvsec (cxl,GPF_PORT_DVSEC_LENGTH ,GPF_PORT_DVSEC , +GPF_PORT_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_port_flexbus ){ + .cap =0x26 ,/* IO, Mem, non-MLD */ + .ctrl =0 , + .status =0x26 ,/* same */ + .rcvd_mod_ts_data_phase1 =0xef ,/* WTF? */ + }; +cxl_component_create_dvsec (cxl,PCIE_FLEXBUS_PORT_DVSEC_LENGTH_2_0 , +PCIE_FLEXBUS_PORT_DVSEC , +PCIE_FLEXBUS_PORT_DVSEC_REVID_2_0 , dvsec); + + dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |CXL_COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + }; +cxl_component_create_dvsec (cxl,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void cxl_rp_realize (DeviceState *dev,Error **errp) +{ +PCIDevice *pci_dev =PCI_DEVICE (dev); +PCIERootPortClass *rpc =PCIE_ROOT_PORT_GET_CLASS (dev); +CXLRootPort *crp =CXL_ROOT_PORT (dev); +CXLComponentState *cxl_cstate = &crp->cxl_cstate; +ComponentRegisters *cregs = &cxl_cstate->crb; +MemoryRegion *component_bar = &cregs->component_registers; +Error *local_err =NULL ; + + rpc->parent_realize (dev, &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } + +int rc = +pci_bridge_qemu_reserve_cap_init (pci_dev,0 , crp->res_reserve, errp); +if (rc <0 ) { + rpc->parent_class.exit (pci_dev); +return ; + } + +if (!crp->res_reserve.io || crp->res_reserve.io ==-1 ) { +pci_word_test_and_clear_mask (pci_dev->wmask +PCI_COMMAND , +PCI_COMMAND_IO ); + pci_dev->wmask[PCI_IO_BASE ] =0 ; + pci_dev->wmask[PCI_IO_LIMIT ] =0 ; + } + + cxl_cstate->dvsec_offset =CXL_ROOT_PORT_DVSEC_OFFSET ; + cxl_cstate->pdev = pci_dev; +build_dvsecs (&crp->cxl_cstate); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_ROOT_PORT ); + +pci_register_bar (pci_dev,CXL_COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + component_bar); +} + +static void cxl_rp_reset (DeviceState *dev) +{ +PCIERootPortClass *rpc =PCIE_ROOT_PORT_GET_CLASS (dev); +CXLRootPort *crp =CXL_ROOT_PORT (dev); + + rpc->parent_reset (dev); + +latch_registers (crp); +} + +static Property gen_rp_props[] = { +DEFINE_PROP_UINT32 ("bus-reserve" , CXLRootPort, res_reserve.bus,-1 ), +DEFINE_PROP_SIZE ("io-reserve" , CXLRootPort, res_reserve.io,-1 ), +DEFINE_PROP_SIZE ("mem-reserve" , CXLRootPort, res_reserve.mem_non_pref,-1 ), +DEFINE_PROP_SIZE ("pref32-reserve" , CXLRootPort, res_reserve.mem_pref_32, +-1 ), +DEFINE_PROP_SIZE ("pref64-reserve" , CXLRootPort, res_reserve.mem_pref_64, +-1 ), +DEFINE_PROP_END_OF_LIST () +}; + +static void cxl_rp_dvsec_write_config (PCIDevice *dev,uint32_t addr, +uint32_t val,int len) +{ +CXLRootPort *crp =CXL_ROOT_PORT (dev); + +if (range_contains (&crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC ], addr)) { +uint8_t *reg = &dev->config[addr]; + addr -= crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC ].lob; +if (addr ==PORT_CONTROL_OFFSET ) {
Quoted 8 lines
hw/pci-bridge/cxl_root_port.c line 171
Quoted 6 lines
hw/pci-bridge/cxl_root_port.c new file +6
-0+if (pci_get_word (reg) &PORT_CONTROL_UNMASK_SBR ) { +/* unmask SBR */ + } +if (pci_get_word (reg) &PORT_CONTROL_ALT_MEMID_EN ) { +/* Alt Memory & ID Space Enable */ + }Can we have
LOG_UNIMPsfor these null implementations please.
Added. I have implementation for SBR unmasking, but that will be part of a future patch set so definitely makes sense to log for now.
The alt memory stuff is about making a CXL 1.1 device that is attached to a CXL switch work (?). In that case we have to make it magically appear as an RCiEP despite being below a switch and hence have to mess around with routing to make that work - effectively we are tunneling an extra PCI topology through the existing one.
Given we generally don't care today about emulating CXL 1.1 devices at all, it may never be implemented in QEMU.
So definitely good to log that...
Quoted 5 lines
Quoted 5 lines
hw/pci-bridge/cxl_root_port.c new file +4
-0+ } + } +} +
...
Quoted 18 lines
Quoted 18 lines
hw/pci-bridge/cxl_root_port.c new file +18
-0+static void cxl_root_port_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *k =PCI_DEVICE_CLASS (oc); +PCIERootPortClass *rpc =PCIE_ROOT_PORT_CLASS (oc); + + k->vendor_id =PCI_VENDOR_ID_INTEL ; + k->device_id =CXL_ROOT_PORT_DID ; + dc->desc ="CXL Root Port" ; + k->revision =0 ; +device_class_set_props (dc, gen_rp_props); + k->config_write = cxl_rp_write_config; + +device_class_set_parent_realize (dc, cxl_rp_realize, &rpc->parent_realize); +device_class_set_parent_reset (dc, cxl_rp_reset, &rpc->parent_reset); + + rpc->aer_offset =GEN_PCIE_ROOT_PORT_AER_OFFSET ; + rpc->acs_offset =GEN_PCIE_ROOT_PORT_ACS_OFFSET ;
Quoted 8 lines
hw/pci-bridge/cxl_root_port.c line 211
Quoted 4 lines
hw/pci-bridge/cxl_root_port.c new file +4
-0+ +/* +* Explain +*/ You might as well either expand the comment or drop it, the code makes it clear it is not hot pluggable (although an explanation would be better - given it's PCIe like roots isn't hotplug a thing you see on CXL?)
Hotplug of devices is indeed something we care about in CXL though for now it's untested. Hotplug of the root ports themselves is going to be 'interesting' due to the memory decoding targeting the ports. For now I've just dropped the pointless comment.
Ben: If you had a planned explanation to put here, can you remember what it was? We can add it later though - for now just dropping the comment seems best way forward to me.
Thanks,
Jonathan
PATCH v6 17/43 hw/cxl/device: Add a memory device (8.2.8.5)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Although the memory device is configured like a normal PCIe device, the memory traffic is on an entirely separate bus conceptually (using the same physical wires as PCIe, but different protocol).
Once the CXL topology is fully configure and address decoders committed, the guest physical address for the memory device is part of a larger window which is owned by the platform. The creation of these windows is later in this series.
The following example will create a 256M device in a 512M window:
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
Note: Dropped PCDIMM info interfaces for now. They can be added if appropriate at a later date.
--- hw/cxl/cxl-mailbox-utils.c | 47 ++++++++++ hw/mem/Kconfig | 5 ++ hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 +++++ include/hw/pci/pci_ids.h | 1 + 7 files changed, 248 insertions(+)
hw/cxl/cxl-mailbox-utils.c +47 -0
@@ -50,6 +50,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -216,6 +218,48 @@ static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +static ret_code cmd_identify_memory_device (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + + id->total_capacity = size / (256 <<20 ); + id->persistent_capacity = size / (256 <<20 ); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -233,8 +277,11 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, + [IDENTIFY ][MEMORY_DEVICE ] = {"IDENTIFY_MEMORY_DEVICE" , + cmd_identify_memory_device, 0, 0 }, }; +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
hw/mem/Kconfig +5 -0
@@ -11,3 +11,8 @@ config NVDIMM config SPARSE_MEM bool + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICE
hw/mem/cxl_type3.c new file +170 -0
170 lines not on this page · raw message
hw/mem/meson.build +1 -0
@@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
include/hw/cxl/cxl.h +2 -0
@@ -17,6 +17,8 @@#define CXL_COMPONENT_REG_BAR_IDX 0#define CXL_DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +typedef struct CXLState {bool is_enabled; }CXLState ;
include/hw/cxl/cxl_pci.h +22 -0
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct cxl_dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct cxl_dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct cxl_dvsec_port_extensions {struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06
Signature · 1 line
2.32.0
Re: [PATCH v6 17/43] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 25 lines · On Fri, 11 Feb 2022 12:07:21 +0000 Jonathan Cameron…
From: Ben Widawsky <[email protected]>
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Although the memory device is configured like a normal PCIe device, the memory traffic is on an entirely separate bus conceptually (using the same physical wires as PCIe, but different protocol).
Once the CXL topology is fully configure and address decoders committed, the guest physical address for the memory device is part of a larger window which is owned by the platform. The creation of these windows is later in this series.
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"Note: Dropped PCDIMM info interfaces for now. They can be added if appropriate at a later date.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
...
Quoted 21 lines
hw/mem/cxl_type3.c new file +21
-0+ +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegion *mr; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +error_setg (errp,"memdev property must be set" ); +return ; + } +memory_region_set_nonvolatile (mr, true); +memory_region_set_enabled (mr, true); +host_memory_backend_set_mapped (ct3d->hostmem, true); + ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size; +} +
hw/mem/cxl_type3.c line 93
Quoted 10 lines
hw/mem/cxl_type3.c new file +10
-0+ +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->hostmem) {
Ben pointed out in reply to v5 that this is backwards. I'll fix in v7. Clearly some of the cxl_setup_memory() logic may also not be needed seeing as it wasn't running.
Quoted 6 lines
hw/mem/cxl_type3.c new file +6
-0+cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); +
PATCH v6 18/43 qtests/cxl: Add initial root port and CXL type3 tests
In reply to Jonathan CameronAt this stage we can boot configurations with host bridges, root ports and type 3 memory devices, so add appropriate tests.
--- tests/qtest/cxl-test.c | 126 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+)
tests/qtest/cxl-test.c +126 -0
@@ -8,6 +8,47 @@#include "qemu/osdep.h" #include "libqtest-single.h" +#define QEMU_PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " + +#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " + +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " + +/* Dual ports on first pxb */ +#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " + +/* Dual ports on each of the pxb instances */ +#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " + +#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " + +#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,id=cxl-pmem2,size=256M " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,id=cxl-pmem3,size=256M " + +static void cxl_basic_hb (void ) +{ +qtest_start ("-machine q35,cxl=on" ); +qtest_end (); +}static void cxl_basic_pxb (void ) {
@@ -15,9 +56,94 @@ static void cxl_basic_pxb(void)qtest_end (); } +static void cxl_pxb_with_window (void ) +{ +qtest_start (QEMU_PXB_CMD ); +qtest_end (); +} + +static void cxl_2pxb_with_window (void ) +{ +qtest_start (QEMU_2PXB_CMD ); +qtest_end (); +} + +static void cxl_root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_RP); +qtest_end (); +} + +static void cxl_2root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_2RP); +qtest_end (); +} + +static void cxl_t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_1pxb_2rp_2t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_2RP QEMU_2T3D, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_2pxb_4rp_4t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_2PXB_CMD QEMU_4RP QEMU_4T3D, + tmpfs, tmpfs, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} +int main (int argc,char **argv) {g_test_init (&argc, &argv,NULL ); + +qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb);qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); +qtest_add_func ("/pci/cxl/pxb_x2_with_window" , cxl_2pxb_with_window); +qtest_add_func ("/pci/cxl/rp" , cxl_root_port); +qtest_add_func ("/pci/cxl/rp_x2" , cxl_2root_port); +qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); +qtest_add_func ("/pci/cxl/rp_x2_type3_x2" , cxl_1pxb_2rp_2t3d); +qtest_add_func ("/pci/cxl/pxb_x2_root_port_x4_type3_x4" , cxl_2pxb_4rp_4t3d);return g_test_run (); }
Signature · 1 line
2.32.0
Re: [PATCH v6 18/43] qtests/cxl: Add initial root port and CXL type3 tests
Quoted 94 lines · Jonathan Cameron
At this stage we can boot configurations with host bridges, root ports and type 3 memory devices, so add appropriate tests.
Signed-off-by Jonathan Cameron <[email protected]>--- tests/qtest/cxl-test.c | 126 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+)tests/qtest/cxl-test.c +70
-0@@ -8,6 +8,47 @@#include "qemu/osdep.h" #include "libqtest-single.h" +#define QEMU_PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " + +#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " + +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " + +/* Dual ports on first pxb */ +#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " + +/* Dual ports on each of the pxb instances */ +#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " + +#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " + +#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,id=cxl-pmem2,size=256M " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,id=cxl-pmem3,size=256M " + +static void cxl_basic_hb (void ) +{ +qtest_start ("-machine q35,cxl=on" ); +qtest_end (); +}static void cxl_basic_pxb (void ) {@@ -15,9 +56,94 @@ static void cxl_basic_pxb(void)qtest_end (); } +static void cxl_pxb_with_window (void ) +{ +qtest_start (QEMU_PXB_CMD ); +qtest_end (); +} + +static void cxl_2pxb_with_window (void ) +{ +qtest_start (QEMU_2PXB_CMD ); +qtest_end (); +} + +static void cxl_root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_RP); +qtest_end (); +} + +static void cxl_2root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_2RP); +qtest_end (); +} + +static void cxl_t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs;
tests/qtest/cxl-test.c static void cxl_basic_pxb(void) · line 97
Quoted 10 lines
tests/qtest/cxl-test.c +10
-0+ + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE);
For future reference you could do:
g_autoptr(GString) cmdline = g_string_new(NULL);
and drop the free ;-)
Quoted 42 lines
tests/qtest/cxl-test.c +39
-0+} + +static void cxl_1pxb_2rp_2t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_2RP QEMU_2T3D, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_2pxb_4rp_4t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_2PXB_CMD QEMU_4RP QEMU_4T3D, + tmpfs, tmpfs, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} +int main (int argc,char **argv) {g_test_init (&argc, &argv,NULL ); + +qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb);
tests/qtest/cxl-test.c static void cxl_basic_pxb(void) · line 149
Quoted 10 lines
tests/qtest/cxl-test.c +7
-0qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); +qtest_add_func ("/pci/cxl/pxb_x2_with_window" , cxl_2pxb_with_window); +qtest_add_func ("/pci/cxl/rp" , cxl_root_port); +qtest_add_func ("/pci/cxl/rp_x2" , cxl_2root_port); +qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); +qtest_add_func ("/pci/cxl/rp_x2_type3_x2" , cxl_1pxb_2rp_2t3d); +qtest_add_func ("/pci/cxl/pxb_x2_root_port_x4_type3_x4" , cxl_2pxb_4rp_4t3d);return g_test_run (); }
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v6 18/43] qtests/cxl: Add initial root port and CXL type3 tests
Quoted 109 lines · On Tue, 01 Mar 2022 18:11:25 +0000 Alex Bennée <[email protected]> wrote:
Quoted 94 lines · Jonathan Cameron
At this stage we can boot configurations with host bridges, root ports and type 3 memory devices, so add appropriate tests.
Signed-off-by Jonathan Cameron <[email protected]>--- tests/qtest/cxl-test.c | 126 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+)tests/qtest/cxl-test.c +70
-0@@ -8,6 +8,47 @@#include "qemu/osdep.h" #include "libqtest-single.h" +#define QEMU_PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " + +#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " + +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " + +/* Dual ports on first pxb */ +#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " + +/* Dual ports on each of the pxb instances */ +#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " + +#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " + +#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,id=cxl-pmem2,size=256M " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,id=cxl-pmem3,size=256M " + +static void cxl_basic_hb (void ) +{ +qtest_start ("-machine q35,cxl=on" ); +qtest_end (); +}static void cxl_basic_pxb (void ) {@@ -15,9 +56,94 @@ static void cxl_basic_pxb(void)qtest_end (); } +static void cxl_pxb_with_window (void ) +{ +qtest_start (QEMU_PXB_CMD ); +qtest_end (); +} + +static void cxl_2pxb_with_window (void ) +{ +qtest_start (QEMU_2PXB_CMD ); +qtest_end (); +} + +static void cxl_root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_RP); +qtest_end (); +} + +static void cxl_2root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_2RP); +qtest_end (); +} + +static void cxl_t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs;tests/qtest/cxl-test.c static void cxl_basic_pxb(void) · line 97
Quoted 10 lines
tests/qtest/cxl-test.c +10
-0+ + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE);For future reference you could do:
Quoted 3 lines
g_autoptr(GString) cmdline = g_string_new(NULL);and drop the free ;-)
nice. Done here and in later qtest patches.
Thanks,
Jonathan
PATCH v6 19/43 hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. Whilst the implementation currently does no validity checks on the encoder set up, future work will add sanity checking specific to the type of cxl component.
--- hw/mem/cxl_type3.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
hw/mem/cxl_type3.c +54 -0
@@ -61,6 +61,56 @@ static void build_dvsecs(CXLType3Dev *ct3d) REG_LOC_DVSEC_REVID, dvsec); } +static void hdm_decoder_commit (CXLType3Dev *ct3d,int which) +{ +ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; +uint32_t *cache_mem = cregs->cache_mem_registers; + +assert (which ==0 ); + +/* TODO: Sanity checks that the decoder is possible */ +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERR ,0 ); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); +} + +static void ct3d_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; +CXLType3Dev *ct3d =container_of (cxl_cstate, CXLType3Dev, cxl_cstate); +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; +int which_hdm =-1 ; + +assert (size ==4 ); + +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); + which_hdm =0 ; +break ; +default : +break ; + } + +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) { +hdm_decoder_commit (ct3d, which_hdm); + } +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; + +g_free ((void *)regs->special_ops); +} +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) {MemoryRegion *mr;
@@ -103,6 +153,9 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) ct3d->cxl_cstate.pdev = pci_dev;build_dvsecs (ct3d); + regs->special_ops =g_new0 (MemoryRegionOps,1 ); + regs->special_ops->write = ct3d_reg_write; +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate,TYPE_CXL_TYPE3_DEV );
@@ -155,6 +208,7 @@ static const TypeInfo ct3d_info = { .parent = TYPE_PCI_DEVICE, .class_init = ct3_class_init, .instance_size = sizeof(CXLType3Dev), + .instance_finalize = ct3_finalize, .interfaces = (InterfaceInfo []) { {INTERFACE_CXL_DEVICE }, {INTERFACE_PCIE_DEVICE },
Signature · 1 line
2.32.0
Re: [PATCH v6 19/43] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
Quoted 49 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. Whilst the implementation currently does no validity checks on the encoder set up, future work will add sanity checking specific to the type of cxl component.
Signed-off-by Ben Widawsky <[email protected]>Co-developed-by Jonathan Cameron <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/mem/cxl_type3.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)hw/mem/cxl_type3.c +15
-0@@ -61,6 +61,56 @@ static void build_dvsecs(CXLType3Dev *ct3d) REG_LOC_DVSEC_REVID, dvsec); } +static void hdm_decoder_commit (CXLType3Dev *ct3d,int which) +{ +ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; +uint32_t *cache_mem = cregs->cache_mem_registers; + +assert (which ==0 ); + +/* TODO: Sanity checks that the decoder is possible */ +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERR ,0 ); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); +} + +static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
hw/mem/cxl_type3.c static void build_dvsecs(CXLType3Dev *ct3d) · line 88
Quoted 10 lines
hw/mem/cxl_type3.c +10
-0+ unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; +CXLType3Dev *ct3d =container_of (cxl_cstate, CXLType3Dev, cxl_cstate); +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; +int which_hdm =-1 ; + +assert (size ==4 );
Maybe add:
g_assert(offset <= (CXL2_COMPONENT_CM_REGION_SIZE >> 2));
Quoted 13 lines
hw/mem/cxl_type3.c +13
-0+ +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); + which_hdm =0 ; +break ; +default : +break ; + } + +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) { +hdm_decoder_commit (ct3d, which_hdm);
hw/mem/cxl_type3.c static void build_dvsecs(CXLType3Dev *ct3d) · line 111
Quoted 10 lines
hw/mem/cxl_type3.c +10
-0+ } +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; + +g_free ((void *)regs->special_ops);
nit: you don't need to cast her.
Quoted 16 lines
hw/mem/cxl_type3.c +5
-0+} +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) {MemoryRegion *mr;@@ -103,6 +153,9 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) ct3d->cxl_cstate.pdev = pci_dev;build_dvsecs (ct3d); + regs->special_ops =g_new0 (MemoryRegionOps,1 ); + regs->special_ops->write = ct3d_reg_write; +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate,TYPE_CXL_TYPE3_DEV );
hw/mem/cxl_type3.c static const TypeInfo ct3d_info = { · line 214
Quoted 7 lines
hw/mem/cxl_type3.c +1
-0@@ -155,6 +208,7 @@ static const TypeInfo ct3d_info = { .parent = TYPE_PCI_DEVICE, .class_init = ct3_class_init, .instance_size = sizeof(CXLType3Dev), + .instance_finalize = ct3_finalize, .interfaces = (InterfaceInfo []) { {INTERFACE_CXL_DEVICE }, {INTERFACE_PCIE_DEVICE },
Signature · 1 line
Alex Bennée
Re: [PATCH v6 19/43] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
Quoted 64 lines · On Tue, 01 Mar 2022 18:17:35 +0000 Alex Bennée <[email protected]> wrote:
Quoted 49 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. Whilst the implementation currently does no validity checks on the encoder set up, future work will add sanity checking specific to the type of cxl component.
Signed-off-by Ben Widawsky <[email protected]>Co-developed-by Jonathan Cameron <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/mem/cxl_type3.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)hw/mem/cxl_type3.c +15
-0@@ -61,6 +61,56 @@ static void build_dvsecs(CXLType3Dev *ct3d) REG_LOC_DVSEC_REVID, dvsec); } +static void hdm_decoder_commit (CXLType3Dev *ct3d,int which) +{ +ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; +uint32_t *cache_mem = cregs->cache_mem_registers; + +assert (which ==0 ); + +/* TODO: Sanity checks that the decoder is possible */ +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERR ,0 ); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); +} + +static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,hw/mem/cxl_type3.c static void build_dvsecs(CXLType3Dev *ct3d) · line 88
Quoted 10 lines
hw/mem/cxl_type3.c +10
-0+ unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; +CXLType3Dev *ct3d =container_of (cxl_cstate, CXLType3Dev, cxl_cstate); +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; +int which_hdm =-1 ; + +assert (size ==4 );Maybe add:
Quoted 1 line
g_assert(offset <= (CXL2_COMPONENT_CM_REGION_SIZE >> 2));
offset is in bytes here, but agreed without the >> 2.
Could move this over to use a 4 byte step if you think that's a good idea.
For other comments changed as suggested.
Thanks,
Jonathan
PATCH v6 20/43 hw/cxl/device: Add some trivial commands
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
GET_FW_INFO and GET_PARTITION_INFO, for this emulation, is equivalent to info already returned in the IDENTIFY command. To have a more robust implementation, add those.
--- hw/cxl/cxl-mailbox-utils.c | 69 +++++++++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +68 -1
@@ -44,6 +44,8 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + FIRMWARE_UPDATE = 0x02, +#define GET_INFO 0x0 TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1
@@ -52,6 +54,8 @@ enum {#define GET_LOG 0x1 IDENTIFY = 0x40,#define MEMORY_DEVICE 0x0 + CCLS = 0x41, +#define GET_PARTITION_INFO 0x0 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -114,6 +118,39 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 );DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +/* 8.2.9.2.1 */ +static ret_code cmd_firmware_update_get_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint8_t slots_supported; +uint8_t slot_info; +uint8_t caps; +uint8_t rsvd[0xd ]; +char fw_rev1[0x10 ]; +char fw_rev2[0x10 ]; +char fw_rev3[0x10 ]; +char fw_rev4[0x10 ]; + } __attribute__((packed)) *fw_info; +_Static_assert (sizeof (*fw_info) ==0x50 ,"Bad firmware info size" ); + +if (cxl_dstate->pmem_size < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + fw_info = (void *)cmd->payload; +memset (fw_info,0 ,sizeof (*fw_info)); + + fw_info->slots_supported =2 ; + fw_info->slot_info =BIT (0 ) |BIT (3 ); + fw_info->caps =0 ; +snprintf (fw_info->fw_rev1,0x10 ,"BWFW VERSION %02d" ,0 ); + + *len =sizeof (*fw_info); +return CXL_MBOX_SUCCESS ; +} +/* 8.2.9.3.1 */ static ret_code cmd_timestamp_get(struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate,
@@ -260,6 +297,33 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +static ret_code cmd_ccls_get_partition_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint64_t active_vmem; +uint64_t active_pmem; +uint64_t next_vmem; +uint64_t next_pmem; + } __attribute__((packed)) *part_info = (void *)cmd->payload; +_Static_assert (sizeof (*part_info) ==0x20 ,"Bad get partition info size" ); +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + +/* PMEM only */ + part_info->active_vmem =0 ; + part_info->next_vmem =0 ; + part_info->active_pmem = size / (256 <<20 ); + part_info->next_pmem = part_info->active_pmem; + + *len =sizeof (*part_info); +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -273,15 +337,18 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, + [FIRMWARE_UPDATE][GET_INFO] = { "FIRMWARE_UPDATE_GET_INFO", + cmd_firmware_update_get_info, 0, 0 }, [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE", cmd_identify_memory_device, 0, 0 }, + [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", + cmd_ccls_get_partition_info, 0, 0 }, }; -void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
2.32.0
Re: [PATCH v6 20/43] hw/cxl/device: Add some trivial commands
Quoted 45 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
GET_FW_INFOandGET_PARTITION_INFO, for this emulation, is equivalent to info already returned in the IDENTIFY command. To have a more robust implementation, add those.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 69 +++++++++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-)hw/cxl/cxl-mailbox-utils.c +11
-0@@ -44,6 +44,8 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + FIRMWARE_UPDATE = 0x02, +#define GET_INFO 0x0 TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1@@ -52,6 +54,8 @@ enum {#define GET_LOG 0x1 IDENTIFY = 0x40,#define MEMORY_DEVICE 0x0 + CCLS = 0x41, +#define GET_PARTITION_INFO 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -114,6 +118,39 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 );DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +/* 8.2.9.2.1 */ +static ret_code cmd_firmware_update_get_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint8_t slots_supported;
hw/cxl/cxl-mailbox-utils.c DEFINE_MAILBOX_HANDLER_NOP(events_clear_records); · line 137
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +9
-0+uint8_t slot_info; +uint8_t caps; +uint8_t rsvd[0xd ]; +char fw_rev1[0x10 ]; +char fw_rev2[0x10 ]; +char fw_rev3[0x10 ]; +char fw_rev4[0x10 ]; + } __attribute__((packed)) *fw_info; +_Static_assert (sizeof (*fw_info) ==0x50 ,"Bad firmware info size");
note: we have QEMU_PACKED, QEMU_BUILD_BUG_ON and friends in compiler.h which are preferred for potential compiler portability reasons.
hw/cxl/cxl-mailbox-utils.c DEFINE_MAILBOX_HANDLER_NOP(events_clear_records); · line 148
Quoted 12 lines
hw/cxl/cxl-mailbox-utils.c +12
-0+ +if (cxl_dstate->pmem_size < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + fw_info = (void *)cmd->payload; +memset (fw_info,0 ,sizeof (*fw_info)); + + fw_info->slots_supported =2 ; + fw_info->slot_info =BIT (0 ) |BIT (3 ); + fw_info->caps =0 ; +snprintf (fw_info->fw_rev1,0x10 ,"BWFW VERSION %02d" ,0 );
Given you have a fixed string here could you not:
pstrcpy(fw_info->fw_rev1, 0x10, "BWFW VERSION 0");
Quoted 62 lines
hw/cxl/cxl-mailbox-utils.c +5
-0+ + *len =sizeof (*fw_info); +return CXL_MBOX_SUCCESS ; +} +/* 8.2.9.3.1 */ static ret_code cmd_timestamp_get(struct cxl_cmd *cmd,CXLDeviceState
*cxl_dstate,hw/cxl/cxl-mailbox-utils.c +31
-1@@ -260,6 +297,33 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +static ret_code cmd_ccls_get_partition_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint64_t active_vmem; +uint64_t active_pmem; +uint64_t next_vmem; +uint64_t next_pmem; + } __attribute__((packed)) *part_info = (void *)cmd->payload; +_Static_assert (sizeof (*part_info) ==0x20 ,"Bad get partition info size" ); +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + +/* PMEM only */ + part_info->active_vmem =0 ; + part_info->next_vmem =0 ; + part_info->active_pmem = size / (256 <<20 ); + part_info->next_pmem = part_info->active_pmem; + + *len =sizeof (*part_info); +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)@@ -273,15 +337,18 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, + [FIRMWARE_UPDATE][GET_INFO] = { "FIRMWARE_UPDATE_GET_INFO", + cmd_firmware_update_get_info, 0, 0 }, [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE", cmd_identify_memory_device, 0, 0 }, + [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", + cmd_ccls_get_partition_info, 0, 0 }, }; -void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
Alex Bennée
Re: [PATCH v6 20/43] hw/cxl/device: Add some trivial commands
Quoted 10 lines · On Tue, 01 Mar 2022 18:46:30 +0000 Alex Bennée <[email protected]> wrote:
Quoted 8 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
GET_FW_INFOandGET_PARTITION_INFO, for this emulation, is equivalent to info already returned in the IDENTIFY command. To have a more robust implementation, add those.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Hi Alex,
Quoted 42 lines
Quoted 42 lines
--- hw/cxl/cxl-mailbox-utils.c | 69 +++++++++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-)hw/cxl/cxl-mailbox-utils.c +16
-0@@ -44,6 +44,8 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + FIRMWARE_UPDATE = 0x02, +#define GET_INFO 0x0 TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1@@ -52,6 +54,8 @@ enum {#define GET_LOG 0x1 IDENTIFY = 0x40,#define MEMORY_DEVICE 0x0 + CCLS = 0x41, +#define GET_PARTITION_INFO 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -114,6 +118,39 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);DEFINE_MAILBOX_HANDLER_ZEROED (events_get_interrupt_policy,4 );DEFINE_MAILBOX_HANDLER_NOP (events_set_interrupt_policy); +/* 8.2.9.2.1 */ +static ret_code cmd_firmware_update_get_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint8_t slots_supported; +uint8_t slot_info; +uint8_t caps; +uint8_t rsvd[0xd ]; +char fw_rev1[0x10 ]; +char fw_rev2[0x10 ];
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);
Quoted 5 lines
hw/cxl/cxl-mailbox-utils.c +4
-0+char fw_rev3[0x10 ]; +char fw_rev4[0x10 ]; + } __attribute__((packed)) *fw_info; +_Static_assert (sizeof (*fw_info) ==0x50 ,"Bad firmware info size");note: we have
QEMU_PACKED,QEMU_BUILD_BUG_ONand friends incompiler.hwhich are preferred for potential compiler portability reasons.
Replaced all instances of alignment, packed and Static_assert in the patch set with the compiler.h equivalents. Not that has minor affect on some earlier patch sets but feels trivial enough that I've kept RBs etc.
Quoted 16 lines
hw/cxl/cxl-mailbox-utils.c DEFINE_MAILBOX_HANDLER_NOP(events_clear_records); · line 148
Quoted 13 lines
hw/cxl/cxl-mailbox-utils.c +12
-0+ +if (cxl_dstate->pmem_size < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + fw_info = (void *)cmd->payload; +memset (fw_info,0 ,sizeof (*fw_info)); + + fw_info->slots_supported =2 ; + fw_info->slot_info =BIT (0 ) |BIT (3 ); + fw_info->caps =0 ; +snprintf (fw_info->fw_rev1,0x10 ,"BWFW VERSION %02d" ,0 );Given you have a fixed string here could you not:
Quoted 1 line
pstrcpy(fw_info->fw_rev1, 0x10, "BWFW VERSION 0");
Make sense.
Quoted 59 lines
Quoted 59 lines
hw/cxl/cxl-mailbox-utils.c +5
-0+ + *len =sizeof (*fw_info); +return CXL_MBOX_SUCCESS ; +} +/* 8.2.9.3.1 */ static ret_code cmd_timestamp_get(struct cxl_cmd *cmd,CXLDeviceState
*cxl_dstate,hw/cxl/cxl-mailbox-utils.c +31
-0@@ -260,6 +297,33 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +static ret_code cmd_ccls_get_partition_info (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint64_t active_vmem; +uint64_t active_pmem; +uint64_t next_vmem; +uint64_t next_pmem; + } __attribute__((packed)) *part_info = (void *)cmd->payload; +_Static_assert (sizeof (*part_info) ==0x20 ,"Bad get partition info size" ); +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + +/* PMEM only */ + part_info->active_vmem =0 ; + part_info->next_vmem =0 ; + part_info->active_pmem = size / (256 <<20 ); + part_info->next_pmem = part_info->active_pmem; + + *len =sizeof (*part_info); +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)@@ -273,15 +337,18 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE }, + [FIRMWARE_UPDATE][GET_INFO] = { "FIRMWARE_UPDATE_GET_INFO", + cmd_firmware_update_get_info, 0, 0 }, [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE }, [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE", cmd_identify_memory_device, 0, 0 }, + [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", + cmd_ccls_get_partition_info, 0, 0 }, };
Quoted 1 line
hw/cxl/cxl-mailbox-utils.c static struct cxl_cmd cxl_cmd_set[256][256] = { · line 284
Quoted 1 line
hw/cxl/cxl-mailbox-utils.c +0
-1-
Also fixed this bit of rebasing mess up.
Thanks,
Jonathan
Quoted 5 lines
Quoted 3 lines
hw/cxl/cxl-mailbox-utils.c +0
-0void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
PATCH v6 21/43 hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This should introduce no change. Subsequent work will make use of this new class member.
--- hw/cxl/cxl-mailbox-utils.c | 3 +++ hw/mem/cxl_type3.c | 24 +++++++++--------------- include/hw/cxl/cxl_device.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-)
hw/cxl/cxl-mailbox-utils.c +3 -0
@@ -278,6 +278,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, } __attribute__((packed)) *id;_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d);uint64_t size = cxl_dstate->pmem_size;if (!QEMU_IS_ALIGNED (size,256 <<20 )) {
@@ -292,6 +294,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, id->total_capacity = size / (256 <<20 ); id->persistent_capacity = size / (256 <<20 ); + id->lsa_size = cvc->get_lsa_size (ct3d); *len =sizeof (*id);return CXL_MBOX_SUCCESS ;
hw/mem/cxl_type3.c +9 -15
@@ -13,21 +13,6 @@#include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" -typedef struct cxl_type3_dev { -/* Private */ -PCIDevice parent_obj; - -/* Properties */ -uint64_t size; -HostMemoryBackend *hostmem; - -/* State */ -CXLComponentState cxl_cstate; -CXLDeviceState cxl_dstate; -}CXLType3Dev ; - -#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) -static void build_dvsecs (CXLType3Dev *ct3d) {CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
@@ -186,10 +171,16 @@ static Property ct3_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static uint64_t get_lsa_size (CXLType3Dev *ct3d) +{ +return 0 ; +} +static void ct3_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +CXLType3Class *cvc =CXL_TYPE3_DEV_CLASS (oc); pc->realize = ct3_realize; pc->class_id =PCI_CLASS_STORAGE_EXPRESS ;
@@ -201,11 +192,14 @@ static void ct3_class_init(ObjectClass *oc, void *data) dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;device_class_set_props (dc, ct3_props); + + cvc->get_lsa_size = get_lsa_size; }static const TypeInfo ct3d_info = { .name =TYPE_CXL_TYPE3_DEV , .parent =TYPE_PCI_DEVICE , + .class_size =sizeof (struct CXLType3Class ), .class_init = ct3_class_init, .instance_size =sizeof (CXLType3Dev), .instance_finalize = ct3_finalize,
include/hw/cxl/cxl_device.h +29 -0
@@ -230,4 +230,33 @@ REG64(CXL_MEM_DEV_STS, 0) FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; +HostMemoryBackend *lsa; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#ifndef TYPE_CXL_TYPE3_DEV +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#endif + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) +OBJECT_DECLARE_TYPE(CXLType3Device, CXLType3Class, CXL_TYPE3_DEV) + +struct CXLType3Class { +/* Private */ +PCIDeviceClass parent_class; + +/* public */ +uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); +}; +#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 21/43] hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
Quoted 47 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This should introduce no change. Subsequent work will make use of this new class member.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 3 +++ hw/mem/cxl_type3.c | 24 +++++++++--------------- include/hw/cxl/cxl_device.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-)hw/cxl/cxl-mailbox-utils.c +3
-0@@ -278,6 +278,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, } __attribute__((packed)) *id;_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d);uint64_t size = cxl_dstate->pmem_size;if (!QEMU_IS_ALIGNED (size,256 <<20 )) {@@ -292,6 +294,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, id->total_capacity = size / (256 <<20 ); id->persistent_capacity = size / (256 <<20 ); + id->lsa_size = cvc->get_lsa_size (ct3d); *len =sizeof (*id);return CXL_MBOX_SUCCESS ;hw/mem/cxl_type3.c +0
-5@@ -13,21 +13,6 @@#include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" -typedef struct cxl_type3_dev { - /* Private */ - PCIDevice parent_obj; - - /* Properties */
hw/mem/cxl_type3.c line 30
Quoted 10 lines
hw/mem/cxl_type3.c +0
-10-uint64_t size; -HostMemoryBackend *hostmem; - -/* State */ -CXLComponentState cxl_cstate; -CXLDeviceState cxl_dstate; -} CXLType3Dev; - -#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) -
If the structure had been in the header to start with it would be easier to see the changes added for this bit.
Quoted 73 lines
hw/mem/cxl_type3.c +9
-0static void build_dvsecs (CXLType3Dev *ct3d) {CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;@@ -186,10 +171,16 @@ static Property ct3_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static uint64_t get_lsa_size (CXLType3Dev *ct3d) +{ +return 0 ; +} +static void ct3_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +CXLType3Class *cvc =CXL_TYPE3_DEV_CLASS (oc); pc->realize = ct3_realize; pc->class_id =PCI_CLASS_STORAGE_EXPRESS ;@@ -201,11 +192,14 @@ static void ct3_class_init(ObjectClass *oc, void *data) dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;device_class_set_props (dc, ct3_props); + + cvc->get_lsa_size = get_lsa_size; }static const TypeInfo ct3d_info = { .name =TYPE_CXL_TYPE3_DEV , .parent =TYPE_PCI_DEVICE , + .class_size =sizeof (struct CXLType3Class ), .class_init = ct3_class_init, .instance_size =sizeof (CXLType3Dev), .instance_finalize = ct3_finalize,include/hw/cxl/cxl_device.h +29
-0@@ -230,4 +230,33 @@ REG64(CXL_MEM_DEV_STS, 0) FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; +HostMemoryBackend *lsa; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#ifndef TYPE_CXL_TYPE3_DEV +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#endif + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) +OBJECT_DECLARE_TYPE(CXLType3Device, CXLType3Class, CXL_TYPE3_DEV) + +struct CXLType3Class { +/* Private */ +PCIDeviceClass parent_class; + +/* public */ +uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); +}; +#endif
Signature · 1 line
Alex Bennée
Re: [PATCH v6 21/43] hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
Quoted 54 lines · On Wed, 02 Mar 2022 10:01:48 +0000 Alex Bennée <[email protected]> wrote:
Quoted 52 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This should introduce no change. Subsequent work will make use of this new class member.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 3 +++ hw/mem/cxl_type3.c | 24 +++++++++--------------- include/hw/cxl/cxl_device.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-)hw/cxl/cxl-mailbox-utils.c +3
-0@@ -278,6 +278,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, } __attribute__((packed)) *id;_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d);uint64_t size = cxl_dstate->pmem_size;if (!QEMU_IS_ALIGNED (size,256 <<20 )) {@@ -292,6 +294,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, id->total_capacity = size / (256 <<20 ); id->persistent_capacity = size / (256 <<20 ); + id->lsa_size = cvc->get_lsa_size (ct3d); *len =sizeof (*id);return CXL_MBOX_SUCCESS ;hw/mem/cxl_type3.c +0
-10@@ -13,21 +13,6 @@#include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" -typedef struct cxl_type3_dev { - /* Private */ - PCIDevice parent_obj; - - /* Properties */ - uint64_t size; - HostMemoryBackend *hostmem; - - /* State */ - CXLComponentState cxl_cstate;
Quoted 9 lines
hw/mem/cxl_type3.c line 30
Quoted 5 lines
hw/mem/cxl_type3.c +0
-5-CXLDeviceState cxl_dstate; -} CXLType3Dev; - -#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) -If the structure had been in the header to start with it would be easier to see the changes added for this bit.
Moved.. One other thing below.
Quoted 53 lines
Quoted 53 lines
hw/mem/cxl_type3.c +9
-0static void build_dvsecs (CXLType3Dev *ct3d) {CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;@@ -186,10 +171,16 @@ static Property ct3_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static uint64_t get_lsa_size (CXLType3Dev *ct3d) +{ +return 0 ; +} +static void ct3_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +CXLType3Class *cvc =CXL_TYPE3_DEV_CLASS (oc); pc->realize = ct3_realize; pc->class_id =PCI_CLASS_STORAGE_EXPRESS ;@@ -201,11 +192,14 @@ static void ct3_class_init(ObjectClass *oc, void *data) dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;device_class_set_props (dc, ct3_props); + + cvc->get_lsa_size = get_lsa_size; }static const TypeInfo ct3d_info = { .name =TYPE_CXL_TYPE3_DEV , .parent =TYPE_PCI_DEVICE , + .class_size =sizeof (struct CXLType3Class ), .class_init = ct3_class_init, .instance_size =sizeof (CXLType3Dev), .instance_finalize = ct3_finalize,include/hw/cxl/cxl_device.h +9
-0@@ -230,4 +230,33 @@ REG64(CXL_MEM_DEV_STS, 0) FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +typedef struct cxl_type3_dev { + /* Private */ + PCIDevice parent_obj; + + /* Properties */ + uint64_t size; + HostMemoryBackend *hostmem; + HostMemoryBackend *lsa; +
Quoted 8 lines
include/hw/cxl/cxl_device.h REG64(CXL_MEM_DEV_STS, 0) · line 249
Quoted 8 lines
include/hw/cxl/cxl_device.h +8
-0+/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +} CXLType3Dev; + +#ifndef TYPE_CXL_TYPE3_DEV +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#endif
I'm not sure why the ifndef is needed. Probably a legacy of some refactoring, so I've dropped that and the other definition of this in cxl.h
Quoted 15 lines
Quoted 13 lines
include/hw/cxl/cxl_device.h +12
-0+ +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) +OBJECT_DECLARE_TYPE(CXLType3Device, CXLType3Class, CXL_TYPE3_DEV) + +struct CXLType3Class { +/* Private */ +PCIDeviceClass parent_class; + +/* public */ +uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); +}; +#endif
PATCH v6 22/43 hw/cxl/device: Implement get/set Label Storage Area (LSA)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
--- hw/cxl/cxl-mailbox-utils.c | 57 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 56 +++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 5 ++++ 3 files changed, 117 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +57 -0
@@ -56,6 +56,8 @@ enum {#define MEMORY_DEVICE 0x0 CCLS = 0x41,#define GET_PARTITION_INFO 0x0 +#define GET_LSA 0x2 +#define SET_LSA 0x3 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -327,7 +329,59 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +static ret_code cmd_ccls_get_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (8 ))) *get_lsa; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint32_t offset, length; + + get_lsa = (void *)cmd->payload; + offset = get_lsa->offset; + length = get_lsa->length; + + *len =0 ; +if (offset + length > cvc->get_lsa_size (ct3d)) { +return CXL_MBOX_INVALID_INPUT ; + } + + *len = cvc->get_lsa (ct3d, get_lsa, length, offset); +return CXL_MBOX_SUCCESS ; +} + +static ret_code cmd_ccls_set_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t rsvd; + } __attribute__((packed,__aligned__ (8 ))) *set_lsa = (void *)cmd->payload; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint16_t plen = *len; + + *len =0 ; +if (!plen) { +return CXL_MBOX_SUCCESS ; + } + +if (set_lsa->offset + plen > cvc->get_lsa_size (ct3d) +sizeof (*set_lsa)) { +return CXL_MBOX_INVALID_INPUT ; + } + + cvc->set_lsa (ct3d, (void *)set_lsa +sizeof (*set_lsa), + plen -sizeof (*set_lsa), set_lsa->offset); +return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 2)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -350,6 +404,9 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_identify_memory_device, 0, 0 }, [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", cmd_ccls_get_partition_info, 0, 0 }, + [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 }, + [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, + ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)
hw/mem/cxl_type3.c +55 -1
85 lines not on this page · raw message
include/hw/cxl/cxl_device.h +5 -0
@@ -257,6 +257,11 @@ struct CXLType3Class {/* public */ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); + +uint64_t (*get_lsa)(CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset); +void (*set_lsa)(CXLType3Dev *ct3d,const void *buf,uint64_t size, + uint64_t offset); };#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 22/43] hw/cxl/device: Implement get/set Label Storage Area (LSA)
Quoted 39 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 57 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 56 +++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 5 ++++ 3 files changed, 117 insertions(+), 1 deletion(-)hw/cxl/cxl-mailbox-utils.c +9
-0@@ -56,6 +56,8 @@ enum {#define MEMORY_DEVICE 0x0 CCLS = 0x41,#define GET_PARTITION_INFO 0x0 +#define GET_LSA 0x2 +#define SET_LSA 0x3 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -327,7 +329,59 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,return CXL_MBOX_SUCCESS ; } +static ret_code cmd_ccls_get_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t length;
hw/cxl/cxl-mailbox-utils.c static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, · line 348
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+ } __attribute__((packed, __aligned__(8))) *get_lsa; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint32_t offset, length; + + get_lsa = (void *)cmd->payload; + offset = get_lsa->offset; + length = get_lsa->length; + + *len =0 ;
This can drop into the failure leg.
Quoted 21 lines
hw/cxl/cxl-mailbox-utils.c +21
-0+if (offset + length > cvc->get_lsa_size (ct3d)) { +return CXL_MBOX_INVALID_INPUT ; + } + + *len = cvc->get_lsa (ct3d, get_lsa, length, offset); +return CXL_MBOX_SUCCESS ; +} + +static ret_code cmd_ccls_set_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t rsvd; + } __attribute__((packed,__aligned__ (8 ))) *set_lsa = (void *)cmd->payload; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint16_t plen = *len; + + *len =0 ;
hw/cxl/cxl-mailbox-utils.c static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, · line 379
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+if (!plen) { +return CXL_MBOX_SUCCESS ; + } + +if (set_lsa->offset + plen > cvc->get_lsa_size (ct3d) +sizeof (*set_lsa)) { +return CXL_MBOX_INVALID_INPUT ; + } + + cvc->set_lsa (ct3d, (void *)set_lsa +sizeof (*set_lsa), + plen -sizeof (*set_lsa), set_lsa->offset);
All these sizeof's make me wonder if these structures should be public with a #define? Failing that having
const size_t lsa_len = sizeof(*set_lsa);
and use that might make the flow a bit easier to scan. That said why isn't set_lsa taking a type of *set_lsa instead of having void casts?
Quoted 127 lines
hw/cxl/cxl-mailbox-utils.c +7
-0+return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 2)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)@@ -350,6 +404,9 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_identify_memory_device, 0, 0 }, [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", cmd_ccls_get_partition_info, 0, 0 }, + [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 }, + [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, + ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)hw/mem/cxl_type3.c +55
-1@@ -8,6 +8,7 @@#include "qapi/error.h" #include "qemu/log.h" #include "qemu/module.h" +#include "qemu/pmem.h" #include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" @@ -114,6 +115,11 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)memory_region_set_enabled (mr, true);host_memory_backend_set_mapped (ct3d->hostmem, true); ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size; + +if (!ct3d->lsa) { +error_setg (errp,"lsa property must be set" ); +return ; + } }@@ -168,12 +174,58 @@ static Property ct3_props[] = {DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ),DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , HostMemoryBackend *), +DEFINE_PROP_LINK ("lsa" , CXLType3Dev, lsa,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), };static uint64_t get_lsa_size (CXLType3Dev *ct3d) { -return 0 ; +MemoryRegion *mr; + + mr =host_memory_backend_get_memory (ct3d->lsa); +return memory_region_size (mr); +} + +static void validate_lsa_access (MemoryRegion *mr,uint64_t size, +uint64_t offset) +{ +assert (offset + size <=memory_region_size (mr)); +assert (offset + size > offset); +} + +static uint64_t get_lsa (CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (buf, lsa, size); + +return size; +} + +static void set_lsa (CXLType3Dev *ct3d,const void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (lsa, buf, size); +memory_region_set_dirty (mr, offset, size); + +/* +* Just like the PMEM, if the guest is not allowed to exit gracefully, label +* updates will get lost. +*/ }static void ct3_class_init (ObjectClass *oc,void *data)@@ -194,6 +246,8 @@ static void ct3_class_init(ObjectClass *oc, void *data)device_class_set_props (dc, ct3_props); cvc->get_lsa_size = get_lsa_size; + cvc->get_lsa = get_lsa; + cvc->set_lsa = set_lsa; }static const TypeInfo ct3d_info = {include/hw/cxl/cxl_device.h +5
-0@@ -257,6 +257,11 @@ struct CXLType3Class {/* public */ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); + +uint64_t (*get_lsa)(CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset); +void (*set_lsa)(CXLType3Dev *ct3d,const void *buf,uint64_t size, + uint64_t offset); };#endif
Signature · 1 line
Alex Bennée
Re: [PATCH v6 22/43] hw/cxl/device: Implement get/set Label Storage Area (LSA)
Quoted 43 lines · On Wed, 02 Mar 2022 10:03:34 +0000 Alex Bennée <[email protected]> wrote:
Quoted 26 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>---
hw/cxl/cxl-mailbox-utils.c +14
-0+ +static ret_code cmd_ccls_set_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t rsvd; + } __attribute__((packed,__aligned__ (8 ))) *set_lsa = (void *)cmd->payload; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint16_t plen = *len; + + *len =0 ;hw/cxl/cxl-mailbox-utils.c static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, · line 379
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+if (!plen) { +return CXL_MBOX_SUCCESS ; + } + +if (set_lsa->offset + plen > cvc->get_lsa_size (ct3d) +sizeof (*set_lsa)) { +return CXL_MBOX_INVALID_INPUT ; + } + + cvc->set_lsa (ct3d, (void *)set_lsa +sizeof (*set_lsa), + plen -sizeof (*set_lsa), set_lsa->offset);All these sizeof's make me wonder if these structures should be public with a #define?
Failing that having
Quoted 1 line
const size_t lsa_len = sizeof(*set_lsa);
The naming of such a variable is a bit complex and I think perhaps we are better off using a 0 length data[] element after the header and then offsetof() to compute the header length.
So (with the addition of some renames that hopefully make things clearer) something like (on top of changes as a result of earlier review comments):
@@ -357,12 +357,15 @@ static ret_code cmd_ccls_set_lsa(struct cxl_cmd *cmd,
CXLDeviceState *cxl_dstate,
uint16_t *len)
{
- struct {
+ struct set_lsa_pl {
uint32_t offset;
uint32_t rsvd;
- } QEMU_PACKED QEMU_ALIGNED(8) *set_lsa = (void *)cmd->payload;
+ uint8_t data[];
+ } QEMU_PACKED;
+ struct set_lsa_pl *set_lsa_payload = (void *)cmd->payload;
CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate);
CXLType3Class *cvc = CXL_TYPE3_DEV_GET_CLASS(ct3d);
+ const size_t hdr_len = offsetof(struct set_lsa_pl, data);
uint16_t plen = *len;
*len = 0;
@@ -370,12 +373,12 @@ static ret_code cmd_ccls_set_lsa(struct cxl_cmd *cmd,
return CXL_MBOX_SUCCESS;
}- if (set_lsa->offset + plen > cvc->get_lsa_size(ct3d) + sizeof(*set_lsa)) {
+ if (set_lsa_payload->offset + plen > cvc->get_lsa_size(ct3d) + hdr_len) {
return CXL_MBOX_INVALID_INPUT;
}+ plen -= hdr_len;
- cvc->set_lsa(ct3d, (void *)set_lsa + sizeof(*set_lsa),
- plen - sizeof(*set_lsa), set_lsa->offset);
+ cvc->set_lsa(ct3d, set_lsa_payload->data, plen, set_lsa_payload->offset);
return CXL_MBOX_SUCCESS;}
the aligned marker doesn't make much sense here as we are using cmd->payload directly so the compiler doesn't get to mess with the alignment anyway. I've aligned markings from similar places in this patch and others.
Quoted 3 lines
and use that might make the flow a bit easier to scan. That said why isn't
set_lsataking a type of*set_lsainstead of having void casts?
We could do that, but I think the intent here is to separate the particular formatting of the set_lsa command from what it is doing which is basically a memcpy so needs an src, size and dst offset.
It's a bit messy that the size to be written is implicit so we have to extract it from the size of the command message accounting for the header.
Hopefully the above changes make that easier to follow?
Thanks,
Jonathan
Quoted 130 lines
Quoted 128 lines
hw/cxl/cxl-mailbox-utils.c +7
-0+return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 2)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)@@ -350,6 +404,9 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_identify_memory_device, 0, 0 }, [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", cmd_ccls_get_partition_info, 0, 0 }, + [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 }, + [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, + ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)hw/mem/cxl_type3.c +55
-1@@ -8,6 +8,7 @@#include "qapi/error.h" #include "qemu/log.h" #include "qemu/module.h" +#include "qemu/pmem.h" #include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" @@ -114,6 +115,11 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)memory_region_set_enabled (mr, true);host_memory_backend_set_mapped (ct3d->hostmem, true); ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size; + +if (!ct3d->lsa) { +error_setg (errp,"lsa property must be set" ); +return ; + } }@@ -168,12 +174,58 @@ static Property ct3_props[] = {DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ),DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , HostMemoryBackend *), +DEFINE_PROP_LINK ("lsa" , CXLType3Dev, lsa,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), };static uint64_t get_lsa_size (CXLType3Dev *ct3d) { -return 0 ; +MemoryRegion *mr; + + mr =host_memory_backend_get_memory (ct3d->lsa); +return memory_region_size (mr); +} + +static void validate_lsa_access (MemoryRegion *mr,uint64_t size, +uint64_t offset) +{ +assert (offset + size <=memory_region_size (mr)); +assert (offset + size > offset); +} + +static uint64_t get_lsa (CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (buf, lsa, size); + +return size; +} + +static void set_lsa (CXLType3Dev *ct3d,const void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (lsa, buf, size); +memory_region_set_dirty (mr, offset, size); + +/* +* Just like the PMEM, if the guest is not allowed to exit gracefully, label +* updates will get lost. +*/ }static void ct3_class_init (ObjectClass *oc,void *data)@@ -194,6 +246,8 @@ static void ct3_class_init(ObjectClass *oc, void *data)device_class_set_props (dc, ct3_props); cvc->get_lsa_size = get_lsa_size; + cvc->get_lsa = get_lsa; + cvc->set_lsa = set_lsa; }static const TypeInfo ct3d_info = {include/hw/cxl/cxl_device.h +5
-0@@ -257,6 +257,11 @@ struct CXLType3Class {/* public */ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); + +uint64_t (*get_lsa)(CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset); +void (*set_lsa)(CXLType3Dev *ct3d,const void *buf,uint64_t size, + uint64_t offset); };#endif
Re: [PATCH v6 22/43] hw/cxl/device: Implement get/set Label Storage Area (LSA)
Quoted 89 lines · On Fri, 4 Mar 2022 14:16:18 +0000 Jonathan Cameron…
Quoted 43 lines · On Wed, 02 Mar 2022 10:03:34 +0000 Alex Bennée <[email protected]> wrote:
Quoted 26 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>---
hw/cxl/cxl-mailbox-utils.c +14
-0+ +static ret_code cmd_ccls_set_lsa (struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate, +uint16_t *len) +{ +struct { +uint32_t offset; +uint32_t rsvd; + } __attribute__((packed,__aligned__ (8 ))) *set_lsa = (void *)cmd->payload; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint16_t plen = *len; + + *len =0 ;hw/cxl/cxl-mailbox-utils.c static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, · line 379
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+if (!plen) { +return CXL_MBOX_SUCCESS ; + } + +if (set_lsa->offset + plen > cvc->get_lsa_size (ct3d) +sizeof (*set_lsa)) { +return CXL_MBOX_INVALID_INPUT ; + } + + cvc->set_lsa (ct3d, (void *)set_lsa +sizeof (*set_lsa), + plen -sizeof (*set_lsa), set_lsa->offset);All these sizeof's make me wonder if these structures should be public with a #define?
Failing that havingQuoted 1 line
const size_t lsa_len = sizeof(*set_lsa);The naming of such a variable is a bit complex and I think perhaps we are better off using a 0 length data[] element after the header and then
offsetof()to compute the header length.So (with the addition of some renames that hopefully make things clearer) something like (on top of changes as a result of earlier review comments):
(quoted diff) +8
-5@@ -357,12 +357,15 @@ static ret_code cmd_ccls_set_lsa(struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate, uint16_t *len) { - struct { + struct set_lsa_pl { uint32_t offset; uint32_t rsvd; - } QEMU_PACKED QEMU_ALIGNED(8) *set_lsa = (void *)cmd->payload; + uint8_t data[]; + } QEMU_PACKED; + struct set_lsa_pl *set_lsa_payload = (void *)cmd->payload; CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); CXLType3Class *cvc = CXL_TYPE3_DEV_GET_CLASS(ct3d); + const size_t hdr_len = offsetof(struct set_lsa_pl, data); uint16_t plen = *len; *len = 0;@@ -370,12 +373,12 @@ static ret_code cmd_ccls_set_lsa(struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } - if (set_lsa->offset + plen > cvc->get_lsa_size(ct3d) + sizeof(*set_lsa)) { + if (set_lsa_payload->offset + plen > cvc->get_lsa_size(ct3d) + hdr_len) { return CXL_MBOX_INVALID_INPUT; } + plen -= hdr_len; - cvc->set_lsa(ct3d, (void *)set_lsa + sizeof(*set_lsa), - plen - sizeof(*set_lsa), set_lsa->offset); + cvc->set_lsa(ct3d, set_lsa_payload->data, plen, set_lsa_payload->offset); return CXL_MBOX_SUCCESS; }
Quoted 3 lines
the aligned marker doesn't make much sense here as we are using
cmd->payloaddirectly so the compiler doesn't get to mess with the alignment anyway. I've aligned markings from similar places in this patch and others.
Turned out I was wrong on that - it's necessary in some places (not here) to avoid the compiler thinking we might be taking addresses of unaligned elements of a structure when we aren't.
This one is safe to remove however.
Quoted 151 lines
Quoted 4 lines
and use that might make the flow a bit easier to scan. That said why isn't
set_lsataking a type of*set_lsainstead of having void casts?We could do that, but I think the intent here is to separate the particular formatting of the
set_lsacommand from what it is doing which is basically a memcpy so needs an src, size and dst offset.It's a bit messy that the size to be written is implicit so we have to extract it from the size of the command message accounting for the header.
Hopefully the above changes make that easier to follow?
Thanks,
Jonathan
Quoted 130 lines
Quoted 128 lines
hw/cxl/cxl-mailbox-utils.c +7
-0+return CXL_MBOX_SUCCESS ; +} +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 2)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)@@ -350,6 +404,9 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { cmd_identify_memory_device, 0, 0 }, [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", cmd_ccls_get_partition_info, 0, 0 }, + [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 }, + [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, + ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, };void cxl_process_mailbox (CXLDeviceState *cxl_dstate)hw/mem/cxl_type3.c +55
-1@@ -8,6 +8,7 @@#include "qapi/error.h" #include "qemu/log.h" #include "qemu/module.h" +#include "qemu/pmem.h" #include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" @@ -114,6 +115,11 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)memory_region_set_enabled (mr, true);host_memory_backend_set_mapped (ct3d->hostmem, true); ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size; + +if (!ct3d->lsa) { +error_setg (errp,"lsa property must be set" ); +return ; + } }@@ -168,12 +174,58 @@ static Property ct3_props[] = {DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ),DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , HostMemoryBackend *), +DEFINE_PROP_LINK ("lsa" , CXLType3Dev, lsa,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), };static uint64_t get_lsa_size (CXLType3Dev *ct3d) { -return 0 ; +MemoryRegion *mr; + + mr =host_memory_backend_get_memory (ct3d->lsa); +return memory_region_size (mr); +} + +static void validate_lsa_access (MemoryRegion *mr,uint64_t size, +uint64_t offset) +{ +assert (offset + size <=memory_region_size (mr)); +assert (offset + size > offset); +} + +static uint64_t get_lsa (CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (buf, lsa, size); + +return size; +} + +static void set_lsa (CXLType3Dev *ct3d,const void *buf,uint64_t size, +uint64_t offset) +{ +MemoryRegion *mr; +void *lsa; + + mr =host_memory_backend_get_memory (ct3d->lsa); +validate_lsa_access (mr, size, offset); + + lsa =memory_region_get_ram_ptr (mr) + offset; +memcpy (lsa, buf, size); +memory_region_set_dirty (mr, offset, size); + +/* +* Just like the PMEM, if the guest is not allowed to exit gracefully, label +* updates will get lost. +*/ }static void ct3_class_init (ObjectClass *oc,void *data)@@ -194,6 +246,8 @@ static void ct3_class_init(ObjectClass *oc, void *data)device_class_set_props (dc, ct3_props); cvc->get_lsa_size = get_lsa_size; + cvc->get_lsa = get_lsa; + cvc->set_lsa = set_lsa; }static const TypeInfo ct3d_info = {include/hw/cxl/cxl_device.h +5
-0@@ -257,6 +257,11 @@ struct CXLType3Class {/* public */ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); + +uint64_t (*get_lsa)(CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset); +void (*set_lsa)(CXLType3Dev *ct3d,const void *buf,uint64_t size, + uint64_t offset); };#endif
PATCH v6 23/43 hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL host bridges themselves may have MMIO. Since host bridges don't have a BAR they are treated as special for MMIO. This patch includes i386/pc support.
Also hook up the device reset now that we have have the MMIO space in which the results are visible.
Note that we duplicate the PCI express case for the aml_build but the implementations will diverge when the CXL specific _OSC is introduced.
--- hw/i386/acpi-build.c | 25 +++++++++++- hw/i386/pc.c | 27 ++++++++++++- hw/pci-bridge/pci_expander_bridge.c | 62 ++++++++++++++++++++++++++++- include/hw/cxl/cxl.h | 3 ++ 4 files changed, 113 insertions(+), 4 deletions(-)
hw/i386/acpi-build.c +23 -2
@@ -28,6 +28,7 @@#include "qemu/bitmap.h" #include "qemu/error-report.h" #include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" #include "hw/core/cpu.h" #include "target/i386/cpu.h" #include "hw/misc/pvpanic.h"
@@ -1564,10 +1565,21 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, } scope =aml_scope ("\\_SB" ); - dev =aml_device ("PC%.02X" , bus_num); + +if (pci_bus_is_cxl (bus)) { + dev =aml_device ("CL%.02X" , bus_num); + }else { + dev =aml_device ("PC%.02X" , bus_num); + }aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -if (pci_bus_is_express (bus)) { +if (pci_bus_is_cxl (bus)) { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); + +/* Expander bridges do not have ACPI PCI Hot-plug enabled */ +aml_append (dev,build_q35_osc_method (true)); + }else if (pci_bus_is_express (bus)) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));
@@ -1587,6 +1599,15 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (dev,aml_name_decl ("_CRS" , crs));aml_append (scope, dev);aml_append (dsdt, scope); + +/* Handle the ranges for the PXB expanders */ +if (pci_bus_is_cxl (bus)) { +MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +uint64_t base = mr->addr; + +crs_range_insert (crs_range_set.mem_ranges, base, + base +memory_region_size (mr) -1 ); + } } }
hw/i386/pc.c +26 -1
@@ -75,6 +75,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/qapi-visit-machine.h"
@@ -815,6 +816,7 @@ void pc_memory_init(PCMachineState *pcms,MachineClass *mc =MACHINE_GET_CLASS (machine);PCMachineClass *pcmc =PC_MACHINE_GET_CLASS (pcms);X86MachineState *x86ms =X86_MACHINE (pcms); +hwaddr cxl_base;assert (machine->ram_size == x86ms->below_4g_mem_size + x86ms->above_4g_mem_size);
@@ -904,6 +906,26 @@ void pc_memory_init(PCMachineState *pcms, &machine->device_memory->mr); } +if (machine->cxl_devices_state->is_enabled) { +MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +hwaddr cxl_size = MiB; + +if (pcmc->has_reserved_memory && machine->device_memory->base) { + cxl_base = machine->device_memory->base; +if (!pcmc->broken_reserved_end) { + cxl_base +=memory_region_size (&machine->device_memory->mr); + } + }else if (pcms->sgx_epc.size !=0 ) { + cxl_base =sgx_epc_above_4g_end (&pcms->sgx_epc); + }else { + cxl_base =0x100000000ULL + x86ms->above_4g_mem_size; + } + +e820_add_entry (cxl_base, cxl_size,E820_RESERVED ); +memory_region_init (mr,OBJECT (machine),"cxl_host_reg" , cxl_size); +memory_region_add_subregion (system_memory, cxl_base, mr); + } +/* Initialize PC system firmware */ pc_system_firmware_init (pcms, rom_memory);
@@ -964,7 +986,10 @@ uint64_t pc_pci_hole64_start(void)X86MachineState *x86ms =X86_MACHINE (pcms);uint64_t hole64_start =0 ; -if (pcmc->has_reserved_memory && ms->device_memory->base) { +if (ms->cxl_devices_state->host_mr.addr) { + hole64_start = ms->cxl_devices_state->host_mr.addr + +memory_region_size (&ms->cxl_devices_state->host_mr); + }else if (pcmc->has_reserved_memory && ms->device_memory->base) { hole64_start = ms->device_memory->base;if (!pcmc->broken_reserved_end) { hole64_start +=memory_region_size (&ms->device_memory->mr);
hw/pci-bridge/pci_expander_bridge.c +61 -1
103 lines not on this page · raw message
include/hw/cxl/cxl.h +3 -0
@@ -18,9 +18,12 @@#define CXL_DEVICE_REG_BAR_IDX 2#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_WINDOW_MAX 10typedef struct CXLState {bool is_enabled; +MemoryRegion host_mr; +unsigned int next_mr_idx; }CXLState ;#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 23/43] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
Quoted 8 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
CXL host bridges themselves may have MMIO. Since host bridges don't have a BAR they are treated as special for MMIO. This patch includes i386/pc support.
Also hook up the device reset now that we have have the MMIO space in which the results are visible.
Quoted 7 lines
Note that we duplicate the PCI express case for the
aml_buildbut the implementations will diverge when the CXL specific_OSCis introduced.Signed-off-by Ben Widawsky <[email protected]>Co-developed-by Jonathan Cameron <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Looks good to me although I'll defer to the i386 maintainers if they think otherwise.
Signature · 1 line
Alex Bennée
PATCH v6 24/43 acpi/cxl: Add _OSC implementation (9.14.2)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL 2.0 specification adds 2 new dwords to the existing _OSC definition from PCIe. The new dwords are accessed with a new uuid. This implementation supports what is in the specification.
--- hw/acpi/Kconfig | 5 ++ hw/acpi/cxl-stub.c | 12 +++++ hw/acpi/cxl.c | 104 ++++++++++++++++++++++++++++++++++++++++++ hw/acpi/meson.build | 4 +- hw/i386/acpi-build.c | 15 ++++-- include/hw/acpi/cxl.h | 23 ++++++++++ 6 files changed, 157 insertions(+), 6 deletions(-)
hw/acpi/Kconfig +5 -0
@@ -5,6 +5,7 @@ config ACPI_X86 bool select ACPI select ACPI_NVDIMM + select ACPI_CXL select ACPI_CPU_HOTPLUG select ACPI_MEMORY_HOTPLUG select ACPI_HMAT
@@ -66,3 +67,7 @@ config ACPI_ERST bool default y depends on ACPI && PCI + +config ACPI_CXL + bool + depends on ACPI
hw/acpi/cxl-stub.c new file +12 -0
@@ -0,0 +1,12 @@ + +/* +* Stubs for ACPI platforms that don't support CXl +*/ +#include "qemu/osdep.h" +#include "hw/acpi/aml-build.h" +#include "hw/acpi/cxl.h" + +void build_cxl_osc_method (Aml *dev) +{ +g_assert_not_reached (); +}
hw/acpi/cxl.c new file +104 -0
104 lines not on this page · raw message
hw/acpi/meson.build +3 -1
@@ -13,6 +13,7 @@ acpi_ss.add(when: 'CONFIG_ACPI_MEMORY_HOTPLUG', if_false: files('acpi-mem-hotplu acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_true: files('nvdimm.c')) acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_false: files('acpi-nvdimm-stub.c')) acpi_ss.add(when: 'CONFIG_ACPI_PCI', if_true: files('pci.c')) +acpi_ss.add(when: 'CONFIG_ACPI_CXL', if_true: files('cxl.c'), if_false: files('cxl-stub.c')) acpi_ss.add(when: 'CONFIG_ACPI_VMGENID', if_true: files('vmgenid.c')) acpi_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device.c')) acpi_ss.add(when: 'CONFIG_ACPI_HMAT', if_true: files('hmat.c'))
@@ -31,4 +32,5 @@ softmmu_ss.add_all(when: 'CONFIG_ACPI', if_true: acpi_ss) softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('acpi-stub.c', 'aml-build-stub.c', 'acpi-x86-stub.c', 'ipmi-stub.c', 'ghes-stub.c', 'acpi-mem-hotplug-stub.c', 'acpi-cpu-hotplug-stub.c', - 'acpi-pci-hotplug-stub.c', 'acpi-nvdimm-stub.c')) + 'acpi-pci-hotplug-stub.c', 'acpi-nvdimm-stub.c', + 'cxl-stub.c'))
hw/i386/acpi-build.c +10 -5
@@ -66,6 +66,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/utils.h" #include "hw/acpi/pci.h" +#include "hw/acpi/cxl.h" #include "qom/qom-qobject.h" #include "hw/i386/amd_iommu.h"
@@ -1574,11 +1575,15 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));if (pci_bus_is_cxl (bus)) { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); - -/* Expander bridges do not have ACPI PCI Hot-plug enabled */ -aml_append (dev,build_q35_osc_method (true)); +struct Aml *pkg =aml_package (2 ); + +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num))); +build_cxl_osc_method (dev); }else if (pci_bus_is_express (bus)) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));
include/hw/acpi/cxl.h new file +23 -0
@@ -0,0 +1,23 @@ +/* +* Copyright (C) 2020 Intel Corporation +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. + +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. + +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef HW_ACPI_CXL_H +#define HW_ACPI_CXL_H + +void build_cxl_osc_method (Aml *dev); + +#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 24/43] acpi/cxl: Add _OSC implementation (9.14.2)
Quoted 8 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
CXL 2.0 specification adds 2 new dwords to the existing
_OSCdefinition from PCIe. The new dwords are accessed with a new uuid. This implementation supports what is in the specification.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
From my limited understanding of the ACPI AML DSL we use:
Signature · 1 line
Alex Bennée
PATCH v6 25/43 acpi/cxl: Create the CEDT (9.14.1)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The CXL Early Discovery Table is defined in the CXL 2.0 specification as a way for the OS to get CXL specific information from the system firmware.
CXL 2.0 specification adds an _HID, ACPI0016, for CXL capable host bridges, with a _CID of PNP0A08 (PCIe host bridge). CXL aware software is able to use this initiate the proper _OSC method, and get the _UID which is referenced by the CEDT. Therefore the existence of an ACPI0016 device allows a CXL aware driver perform the necessary actions. For a CXL capable OS, this works. For a CXL unaware OS, this works.
CEDT awaremess requires more. The motivation for ACPI0017 is to provide the possibility of having a Linux CXL module that can work on a legacy Linux kernel. Linux core PCI/ACPI which won't be built as a module, will see the _CID of PNP0A08 and bind a driver to it. If we later loaded a driver for ACPI0016, Linux won't be able to bind it to the hardware because it has already bound the PNP0A08 driver. The ACPI0017 device is an opportunity to have an object to bind a driver will be used by a Linux driver to walk the CXL topology and do everything that we would have preferred to do with ACPI0016.
There is another motivation for an ACPI0017 device which isn't implemented here. An operating system needs an attach point for a non-volatile region provider that understands cross-hostbridge interleaving. Since QEMU emulation doesn't support interleaving yet, this is more important on the OS side, for now.
As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge Structure (CHBS) which is primarily useful for telling the OS exactly where the MMIO for the host bridge is.
--- hw/acpi/cxl.c | 68 +++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 27 ++++++++++++ hw/pci-bridge/pci_expander_bridge.c | 17 -------- include/hw/acpi/cxl.h | 5 +++ include/hw/pci/pci_bridge.h | 20 +++++++++ 5 files changed, 120 insertions(+), 17 deletions(-)
hw/acpi/cxl.c +68 -0
@@ -18,7 +18,11 @@*/ #include "qemu/osdep.h" +#include "hw/sysbus.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" +#include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h" #include "hw/acpi/bios-linker-loader.h"
@@ -26,6 +30,70 @@#include "qapi/error.h" #include "qemu/uuid.h" +static void cedt_build_chbs (GArray *table_data,PXBDev *cxl) +{ +SysBusDevice *sbd =SYS_BUS_DEVICE (cxl->cxl.cxl_host_bridge); +struct MemoryRegion *mr = sbd->mmio[0 ].memory; + +/* Type */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Record Length */ +build_append_int_noprefix (table_data,32 ,2 ); + +/* UID - currently equal to bus number */ +build_append_int_noprefix (table_data, cxl->bus_nr,4 ); + +/* Version */ +build_append_int_noprefix (table_data,1 ,4 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,4 ); + +/* Base - subregion within a container that is in PA space */ +build_append_int_noprefix (table_data, mr->container->addr + mr->addr,8 ); + +/* Length */ +build_append_int_noprefix (table_data,memory_region_size (mr),8 ); +} + +static int cxl_foreach_pxb_hb (Object *obj,void *opaque) +{ +Aml *cedt = opaque; + +if (object_dynamic_cast (obj,TYPE_PXB_CXL_DEVICE )) { +cedt_build_chbs (cedt->buf,PXB_CXL_DEV (obj)); + } + +return 0 ; +} + +void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +BIOSLinker *linker,const char *oem_id, +const char *oem_table_id) +{ +Aml *cedt; +AcpiTable table = { .sig ="CEDT" , .rev =1 , .oem_id = oem_id, + .oem_table_id = oem_table_id }; + +acpi_add_table (table_offsets, table_data); +acpi_table_begin (&table, table_data); + cedt =init_aml_allocator (); + +/* reserve space for CEDT header */ + +object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); + +/* copy AML table into ACPI tables blob and patch header there */ +g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len); +free_aml_allocator (); + +acpi_table_end (linker, &table); +} +static Aml *__build_cxl_osc_method (void ) {Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
hw/i386/acpi-build.c +27 -0
@@ -76,6 +76,7 @@#include "hw/acpi/ipmi.h" #include "hw/acpi/hmat.h" #include "hw/acpi/viot.h" +#include "hw/acpi/cxl.h" #include CONFIG_DEVICES
@@ -1403,6 +1404,22 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } +static void build_acpi0017 (Aml *table) +{ +Aml *dev, *scope, *method; + + scope =aml_scope ("_SB" ); + dev =aml_device ("CXLM" ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + + method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED ); +aml_append (method,aml_return (aml_int (0x01 ))); +aml_append (dev, method); + +aml_append (scope, dev); +aml_append (table, scope); +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1422,6 +1439,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,#ifdef CONFIG_TPM TPMIf *tpm =tpm_find ();#endif +bool cxl_present = false;int i;VMBusBridge *vmbus_bridge =vmbus_bridge_find ();AcpiTable table = { .sig ="DSDT" , .rev =1 , .oem_id = x86ms->oem_id,
@@ -1610,12 +1628,17 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,MemoryRegion *mr = &machine->cxl_devices_state->host_mr;uint64_t base = mr->addr; + cxl_present = true;crs_range_insert (crs_range_set.mem_ranges, base, base +memory_region_size (mr) -1 ); } } } +if (cxl_present) { +build_acpi0017 (dsdt); + } +/* * At this point crs_range_set has all the ranges used by pci * busses *other* than PCI0. These ranges will be excluded from
@@ -2680,6 +2703,10 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine) machine->nvdimms_state, machine->ram_slots, x86ms->oem_id, x86ms->oem_table_id); } +if (machine->cxl_devices_state->is_enabled) { +cxl_build_cedt (machine, table_offsets, tables_blob, tables->linker, + x86ms->oem_id, x86ms->oem_table_id); + }acpi_add_table (table_offsets, tables_blob);build_waet (tables_blob, tables->linker, x86ms->oem_id, x86ms->oem_table_id);
hw/pci-bridge/pci_expander_bridge.c +0 -17
@@ -57,29 +57,12 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) -#define TYPE_PXB_CXL_DEVICE "pxb-cxl" -DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , - TYPE_PXB_CXL_DEVICE) -typedef struct CXLHost {PCIHostState parent_obj;CXLComponentState cxl_cstate; }CXLHost ; -struct PXBDev { -/*< private >*/ -PCIDevice parent_obj; -/*< public >*/ - -uint8_t bus_nr; -uint16_t numa_node; -bool bypass_iommu; -struct cxl_dev { -CXLHost *cxl_host_bridge; - } cxl; -}; -static PXBDev *convert_to_pxb (PCIDevice *dev) {/* A CXL PXB's parent bus is PCIe, so the normal check won't work */
include/hw/acpi/cxl.h +5 -0
@@ -18,6 +18,11 @@#ifndef HW_ACPI_CXL_H #define HW_ACPI_CXL_H +#include "hw/acpi/bios-linker-loader.h" + +void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +BIOSLinker *linker,const char *oem_id, +const char *oem_table_id);void build_cxl_osc_method (Aml *dev);#endif
include/hw/pci/pci_bridge.h +20 -0
@@ -28,6 +28,7 @@#include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" +#include "hw/cxl/cxl.h" #include "qom/object.h" typedef struct PCIBridgeWindows PCIBridgeWindows ;
@@ -80,6 +81,25 @@ struct PCIBridge {#define PCI_BRIDGE_DEV_PROP_CHASSIS_NR "chassis_nr"#define PCI_BRIDGE_DEV_PROP_MSI "msi"#define PCI_BRIDGE_DEV_PROP_SHPC "shpc" +typedef struct CXLHost CXLHost ; + +struct PXBDev { +/*< private >*/ +PCIDevice parent_obj; +/*< public >*/ + +uint8_t bus_nr; +uint16_t numa_node; +bool bypass_iommu; +struct cxl_dev { +CXLHost *cxl_host_bridge;/* Pointer to a CXLHost */ + } cxl; +}; + +typedef struct PXBDev PXBDev ; +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) int pci_bridge_ssvid_init(PCIDevice *dev, uint8_t offset, uint16_t svid, uint16_t ssid,
Signature · 1 line
2.32.0
Re: [PATCH v6 25/43] acpi/cxl: Create the CEDT (9.14.1)
Quoted 29 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
The CXL Early Discovery Table is defined in the CXL 2.0 specification as a way for the OS to get CXL specific information from the system firmware.
CXL 2.0 specification adds an
_HID, ACPI0016, for CXL capable host bridges, with a_CIDof PNP0A08 (PCIe host bridge). CXL aware software is able to use this initiate the proper_OSCmethod, and get the_UIDwhich is referenced by the CEDT. Therefore the existence of an ACPI0016 device allows a CXL aware driver perform the necessary actions. For a CXL capable OS, this works. For a CXL unaware OS, this works.CEDT awaremess requires more. The motivation for ACPI0017 is to provide the possibility of having a Linux CXL module that can work on a legacy Linux kernel. Linux core PCI/ACPI which won't be built as a module, will see the
_CIDof PNP0A08 and bind a driver to it. If we later loaded a driver for ACPI0016, Linux won't be able to bind it to the hardware because it has already bound the PNP0A08 driver. The ACPI0017 device is an opportunity to have an object to bind a driver will be used by a Linux driver to walk the CXL topology and do everything that we would have preferred to do with ACPI0016.There is another motivation for an ACPI0017 device which isn't implemented here. An operating system needs an attach point for a non-volatile region provider that understands cross-hostbridge interleaving. Since QEMU emulation doesn't support interleaving yet, this is more important on the OS side, for now.
Quoted 7 lines
As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge Structure (CHBS) which is primarily useful for telling the OS exactly where the MMIO for the host bridge is.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 26/43 hw/cxl/component: Add utils for interleave parameter encoding/decoding
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Both registers and the CFMWS entries in CDAT use simple encodings for the number of interleave ways and the interleave granularity. Introduce simple conversion functions to/from the unencoded number / size. So far the iw decode has not been needed so is it not implemented.
--- hw/cxl/cxl-component-utils.c | 34 ++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 8 ++++++++ 2 files changed, 42 insertions(+)
hw/cxl/cxl-component-utils.c +34 -0
@@ -9,6 +9,7 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qapi/error.h" #include "hw/pci/pci.h" #include "hw/cxl/cxl.h"
@@ -217,3 +218,36 @@ void cxl_component_create_dvsec(CXLComponentState *cxl, uint16_t length,range_init_nofail (&cxl->dvsecs[type], cxl->dvsec_offset, length); cxl->dvsec_offset += length; } + +uint8_t cxl_interleave_ways_enc (int iw,Error **errp) +{ +switch (iw) { +case 1 :return 0x0 ; +case 2 :return 0x1 ; +case 4 :return 0x2 ; +case 8 :return 0x3 ; +case 16 :return 0x4 ; +case 3 :return 0x8 ; +case 6 :return 0x9 ; +case 12 :return 0xa ; +default : +error_setg (errp,"Interleave ways: %d not supported" , iw); +return 0 ; + } +} + +uint8_t cxl_interleave_granularity_enc (uint64_t gran,Error **errp) +{ +switch (gran) { +case 256 :return 0 ; +case 512 :return 1 ; +case 1024 :return 2 ; +case 2048 :return 3 ; +case 4096 :return 4 ; +case 8192 :return 5 ; +case 16384 :return 6 ; +default : +error_setg (errp,"Interleave granularity: %" PRIu64" invalid" , gran); +return 0 ; + } +}
include/hw/cxl/cxl_component.h +8 -0
@@ -193,4 +193,12 @@ void cxl_component_register_init_common(uint32_t *reg_state,void cxl_component_create_dvsec (CXLComponentState *cxl_cstate,uint16_t length,uint16_t type,uint8_t rev,uint8_t *body); +uint8_t cxl_interleave_ways_enc (int iw,Error **errp); +uint8_t cxl_interleave_granularity_enc (uint64_t gran,Error **errp); + +static inline hwaddr cxl_decode_ig (int ig) +{ +return 1 << (ig +8 ); +} +#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 26/43] hw/cxl/component: Add utils for interleave parameter encoding/decoding
Quoted 9 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
Both registers and the CFMWS entries in CDAT use simple encodings for the number of interleave ways and the interleave granularity. Introduce simple conversion functions to/from the unencoded number / size. So far the iw decode has not been needed so is it not implemented.
Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 27/43 hw/cxl/host: Add support for CXL Fixed Memory Windows.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions.
ExampleS:
-cxl-fixed-memory-window targets=cxl.0,size=128G
-cxl-fixed-memory-window targets=cxl.1,size=128G
-cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
Specifies
* 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity.
When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added.
Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
--- hw/cxl/cxl-host-stubs.c | 22 +++++++ hw/cxl/cxl-host.c | 138 ++++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 6 ++ include/hw/cxl/cxl.h | 20 ++++++ qapi/machine.json | 15 +++++ qemu-options.hx | 37 +++++++++++ softmmu/vl.c | 11 ++++ 7 files changed, 249 insertions(+)
hw/cxl/cxl-host-stubs.c new file +22 -0
@@ -0,0 +1,22 @@ +/* +* CXL host parameter parsing routine stubs +* +* Copyright (c) 2022 Huawei +*/ +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/option.h" +#include "hw/cxl/cxl.h" + +QemuOptsList qemu_cxl_fixed_window_opts = { + .name ="cxl-fixed-memory-window" , + .implied_opt_name ="type" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_cxl_fixed_window_opts.head), + .desc = { {0 } } +}; + +void parse_cxl_fixed_memory_window_opts (MachineState *ms) {}; + +void cxl_fixed_memory_window_link_targets (Error **errp) {}; + +const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c new file +138 -0
138 lines not on this page · raw message
hw/cxl/meson.build +6 -0
@@ -3,4 +3,10 @@ softmmu_ss.add(when: 'CONFIG_CXL', 'cxl-component-utils.c', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', + 'cxl-host.c', + ), + if_false: files( + 'cxl-host-stubs.c', )) + +softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))
include/hw/cxl/cxl.h +20 -0
@@ -10,6 +10,9 @@#ifndef CXL_H #define CXL_H +#include "qapi/qapi-types-machine.h" +#include "qemu/option.h" +#include "hw/pci/pci_bridge.h" #include "cxl_pci.h" #include "cxl_component.h" #include "cxl_device.h"
@@ -20,10 +23,27 @@#define TYPE_CXL_TYPE3_DEV "cxl-type3"#define CXL_WINDOW_MAX 10 +typedef struct CXLFixedWindow { +uint64_t size; +char **targets; +struct PXBDev *target_hbs[8 ]; +uint8_t num_targets; +uint8_t enc_int_ways; +uint8_t enc_int_gran; +/* Todo: XOR based interleaving */ +MemoryRegion mr; +hwaddr base; +}CXLFixedWindow ; +typedef struct CXLState {bool is_enabled;MemoryRegion host_mr;unsigned int next_mr_idx; +GList *fixed_windows; }CXLState ; +extern QemuOptsList qemu_cxl_fixed_window_opts; +void parse_cxl_fixed_memory_window_opts (MachineState *ms); +void cxl_fixed_memory_window_link_targets (Error **errp); +#endif
qapi/machine.json +15 -0
@@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDs +# +# Since X.X //fixme +## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: #
qemu-options.hx +37 -0
@@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)
softmmu/vl.c +11 -0
@@ -91,6 +91,7 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif
@@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); +cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {
@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_add_opts (&qemu_msg_opts);qemu_add_opts (&qemu_name_opts);qemu_add_opts (&qemu_numa_opts); +qemu_add_opts (&qemu_cxl_fixed_window_opts);qemu_add_opts (&qemu_icount_opts);qemu_add_opts (&qemu_semihosting_config_opts);qemu_add_opts (&qemu_fw_cfg_opts);
@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)exit (1 ); }break ; +case QEMU_OPTION_cxl_fixed_memory_window: + opts =qemu_opts_parse_noisily (qemu_find_opts ("cxl-fixed-memory-window" ), + optarg, true); +if (!opts) { +exit (1 ); + } +break ;case QEMU_OPTION_display:parse_display (optarg);break ;
@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_resolve_machine_memdev ();parse_numa_opts (current_machine); +parse_cxl_fixed_memory_window_opts (current_machine);if (vmstate_dump_file) {/* dump and exit */
Signature · 1 line
2.32.0
Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
Quoted 50 lines · Jonathan Cameron via
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions.
ExampleS:
(quoted diff) +0
-3-cxl-fixed-memory-window targets=cxl.0,size=128G -cxl-fixed-memory-window targets=cxl.1,size=128G -cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2kSpecifies
* 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity.When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added.Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
Signed-off-by Jonathan Cameron <[email protected]>
[...]
Quoted 5 lines
hw/cxl/cxl-host.c new file +0
-0
[...]
Quoted 7 lines
hw/cxl/cxl-host.c new file +7
-0+QemuOptsList qemu_cxl_fixed_window_opts = { + .name ="cxl-fixed-memory-window" , + .implied_opt_name ="type" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_cxl_fixed_window_opts.head), + .desc = { {0 } } +}; +
[...]
Quoted 30 lines
hw/cxl/cxl-host.c new file +30
-0+static int parse_cxl_fixed_memory_window (void *opaque,QemuOpts *opts, +Error **errp) +{ +CXLFixedMemoryWindowOptions *object =NULL ; +MachineState *ms =MACHINE (opaque); +Error *err =NULL ; +Visitor *v =opts_visitor_new (opts); + +visit_type_CXLFixedMemoryWindowOptions (v,NULL , &object, errp); +visit_free (v); +if (!object) { +return -1 ; + } + +set_cxl_fixed_memory_window_options (ms, object, &err); + +qapi_free_CXLFixedMemoryWindowOptions (object); +if (err) { +error_propagate (errp, err); +return -1 ; + } + +return 0 ; +} + +void parse_cxl_fixed_memory_window_opts (MachineState *ms) +{ +qemu_opts_foreach (qemu_find_opts ("cxl-fixed-memory-window" ), + parse_cxl_fixed_memory_window, ms, &error_fatal); +}
[...]
Quoted 5 lines
qapi/machine.json +0
-0
qapi/machine.json line 512
Quoted 9 lines
qapi/machine.json +6
-0@@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDs
Missing: @size, @targets.
qapi/machine.json line 514
Quoted 2 lines
qapi/machine.json +2
-0+# +# Since X.X //fixme
Well, "fix me, please".
Quoted 10 lines
qapi/machine.json +7
-0+## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: #
[...]
Quoted 91 lines
qemu-options.hx +37
-0@@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)softmmu/vl.c +10
-0@@ -91,6 +91,7 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif @@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); +cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_add_opts (&qemu_msg_opts);qemu_add_opts (&qemu_name_opts);qemu_add_opts (&qemu_numa_opts); +qemu_add_opts (&qemu_cxl_fixed_window_opts);qemu_add_opts (&qemu_icount_opts);qemu_add_opts (&qemu_semihosting_config_opts);qemu_add_opts (&qemu_fw_cfg_opts);@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)exit (1 ); }break ; +case QEMU_OPTION_cxl_fixed_memory_window: + opts =qemu_opts_parse_noisily (qemu_find_opts ("cxl-fixed-memory-window" ), + optarg, true); +if (!opts) { +exit (1 ); + } +break ;case QEMU_OPTION_display:parse_display (optarg);break ;
softmmu/vl.c void qemu_init(int argc, char **argv, char **envp) · line 3780
Quoted 7 lines
softmmu/vl.c +1
-0@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_resolve_machine_memdev ();parse_numa_opts (current_machine); +parse_cxl_fixed_memory_window_opts (current_machine);if (vmstate_dump_file) {/* dump and exit */
Have you considered using qobject_input_visitor_new_str() instead of QemuOpts?
Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
Quoted 237 lines · On Wed, 02 Mar 2022 07:55:45 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 50 lines · Jonathan Cameron via
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions.
ExampleS:
(quoted diff) +0
-3-cxl-fixed-memory-window targets=cxl.0,size=128G -cxl-fixed-memory-window targets=cxl.1,size=128G -cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2kSpecifies
* 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity.When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added.Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
Signed-off-by Jonathan Cameron <[email protected]>[...]
Quoted 5 lines
hw/cxl/cxl-host.c new file +0
-0[...]
Quoted 7 lines
hw/cxl/cxl-host.c new file +7
-0+QemuOptsList qemu_cxl_fixed_window_opts = { + .name ="cxl-fixed-memory-window" , + .implied_opt_name ="type" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_cxl_fixed_window_opts.head), + .desc = { {0 } } +}; +[...]
Quoted 30 lines
hw/cxl/cxl-host.c new file +30
-0+static int parse_cxl_fixed_memory_window (void *opaque,QemuOpts *opts, +Error **errp) +{ +CXLFixedMemoryWindowOptions *object =NULL ; +MachineState *ms =MACHINE (opaque); +Error *err =NULL ; +Visitor *v =opts_visitor_new (opts); + +visit_type_CXLFixedMemoryWindowOptions (v,NULL , &object, errp); +visit_free (v); +if (!object) { +return -1 ; + } + +set_cxl_fixed_memory_window_options (ms, object, &err); + +qapi_free_CXLFixedMemoryWindowOptions (object); +if (err) { +error_propagate (errp, err); +return -1 ; + } + +return 0 ; +} + +void parse_cxl_fixed_memory_window_opts (MachineState *ms) +{ +qemu_opts_foreach (qemu_find_opts ("cxl-fixed-memory-window" ), + parse_cxl_fixed_memory_window, ms, &error_fatal); +}[...]
Quoted 5 lines
qapi/machine.json +0
-0qapi/machine.json line 512
Quoted 9 lines
qapi/machine.json +6
-0@@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDsMissing: @size, @targets.
qapi/machine.json line 514
Quoted 2 lines
qapi/machine.json +2
-0+# +# Since X.X //fixmeWell, "fix me, please".
Quoted 10 lines
qapi/machine.json +7
-0+## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: #[...]
Quoted 96 lines
qemu-options.hx +37
-0@@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)softmmu/vl.c +11
-0@@ -91,6 +91,7 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif @@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); +cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_add_opts (&qemu_msg_opts);qemu_add_opts (&qemu_name_opts);qemu_add_opts (&qemu_numa_opts); +qemu_add_opts (&qemu_cxl_fixed_window_opts);qemu_add_opts (&qemu_icount_opts);qemu_add_opts (&qemu_semihosting_config_opts);qemu_add_opts (&qemu_fw_cfg_opts);@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)exit (1 ); }break ; +case QEMU_OPTION_cxl_fixed_memory_window: + opts =qemu_opts_parse_noisily (qemu_find_opts ("cxl-fixed-memory-window" ), + optarg, true); +if (!opts) { +exit (1 ); + } +break ;case QEMU_OPTION_display:parse_display (optarg);break ;@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_resolve_machine_memdev ();parse_numa_opts (current_machine); +parse_cxl_fixed_memory_window_opts (current_machine);
Quoted 6 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char **envp) · line 3780
Quoted 2 lines
softmmu/vl.c +0
-0if (vmstate_dump_file) {/* dump and exit */ Have you considered using
qobject_input_visitor_new_str()instead of QemuOpts?
Umm. No. Why might that be a better approach?
Thanks,
Jonathan
Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
Quoted 240 lines · On Fri, 4 Mar 2022 15:56:38 +0000 Jonathan Cameron…
Quoted 237 lines · On Wed, 02 Mar 2022 07:55:45 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 50 lines · Jonathan Cameron via
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions.
ExampleS:
(quoted diff) +0
-3-cxl-fixed-memory-window targets=cxl.0,size=128G -cxl-fixed-memory-window targets=cxl.1,size=128G -cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2kSpecifies
* 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity.When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added.Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
Signed-off-by Jonathan Cameron <[email protected]>[...]
Quoted 5 lines
hw/cxl/cxl-host.c new file +0
-0[...]
Quoted 7 lines
hw/cxl/cxl-host.c new file +7
-0+QemuOptsList qemu_cxl_fixed_window_opts = { + .name ="cxl-fixed-memory-window" , + .implied_opt_name ="type" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_cxl_fixed_window_opts.head), + .desc = { {0 } } +}; +[...]
Quoted 30 lines
hw/cxl/cxl-host.c new file +30
-0+static int parse_cxl_fixed_memory_window (void *opaque,QemuOpts *opts, +Error **errp) +{ +CXLFixedMemoryWindowOptions *object =NULL ; +MachineState *ms =MACHINE (opaque); +Error *err =NULL ; +Visitor *v =opts_visitor_new (opts); + +visit_type_CXLFixedMemoryWindowOptions (v,NULL , &object, errp); +visit_free (v); +if (!object) { +return -1 ; + } + +set_cxl_fixed_memory_window_options (ms, object, &err); + +qapi_free_CXLFixedMemoryWindowOptions (object); +if (err) { +error_propagate (errp, err); +return -1 ; + } + +return 0 ; +} + +void parse_cxl_fixed_memory_window_opts (MachineState *ms) +{ +qemu_opts_foreach (qemu_find_opts ("cxl-fixed-memory-window" ), + parse_cxl_fixed_memory_window, ms, &error_fatal); +}[...]
Quoted 5 lines
qapi/machine.json +0
-0qapi/machine.json line 512
Quoted 9 lines
qapi/machine.json +6
-0@@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDsMissing: @size, @targets.
qapi/machine.json line 514
Quoted 2 lines
qapi/machine.json +2
-0+# +# Since X.X //fixmeWell, "fix me, please".
Quoted 10 lines
qapi/machine.json +7
-0+## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: #[...]
Quoted 96 lines
qemu-options.hx +37
-0@@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)softmmu/vl.c +11
-0@@ -91,6 +91,7 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif @@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); +cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_add_opts (&qemu_msg_opts);qemu_add_opts (&qemu_name_opts);qemu_add_opts (&qemu_numa_opts); +qemu_add_opts (&qemu_cxl_fixed_window_opts);qemu_add_opts (&qemu_icount_opts);qemu_add_opts (&qemu_semihosting_config_opts);qemu_add_opts (&qemu_fw_cfg_opts);@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)exit (1 ); }break ; +case QEMU_OPTION_cxl_fixed_memory_window: + opts =qemu_opts_parse_noisily (qemu_find_opts ("cxl-fixed-memory-window" ), + optarg, true); +if (!opts) { +exit (1 ); + } +break ;case QEMU_OPTION_display:parse_display (optarg);break ;@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_resolve_machine_memdev ();parse_numa_opts (current_machine); +parse_cxl_fixed_memory_window_opts (current_machine);
Quoted 8 lines
Quoted 6 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char **envp) · line 3780
Quoted 2 lines
softmmu/vl.c +0
-0if (vmstate_dump_file) {/* dump and exit */ Have you considered using
qobject_input_visitor_new_str()instead of QemuOpts?Umm. No. Why might that be a better approach?
I did some more digging and found some of the discussions on this topic so seems like a sensible change to make for new API elements.
I have a naive go at this and appears the previous syntax of targets=cxl.1,targets=cxl.2 etc doesn't work and instead I need to use targets.0=cxl.1,targets.1=cxl.2
which I seems a little more clunky but I can live with it.
Jonathan
Quoted 6 lines
Thanks,
Jonathan
PATCH v6 28/43 acpi/cxl: Introduce CFMWS structures in CEDT
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The CEDT CXL Fixed Window Memory Window Structures (CFMWs) define regions of the host phyiscal address map which (via an impdef means) are configured such that they have a particular interleave setup across one or more CXL Host Bridges.
--- hw/acpi/cxl.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+)
hw/acpi/cxl.c +59 -0
@@ -60,6 +60,64 @@ static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)build_append_int_noprefix (table_data,memory_region_size (mr),8 ); } +/* +* CFMWS entries in CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. +* Interleave ways encoding in CXL 2.0 ECN: 3, 6, 12 and 16-way memory +* interleaving. +*/ +static void cedt_build_cfmws (GArray *table_data,MachineState *ms) +{ +CXLState *cxls = ms->cxl_devices_state; +GList *it; + +for (it = cxls->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; +int i; + +/* Type */ +build_append_int_noprefix (table_data,1 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Record Length */ +build_append_int_noprefix (table_data,36 +4 * fw->num_targets,2 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,4 ); + +/* Base HPA */ +build_append_int_noprefix (table_data, fw->mr.addr,8 ); + +/* Window Size */ +build_append_int_noprefix (table_data, fw->size,8 ); + +/* Host Bridge Interleave Ways */ +build_append_int_noprefix (table_data, fw->enc_int_ways,1 ); + +/* Host Bridge Interleave Arithmetic */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,2 ); + +/* Host Bridge Interleave Granularity */ +build_append_int_noprefix (table_data, fw->enc_int_gran,4 ); + +/* Window Restrictions */ +build_append_int_noprefix (table_data,0x0f ,2 );/* No restrictions */ + +/* QTG ID */ +build_append_int_noprefix (table_data,0 ,2 ); + +/* Host Bridge List (list of UIDs - currently bus_nr) */ +for (i =0 ; i < fw->num_targets; i++) { +g_assert (fw->target_hbs[i]); +build_append_int_noprefix (table_data, fw->target_hbs[i]->bus_nr,4 ); + } + } +} +static int cxl_foreach_pxb_hb (Object *obj,void *opaque) {Aml *cedt = opaque;
@@ -86,6 +144,7 @@ void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,/* reserve space for CEDT header */ object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); +cedt_build_cfmws (cedt->buf, ms);/* copy AML table into ACPI tables blob and patch header there */ g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len);
Signature · 1 line
2.32.0
Re: [PATCH v6 28/43] acpi/cxl: Introduce CFMWS structures in CEDT
Quoted 10 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
The CEDT CXL Fixed Window Memory Window Structures (CFMWs) define regions of the host phyiscal address map which (via an impdef means) are configured such that they have a particular interleave setup across one or more CXL Host Bridges.
Reported-by Alison Schofield <[email protected]>Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 29/43 hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
In reply to Jonathan CameronThis adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
--- hw/arm/Kconfig | 1 + hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)
hw/arm/Kconfig +1 -0
@@ -29,6 +29,7 @@ config ARM_VIRT select ACPI_APEI select ACPI_VIOT select VIRTIO_MEM_SUPPORTED + select ACPI_CXL config CHEETAH bool
hw/pci-host/gpex-acpi.c +19 -3
@@ -5,6 +5,7 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" +#include "hw/acpi/cxl.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq) {
@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;
@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); + dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); +if (is_cxl) { +struct Aml *pkg =aml_package (2 ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_STR" ,aml_unicode ("pxb Device" )));
@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs)); -acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev); + }else { +acpi_dsdt_add_pci_osc (dev); + }aml_append (scope, dev); }
Signature · 1 line
2.32.0
Re: [PATCH v6 29/43] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
Quoted 7 lines · Jonathan Cameron
This adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Ben Widawsky <[email protected]>
<snip>
Quoted 9 lines
hw/pci-host/gpex-acpi.c +1
-0@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) · line 159
Quoted 5 lines
hw/pci-host/gpex-acpi.c +2
-0@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); +
nit: you could hoist the setting of is_cxl to the declaration and leave it to the compiler to move it about if you want.
Anyway:
Signature · 1 line
Alex Bennée
PATCH v6 30/43 pci/pcie_port: Add pci_find_port_by_pn()
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Simple function to search a PCIBus to find a port by it's port number.
CXL interleave decoding uses the port number as a target so it is necessary to locate the port when doing interleave decoding.
--- hw/pci/pcie_port.c | 25 +++++++++++++++++++++++++ include/hw/pci/pcie_port.h | 2 ++ 2 files changed, 27 insertions(+)
hw/pci/pcie_port.c +25 -0
@@ -136,6 +136,31 @@ static void pcie_port_class_init(ObjectClass *oc, void *data)device_class_set_props (dc, pcie_port_props); } +PCIDevice *pcie_find_port_by_pn (PCIBus *bus,uint8_t pn) +{ +int devfn; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; +PCIEPort *port; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } + +if (!object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { +continue ; + } + + port =PCIE_PORT (d); +if (port->port == pn) { +return d; + } + } + +return NULL ; +} +static const TypeInfo pcie_port_type_info = { .name =TYPE_PCIE_PORT , .parent =TYPE_PCI_BRIDGE ,
include/hw/pci/pcie_port.h +2 -0
@@ -39,6 +39,8 @@ struct PCIEPort {void pcie_port_init_reg (PCIDevice *d); +PCIDevice *pcie_find_port_by_pn (PCIBus *bus,uint8_t pn); +#define TYPE_PCIE_SLOT "pcie-slot"OBJECT_DECLARE_SIMPLE_TYPE (PCIESlot,PCIE_SLOT )
Signature · 1 line
2.32.0
Re: [PATCH v6 30/43] pci/pcie_port: Add pci_find_port_by_pn()
Quoted 10 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
Simple function to search a PCIBus to find a port by it's port number.
CXL interleave decoding uses the port number as a target so it is necessary to locate the port when doing interleave decoding.
Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 31/43 CXL/cxl_component: Add cxl_get_hb_cstate()
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Accessor to get hold of the cxl state for a CXL host bridge without exposing the internals of the implementation.
--- hw/pci-bridge/pci_expander_bridge.c | 7 +++++++ include/hw/cxl/cxl_component.h | 2 ++ 2 files changed, 9 insertions(+)
hw/pci-bridge/pci_expander_bridge.c +7 -0
@@ -81,6 +81,13 @@ static GList *pxb_dev_list;#define TYPE_PXB_CXL_HOST "pxb-cxl-host"#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb) +{ +CXLHost *host =PXB_CXL_HOST (hb); + +return &host->cxl_cstate; +} +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);
include/hw/cxl/cxl_component.h +2 -0
@@ -201,4 +201,6 @@ static inline hwaddr cxl_decode_ig(int ig) return 1 << (ig + 8); } +CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +#endif
Signature · 1 line
2.32.0
Re: [PATCH v6 31/43] CXL/cxl_component: Add cxl_get_hb_cstate()
Quoted 6 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
Accessor to get hold of the cxl state for a CXL host bridge without exposing the internals of the implementation.
Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
PATCH v6 32/43 mem/cxl_type3: Add read and write functions for associated hostmem.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Once a read or write reaches a CXL type 3 device, the HDM decoders on the device are used to establish the Device Physical Address which should be accessed. These functions peform the required maths and then directly access the hostmem->mr to fullfil the actual operation. Note that failed writes are silent, but failed reads return poison. Note this is based loosely on:
https://lore.kernel.org/qemu-devel/[email protected]/
[RFC PATCH 0/9] hw/misc: Add support for interleaved memory accesses
Only lightly tested so far. More complex test cases yet to be written.
--- hw/mem/cxl_type3.c | 81 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 5 +++ 2 files changed, 86 insertions(+)
hw/mem/cxl_type3.c +81 -0
@@ -161,6 +161,87 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) &ct3d->cxl_dstate.device_registers); } +/* TODO: Support multiple HDM decoders and DPA skip */ +static bool cxl_type3_dpa (CXLType3Dev *ct3d,hwaddr host_addr,uint64_t *dpa) +{ +uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers; +uint64_t decoder_base, decoder_size, hpa_offset; +uint32_t hdm0_ctrl; +int ig, iw; + + decoder_base = (((uint64_t )cache_mem[R_CXL_HDM_DECODER0_BASE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_BASE_LO ]); +if ((uint64_t )host_addr < decoder_base) { +return false; + } + + hpa_offset = (uint64_t )host_addr - decoder_base; + + decoder_size = ((uint64_t )cache_mem[R_CXL_HDM_DECODER0_SIZE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_SIZE_LO ]; +if (hpa_offset >= decoder_size) { +return false; + } + + hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL ]; + iw =FIELD_EX32 (hdm0_ctrl,CXL_HDM_DECODER0_CTRL ,IW ); + ig =FIELD_EX32 (hdm0_ctrl,CXL_HDM_DECODER0_CTRL ,IG ); + + *dpa = (MAKE_64BIT_MASK (0 ,8 + ig) & hpa_offset) | + ((MAKE_64BIT_MASK (8 + ig + iw,64 -8 - ig - iw) & hpa_offset) >> iw); + +return true; +} + +MemTxResult cxl_type3_read (PCIDevice *d,hwaddr host_addr,uint64_t *data, +unsigned size,MemTxAttrs attrs) +{ +CXLType3Dev *ct3d =CT3 (d); +uint64_t dpa_offset; +MemoryRegion *mr; + +/* TODO support volatile region */ + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return MEMTX_ERROR ; + } + +if (!cxl_type3_dpa (ct3d, host_addr, &dpa_offset)) { +return MEMTX_ERROR ; + } + +if (dpa_offset >int128_get64 (mr->size)) { +return MEMTX_ERROR ; + } + +return memory_region_dispatch_read (mr, dpa_offset, data, +size_memop (size), attrs); +} + +MemTxResult cxl_type3_write (PCIDevice *d,hwaddr host_addr,uint64_t data, +unsigned size,MemTxAttrs attrs) +{ +CXLType3Dev *ct3d =CT3 (d); +uint64_t dpa_offset; +MemoryRegion *mr; + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return MEMTX_OK ; + } + +if (!cxl_type3_dpa (ct3d, host_addr, &dpa_offset)) { +return MEMTX_OK ; + } + +if (dpa_offset >int128_get64 (mr->size)) { +return MEMTX_OK ; + } + +return memory_region_dispatch_write (mr, dpa_offset, data, +size_memop (size), attrs); +} +static void ct3d_reset (DeviceState *dev) {CXLType3Dev *ct3d =CT3 (dev);
include/hw/cxl/cxl_device.h +5 -0
@@ -264,4 +264,9 @@ struct CXLType3Class { uint64_t offset); }; +MemTxResult cxl_type3_read (PCIDevice *d,hwaddr host_addr,uint64_t *data, +unsigned size,MemTxAttrs attrs); +MemTxResult cxl_type3_write (PCIDevice *d,hwaddr host_addr,uint64_t data, +unsigned size,MemTxAttrs attrs); +#endif
Signature · 1 line
2.32.0
PATCH v6 33/43 cxl/cxl-host: Add memops for CFMWS region.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
These memops perform interleave decoding, walking down the CXL topology from CFMWS described host interleave decoder via CXL host bridge HDM decoders, through the CXL root ports and finally call CXL type 3 specific read and write functions.
Note that, whilst functional the current implementation does not support:
* switches
* multiple HDM decoders at a given level.
* unaligned accesses across the interleave boundaries
--- hw/cxl/cxl-host.c | 124 +++++++++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl.h | 2 + 2 files changed, 126 insertions(+)
hw/cxl/cxl-host.c +124 -0
@@ -136,3 +136,127 @@ void cxl_fixed_memory_window_link_targets(Error **errp) } } } + +/* TODO: support, multiple hdm decoders */ +static bool cxl_hdm_find_target (uint32_t *cache_mem,hwaddr addr, +uint8_t *target) +{ +uint32_t ctrl; +uint32_t ig_enc; +uint32_t iw_enc; +uint32_t target_reg; +uint32_t target_idx; + + ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL ]; +if (!FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,COMMITTED )) { +return false; + } + + ig_enc =FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,IG ); + iw_enc =FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,IW ); + target_idx = (addr /cxl_decode_ig (ig_enc)) % (1 << iw_enc); + +if (target_idx >4 ) { + target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO ]; + target_reg >>= target_idx *8 ; + }else { + target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO ]; + target_reg >>= (target_idx -4 ) *8 ; + } + *target = target_reg &0xff ; + +return true; +} + +static PCIDevice *cxl_cfmws_find_device (CXLFixedWindow *fw,hwaddr addr) +{ +CXLComponentState *hb_cstate; +PCIHostState *hb; +int rb_index; +uint32_t *cache_mem; +uint8_t target; +bool target_found; +PCIDevice *rp, *d; + +/* Address is relative to memory region. Convert to HPA */ + addr += fw->base; + + rb_index = (addr /cxl_decode_ig (fw->enc_int_gran)) % fw->num_targets; + hb =PCI_HOST_BRIDGE (fw->target_hbs[rb_index]->cxl.cxl_host_bridge); +if (!hb || !hb->bus || !pci_bus_is_cxl (hb->bus)) { +return NULL ; + } + + hb_cstate =cxl_get_hb_cstate (hb); +if (!hb_cstate) { +return NULL ; + } + + cache_mem = hb_cstate->crb.cache_mem_registers; + + target_found =cxl_hdm_find_target (cache_mem, addr, &target); +if (!target_found) { +return NULL ; + } + + rp =pcie_find_port_by_pn (hb->bus, target); +if (!rp) { +return NULL ; + } + + d =pci_bridge_get_sec_bus (PCI_BRIDGE (rp))->devices[0 ]; + +if (!d || !object_dynamic_cast (OBJECT (d),TYPE_CXL_TYPE3_DEV )) { +return NULL ; + } + +return d; +} + +static MemTxResult cxl_read_cfmws (void *opaque,hwaddr addr,uint64_t *data, +unsigned size,MemTxAttrs attrs) +{ +CXLFixedWindow *fw = opaque; +PCIDevice *d; + + d =cxl_cfmws_find_device (fw, addr); +if (d ==NULL ) { + *data =0 ; +/* Reads to invalid address return poison */ +return MEMTX_ERROR ; + } + +return cxl_type3_read (d, addr + fw->base, data, size, attrs); +} + +static MemTxResult cxl_write_cfmws (void *opaque,hwaddr addr, +uint64_t data,unsigned size, +MemTxAttrs attrs) +{ +CXLFixedWindow *fw = opaque; +PCIDevice *d; + + d =cxl_cfmws_find_device (fw, addr); +if (d ==NULL ) { +/* Writes to invalid address are silent */ +return MEMTX_OK ; + } + +return cxl_type3_write (d, addr + fw->base, data, size, attrs); +} + +const MemoryRegionOps cfmws_ops = { + .read_with_attrs = cxl_read_cfmws, + .write_with_attrs = cxl_write_cfmws, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = true, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = true, + }, +};
include/hw/cxl/cxl.h +2 -0
@@ -46,4 +46,6 @@ extern QemuOptsList qemu_cxl_fixed_window_opts;void parse_cxl_fixed_memory_window_opts (MachineState *ms);void cxl_fixed_memory_window_link_targets (Error **errp); +extern const MemoryRegionOps cfmws_ops; +#endif
Signature · 1 line
2.32.0
PATCH v6 34/43 RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Inorder to implement memory interleaving we need a means to proxy the calls. Adding mem_ops allows such proxying.
Note should have no impact on use cases not using _dispatch_read/write. For now, only file backed hostmem is considered to seek feedback on the approach before considering other hostmem backends.
--- softmmu/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+)
softmmu/memory.c +9 -0
@@ -1606,6 +1606,15 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,Error *err =NULL ;memory_region_init (mr, owner, name, size); mr->ram = true; + +/* +* ops used only when directly accessing via +* - memory_region_dispatch_read() +* - memory_region_dispatch_write() +*/ + mr->ops = &ram_device_mem_ops; + mr->opaque = mr; + mr->readonly = readonly; mr->terminates = true; mr->destructor = memory_region_destructor_ram;
Signature · 1 line
2.32.0
PATCH v6 35/43 hw/cxl/component Add a dumb HDM decoder handler
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Add a trivial handler for now to cover the root bridge where we could do some error checking in future.
--- hw/cxl/cxl-component-utils.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
hw/cxl/cxl-component-utils.c +31 -0
@@ -32,6 +32,31 @@ static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset, } } +static void dumb_hdm_handler (CXLComponentState *cxl_cstate,hwaddr offset, +uint32_t value) +{ +ComponentRegisters *cregs = &cxl_cstate->crb; +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; + +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); +break ; +default : +break ; + } + +memory_region_transaction_begin (); +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) { +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERR ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); + } +memory_region_transaction_commit (); +} +static void cxl_cache_mem_write_reg (void *opaque,hwaddr offset,uint64_t value,unsigned size) {
@@ -45,6 +70,12 @@ static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value, }if (cregs->special_ops && cregs->special_ops->write) { cregs->special_ops->write (cxl_cstate, offset, value, size); +return ; + } + +if (offset >=A_CXL_HDM_DECODER_CAPABILITY && + offset <=A_CXL_HDM_DECODER0_TARGET_LIST_HI ) { +dumb_hdm_handler (cxl_cstate, offset, value); }else { cregs->cache_mem_registers[offset /4 ] = value; }
Signature · 1 line
2.32.0
PATCH v6 36/43 i386/pc: Enable CXL fixed memory windows
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Add the CFMWs memory regions to the memorymap and adjust the PCI window to avoid hitting the same memory.
--- hw/i386/pc.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
hw/i386/pc.c +30 -1
@@ -816,7 +816,7 @@ void pc_memory_init(PCMachineState *pcms,MachineClass *mc =MACHINE_GET_CLASS (machine);PCMachineClass *pcmc =PC_MACHINE_GET_CLASS (pcms);X86MachineState *x86ms =X86_MACHINE (pcms); -hwaddr cxl_base; +hwaddr cxl_base, cxl_resv_end =0 ;assert (machine->ram_size == x86ms->below_4g_mem_size + x86ms->above_4g_mem_size);
@@ -924,6 +924,24 @@ void pc_memory_init(PCMachineState *pcms,e820_add_entry (cxl_base, cxl_size,E820_RESERVED );memory_region_init (mr,OBJECT (machine),"cxl_host_reg" , cxl_size);memory_region_add_subregion (system_memory, cxl_base, mr); + cxl_resv_end = cxl_base + cxl_size; +if (machine->cxl_devices_state->fixed_windows) { +hwaddr cxl_fmw_base; +GList *it; + + cxl_fmw_base =ROUND_UP (cxl_base + cxl_size,256 * MiB); +for (it = machine->cxl_devices_state->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; + + fw->base = cxl_fmw_base; +memory_region_init_io (&fw->mr,OBJECT (machine), &cfmws_ops, fw, +"cxl-fixed-memory-region" , fw->size); +memory_region_add_subregion (system_memory, fw->base, &fw->mr); +e820_add_entry (fw->base, fw->size,E820_RESERVED ); + cxl_fmw_base += fw->size; + cxl_resv_end = cxl_fmw_base; + } + } }/* Initialize PC system firmware */
@@ -953,6 +971,10 @@ void pc_memory_init(PCMachineState *pcms,if (!pcmc->broken_reserved_end) { res_mem_end +=memory_region_size (&machine->device_memory->mr); } + +if (machine->cxl_devices_state->is_enabled) { + res_mem_end = cxl_resv_end; + } *val =cpu_to_le64 (ROUND_UP (res_mem_end,1 * GiB));fw_cfg_add_file (fw_cfg,"etc/reserved-memory-end" , val,sizeof (*val)); }
@@ -989,6 +1011,13 @@ uint64_t pc_pci_hole64_start(void)if (ms->cxl_devices_state->host_mr.addr) { hole64_start = ms->cxl_devices_state->host_mr.addr +memory_region_size (&ms->cxl_devices_state->host_mr); +if (ms->cxl_devices_state->fixed_windows) { +GList *it; +for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; + hole64_start = fw->mr.addr +memory_region_size (&fw->mr); + } + } }else if (pcmc->has_reserved_memory && ms->device_memory->base) { hole64_start = ms->device_memory->base;if (!pcmc->broken_reserved_end) {
Signature · 1 line
2.32.0
PATCH v6 37/43 tests/acpi: q35: Allow addition of a CXL test.
In reply to Jonathan CameronAdd exceptions for the DSDT and the new CEDT tables specific to a new CXL test in the following patch.
--- tests/data/acpi/q35/CEDT.cxl | 0 tests/data/acpi/q35/DSDT.cxl | 0 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++ 3 files changed, 2 insertions(+)
tests/data/acpi/q35/CEDT.cxl new file +0 -0
tests/data/acpi/q35/DSDT.cxl new file +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +2 -0
@@ -1 +1,3 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.cxl" , +"tests/data/acpi/q35/CEDT.cxl",
Signature · 1 line
2.32.0
PATCH v6 38/43 qtests/bios-tables-test: Add a test for CXL emulation.
In reply to Jonathan CameronThe DSDT includes several CXL specific elements and the CEDT table is only present if we enable CXL.
The test exercises all current functionality with several CFMWS, CHBS structures in CEDT and ACPI0016/ACPI00017 and _OSC entries in DSDT.
--- tests/qtest/bios-tables-test.c | 39 ++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
tests/qtest/bios-tables-test.c +39 -0
@@ -1537,6 +1537,44 @@ static void test_acpi_q35_viot(void)free_test_data (&data); } +static void test_acpi_q35_cxl (void ) +{ +gchar *tmp_path =g_dir_make_tmp ("qemu-test-cxl.XXXXXX" ,NULL ); +gchar *params; + +test_data data = { + .machine =MACHINE_Q35 , + .variant =".cxl" , + }; +/* +* A complex CXL setup. +*/ + params =g_strdup_printf (" -machine cxl=on" +" -object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M" +" -object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M" +" -object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M" +" -object memory-backend-file,id=cxl-mem4,mem-path=%s,size=256M" +" -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1" +" -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2" +" -device cxl-rp,port=0,bus=cxl.1,id=rp1,chassis=0,slot=2" +" -device cxl-type3,bus=rp1,memdev=cxl-mem1,size=256M" +" -device cxl-rp,port=1,bus=cxl.1,id=rp2,chassis=0,slot=3" +" -device cxl-type3,bus=rp2,memdev=cxl-mem2,size=256M" +" -device cxl-rp,port=0,bus=cxl.2,id=rp3,chassis=0,slot=5" +" -device cxl-type3,bus=rp3,memdev=cxl-mem3,size=256M" +" -device cxl-rp,port=1,bus=cxl.2,id=rp4,chassis=0,slot=6" +" -device cxl-type3,bus=rp4,memdev=cxl-mem4,size=256M" +" -cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k" +" -cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k" , + tmp_path, tmp_path, tmp_path, tmp_path); +test_acpi_one (params, &data); + +g_free (params); +g_assert (g_rmdir (tmp_path) ==0 ); +g_free (tmp_path); +free_test_data (&data); +} +static void test_acpi_virt_viot (void ) {test_data data = {
@@ -1742,6 +1780,7 @@ int main(int argc, char *argv[])qtest_add_func ("acpi/q35/kvm/dmar" , test_acpi_q35_kvm_dmar); }qtest_add_func ("acpi/q35/viot" , test_acpi_q35_viot); +qtest_add_func ("acpi/q35/cxl" , test_acpi_q35_cxl);qtest_add_func ("acpi/q35/slic" , test_acpi_q35_slic); }else if (strcmp (arch,"aarch64" ) ==0 ) {if (has_tcg) {
Signature · 1 line
2.32.0
PATCH v6 39/43 tests/acpi: Add tables for CXL emulation.
In reply to Jonathan CameronTables that differ from normal Q35 tables when running the CXL test.
--- tests/data/acpi/q35/CEDT.cxl | Bin 0 -> 184 bytes tests/data/acpi/q35/DSDT.cxl | Bin 0 -> 9627 bytes tests/qtest/bios-tables-test-allowed-diff.h | 2 -- 3 files changed, 2 deletions(-)
tests/data/acpi/q35/CEDT.cxl binary +0 -0
tests/data/acpi/q35/DSDT.cxl binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -2
@@ -1,3 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/q35/DSDT.cxl" , -"tests/data/acpi/q35/CEDT.cxl",
Signature · 1 line
2.32.0
PATCH v6 40/43 qtest/cxl: Add more complex test cases with CFMWs
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Add CXL Fixed Memory Windows to the CXL tests.
--- tests/qtest/cxl-test.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
tests/qtest/cxl-test.c +5 -4
@@ -9,11 +9,13 @@#include "libqtest-single.h" #define QEMU_PXB_CMD "-machine q35,cxl=on " \ - "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-cxl-fixed-memory-window targets=cxl.0,size=4G " -#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ +#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ - "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ + "-cxl-fixed-memory-window targets=cxl.0,targets=cxl.1,size=4G "#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
@@ -135,7 +137,6 @@ static void cxl_2pxb_4rp_4t3d(void)int main (int argc,char **argv) {g_test_init (&argc, &argv,NULL ); -qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb);qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb);qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window);
Signature · 1 line
2.32.0
PATCH v6 41/43 hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl
In reply to Jonathan CameronCode based on i386/pc enablement.
The memory layout places space for 16 host bridge register regions after the GIC_REDIST2 in the extended memmap.
The CFMWs are placed above the extended memmap.
Only create the CEDT table if cxl=on set for the machine.
--- hw/arm/virt-acpi-build.c | 33 +++++++++++++++++++++++++++++++++ hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 1 + 3 files changed, 73 insertions(+), 1 deletion(-)
hw/arm/virt-acpi-build.c +33 -0
@@ -39,9 +39,11 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/utils.h" #include "hw/acpi/pci.h" +#include "hw/acpi/cxl.h" #include "hw/acpi/memory_hotplug.h" #include "hw/acpi/generic_event_device.h" #include "hw/acpi/tpm.h" +#include "hw/cxl/cxl.h" #include "hw/pci/pcie_host.h" #include "hw/pci/pci.h" #include "hw/pci/pci_bus.h"
@@ -157,10 +159,29 @@ static void acpi_dsdt_add_virtio(Aml *scope, } } +/* Uses local definition of AcpiBuildState so can't easily be common code */ +static void build_acpi0017 (Aml *table) +{ +Aml *dev, *scope, *method; + + scope =aml_scope ("_SB" ); + dev =aml_device ("CXLM" ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + + method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED ); +aml_append (method,aml_return (aml_int (0x01 ))); +aml_append (dev, method); + +aml_append (scope, dev); +aml_append (table, scope); +} +static void acpi_dsdt_add_pci (Aml *scope,const MemMapEntry *memmap,uint32_t irq,VirtMachineState *vms) {int ecam_id =VIRT_ECAM_ID (vms->highmem_ecam); +bool cxl_present = false; +PCIBus *bus = vms->bus;struct GPEXConfig cfg = { .mmio32 = memmap[VIRT_PCIE_MMIO ], .pio = memmap[VIRT_PCIE_PIO ],
@@ -174,6 +195,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, }acpi_dsdt_add_gpex (scope, &cfg); + QLIST_FOREACH(bus, &vms->bus->child, sibling) { +if (pci_bus_is_cxl (bus)) { + cxl_present = true; + } + } +if (cxl_present) { +build_acpi0017 (scope); + } } static void acpi_dsdt_add_gpio(Aml *scope, const MemMapEntry *gpio_memmap,
@@ -991,6 +1020,10 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables) vms->oem_table_id); } } +if (ms->cxl_devices_state->is_enabled) { +cxl_build_cedt (ms, table_offsets, tables_blob, tables->linker, + vms->oem_id, vms->oem_table_id); + }if (ms->nvdimms_state->is_enabled) { nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
hw/arm/virt.c +39 -1
@@ -78,6 +78,7 @@#include "hw/virtio/virtio-mem-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" +#include "hw/cxl/cxl.h" #include "qemu/guest-random.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \
@@ -178,6 +179,7 @@ static const MemMapEntry base_memmap[] = {static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_CXL_HOST ] = {0x0 ,64 * KiB *16 },/* 16 UID */ [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB },/* Second PCIe window */ [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB },
@@ -1508,6 +1510,17 @@ static void create_pcie(VirtMachineState *vms) } } +static void create_cxl_host_reg_region (VirtMachineState *vms) +{ +MemoryRegion *sysmem =get_system_memory (); +MachineState *ms =MACHINE (vms); +MemoryRegion *mr = &ms->cxl_devices_state->host_mr; + +memory_region_init (mr,OBJECT (ms),"cxl_host_reg" , + vms->memmap[VIRT_CXL_HOST ].size); +memory_region_add_subregion (sysmem, vms->memmap[VIRT_CXL_HOST ].base, mr); +} +static void create_platform_bus (VirtMachineState *vms) {DeviceState *dev;
@@ -1670,7 +1683,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms,int pa_bits) {MachineState *ms =MACHINE (vms); -hwaddr base, device_memory_base, device_memory_size, memtop; +hwaddr base, device_memory_base, device_memory_size, memtop, cxl_fmw_base;int i; vms->memmap = extended_memmap;
@@ -1762,6 +1775,20 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits)memory_region_init (&ms->device_memory->mr,OBJECT (vms),"device-memory" , device_memory_size); } + +if (ms->cxl_devices_state->fixed_windows) { +GList *it; + + cxl_fmw_base =ROUND_UP (base,256 * MiB); +for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; + + fw->base = cxl_fmw_base; +memory_region_init_io (&fw->mr,OBJECT (vms), &cfmws_ops, fw, +"cxl-fixed-memory-region" , fw->size); + cxl_fmw_base += fw->size; + } + } }/*
@@ -2164,6 +2191,15 @@ static void machvirt_init(MachineState *machine)memory_region_add_subregion (sysmem, machine->device_memory->base, &machine->device_memory->mr); } +if (machine->cxl_devices_state->fixed_windows) { +GList *it; +for (it = machine->cxl_devices_state->fixed_windows; it; + it = it->next) { +CXLFixedWindow *fw = it->data; + +memory_region_add_subregion (sysmem, fw->base, &fw->mr); + } + }virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem);
@@ -2190,6 +2226,7 @@ static void machvirt_init(MachineState *machine)create_rtc (vms);create_pcie (vms); +create_cxl_host_reg_region (vms);if (has_ged && aarch64 && firmware_loaded &&virt_is_acpi_enabled (vms)) { vms->acpi_dev =create_acpi_ged (vms);
@@ -2845,6 +2882,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->unplug = virt_machine_device_unplug_cb; mc->nvdimm_supported = true; mc->smp_props.clusters_supported = true; + mc->cxl_supported = true; mc->auto_enable_numa_with_memhp = true; mc->auto_enable_numa_with_memdev = true; mc->default_ram_id ="mach-virt.ram" ;
include/hw/arm/virt.h +1 -0
@@ -92,6 +92,7 @@ enum {/* indices of IO regions located after the RAM */ enum {VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_CXL_HOST ,VIRT_HIGH_PCIE_ECAM ,VIRT_HIGH_PCIE_MMIO , };
Signature · 1 line
2.32.0
PATCH v6 42/43 qtest/cxl: Add aarch64 virt test for CXL
In reply to Jonathan CameronAdd a single complex case for aarch64 virt machine.
--- tests/qtest/cxl-test.c | 49 +++++++++++++++++++++++++++++++++-------- tests/qtest/meson.build | 1 + 2 files changed, 41 insertions(+), 9 deletions(-)
tests/qtest/cxl-test.c +40 -9
@@ -17,6 +17,11 @@"-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ "-cxl-fixed-memory-window targets=cxl.0,targets=cxl.1,size=4G " +#define QEMU_VIRT_2PXB_CMD "-machine virt,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \ + "-cxl-fixed-memory-window targets=cxl.0,targets=cxl.1,size=4G " +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "/* Dual ports on first pxb */
@@ -134,17 +139,43 @@ static void cxl_2pxb_4rp_4t3d(void)g_string_free (cmdline, TRUE); } +static void cxl_virt_2pxb_4rp_4t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_VIRT_2PXB_CMD QEMU_4RP QEMU_4T3D, + tmpfs, tmpfs, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} +int main (int argc,char **argv) { +const char *arch =qtest_get_arch (); +g_test_init (&argc, &argv,NULL ); -qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb); -qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); -qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); -qtest_add_func ("/pci/cxl/pxb_x2_with_window" , cxl_2pxb_with_window); -qtest_add_func ("/pci/cxl/rp" , cxl_root_port); -qtest_add_func ("/pci/cxl/rp_x2" , cxl_2root_port); -qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); -qtest_add_func ("/pci/cxl/rp_x2_type3_x2" , cxl_1pxb_2rp_2t3d); -qtest_add_func ("/pci/cxl/pxb_x2_root_port_x4_type3_x4" , cxl_2pxb_4rp_4t3d); +if (strcmp (arch,"i386" ) ==0 ||strcmp (arch,"x86_64" ) ==0 ) { +qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb); +qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); +qtest_add_func ("/pci/cxl/pxb_x2_with_window" , cxl_2pxb_with_window); +qtest_add_func ("/pci/cxl/rp" , cxl_root_port); +qtest_add_func ("/pci/cxl/rp_x2" , cxl_2root_port); +qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); +qtest_add_func ("/pci/cxl/rp_x2_type3_x2" , cxl_1pxb_2rp_2t3d); +qtest_add_func ("/pci/cxl/pxb_x2_root_port_x4_type3_x4" , + cxl_2pxb_4rp_4t3d); + }else if (strcmp (arch,"aarch64" ) ==0 ) { +qtest_add_func ("/pci/cxl/virt/pxb_x2_root_port_x4_type3_x4" , + cxl_virt_2pxb_4rp_4t3d); + }return g_test_run (); }
tests/qtest/meson.build +1 -0
@@ -221,6 +221,7 @@ qtests_aarch64 = \ (config_all_devices.has_key('CONFIG_TPM_TIS_SYSBUS') ? ['tpm-tis-device-test'] : []) + \ (config_all_devices.has_key('CONFIG_TPM_TIS_SYSBUS') ? ['tpm-tis-device-swtpm-test'] : []) + \ (config_all_devices.has_key('CONFIG_XLNX_ZYNQMP_ARM') ? ['xlnx-can-test', 'fuzz-xlnx-dp-test'] : []) + \ + qtests_cxl + \ ['arm-cpu-features', 'numa-test', 'boot-serial-test',
Signature · 1 line
2.32.0
PATCH v6 43/43 docs/cxl: Add initial Compute eXpress Link (CXL) documentation.
In reply to Jonathan CameronProvide an introduction to the main components of a CXL system, with detailed explanation of memory interleaving, example command lines and kernel configuration.
This was a challenging document to write due to the need to extract only that subset of CXL information which is relevant to either users of QEMU emulation of CXL or to those interested in the implementation. Much of CXL is concerned with specific elements of the protocol, management of memory pooling etc which is simply not relevant to what is currently planned for CXL emulation in QEMU. All comments welcome
--- docs/system/device-emulation.rst | 1 + docs/system/devices/cxl.rst | 302 +++++++++++++++++++++++++++++++ 2 files changed, 303 insertions(+)
docs/system/device-emulation.rst +1 -0
@@ -83,6 +83,7 @@ Emulated Devices :maxdepth: 1 devices/can.rst + devices/cxl.rst devices/ivshmem.rst devices/net.rst devices/nvme.rst
docs/system/devices/cxl.rst new file +302 -0
302 lines not on this page · raw message
Signature · 1 line
2.32.0
Quoted 3 lines · On Fri, 11 Feb 2022 12:07:04 +0000 Jonathan Cameron…
This version is mainly a reorganization of v5 patch ordering to make for more sensible stepwise review and merging. I took the opportunity to add a few more tests and introduce an overview document.
In case anyone is interested at:
https://gitlab.com/jic23/qemu/-/commits/cxl-v7-draft-for-test
is a draft of what will be v7 with basic CXL switch support and the issue that Ben pointed out in patch 17 fixed.
No change to anything before patch 17 so does not affect the proposed first block of patches to apply (i.e. no need to wait for v7 if it makes sense to pick those up sooner!)
There was some follow through from the fix in the later qtests as they now correctly require LSA / Label Storage Areas to be provided for CXL-mem types. I've also moved the initial type 3 test later in the series so that lsa parameter is required by the time it is introduced. The LSA requirement may be relaxed in the future when volatile memory support is added as then an LSA becomes optional.
The switch support includes interleave decoding for one layer of switches and at least for basic tests it all seems to work. Note that with current kernel proposal you'll need 2 ports on a host bridge (one can have nothing attached to it) so as to ensure the kernel programs the HDM decoders. I'll see if I can find an elegant solution on the kernel side for this as single port HB with HDM decoders is allowed by the CXL spec.
Next up is fleshing out sanity checks / write masks etc for various interfaces so that the emulation will reject invalid writes etc much as real hardware would be expected to do.
Thanks,
Jonathan
Quoted 240 lines
Changes since v5: Thanks to Michael and Igor for comments.
- Reorganize series to make it more sensible to take in several chunks. Includes splitting qtest/cxl-test up so that we have basic testing for each stage. Suggested chunks as follows:
1-15: Infrastructure + host bridge
At this stage the pxb-cxl host bridge it is presented to the OS as a legacy / pcie host bridge.
This set is close to those Michael Tsirkin suggested picking up in v5, but the cxl=on machine parameter was pulled forwards to avoid breaking backwards compatibility. It could logically be delayed until HB MMIO is introduced but only at cost of then failing to instantiate pxb-cxl in cases that would previously have 'worked'.
16-22: Root port, plus CXL type 3 device.
Will allow Linux cxl-pci driver to bind and present some basic features of the type 3 devices.
23-40: x86 PC support including HB MMIO and CFMWS
At this region creation and use is possible on x86 platforms. Includes a new bios-table-test and more complex qtests/cxl-test cases. Includes the interleaving code and an RFC about adding ops tomemory_region_ram_init_from_file().
41-42: ARM virt enabling equivalent to that done for x86. 43: Documentation
I can break this down if helpful as it includes command line examples only possible after support is in place for all elements.
I've had gitlab CI pipelines testing at patch 15, 22, and 43 and other than the build-oss-fuzz which is timing out and a false positive (I think) in checkpatch they are clean.
http://gitlab.com/jic23/qemu/~/pipelines
(trivial commit message update followed pushing those out, but
the code is the same).- Add a documentation patch as requested by Alex in v4 review.
- Drop the code unification around PCI host bridge setup.
It will need extensive testing and was only of marginal benefit for this series so get it off the critical path.
- Don't build CEDT unless machine parameter cxl=on is set. Tests added to exercise the cxl=on case and generate meaningful tables (not empty).
- Move handling of pxb-cxl reset to a later patch where the required infrastructure has been created (avoids a crash).
- Split up qtests/cxl-test to build up step by step. Introduced an aarch64 test case covering most complex case only as this probably sufficient to catch any arch specific problems.
- Introduce test plus bios tables CEDT.cxl and DSDT.cxl.Updated background info from v5:
Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.Big TODOs:
* Lack of unaligned accesses to interleaved memory. Are there any side effects of this restriction?
* Volatile memory devices (easy but it's more code so left for now).
* Switch support. Linux kernel support is under review currently, so there is now something to test against.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
* CFMWS flags as requested by Ben.Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches. The kernel CXL mocking code allows some forms of testing, but QEMU provides a more versatile and exensible platform.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/What we have here is about what you need for it to be useful for testing currently kernel code. Note the kernel code is moving fast so since v4, some features have been introduced we don't yet support in QEMU (e.g. use of the PCIe serial number extended capability).
All comments welcome.
Additional info that was here in v5 is now in the documentation patch.
Thanks,
Jonathan
Ben Widawsky (24):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory device utilities hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge) hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) hw/cxl/device: Add some trivial commands hw/cxl/device: Plumb real Label Storage Area (LSA) sizing hw/cxl/device: Implement get/set Label Storage Area (LSA) hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/cxl: Add_OSCimplementation (9.14.2)
acpi/cxl: Create the CEDT (9.14.1)
acpi/cxl: Introduce CFMWS structures in CEDT hw/cxl/component Add a dumb HDM decoder handler qtest/cxl: Add more complex test cases with CFMWsJonathan Cameron (19):
MAINTAINERS: Add entry for Compute Express Link Emulation
cxl: Machine level control on whether CXL support is enabled
qtest/cxl: Introduce initial test for pxb-cxl only.
qtests/cxl: Add initial root port and CXL type3 tests
hw/cxl/component: Add utils for interleave parameter encoding/decoding
hw/cxl/host: Add support for CXL Fixed Memory Windows.
hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
pci/pcie_port: Addpci_find_port_by_pn()
CXL/cxl_component: Addcxl_get_hb_cstate()
mem/cxl_type3: Add read and write functions for associated hostmem.
cxl/cxl-host: Add memops for CFMWS region.
RFC: softmmu/memory: Add ops tomemory_region_ram_init_from_file
i386/pc: Enable CXL fixed memory windows
tests/acpi: q35: Allow addition of a CXL test.
qtests/bios-tables-test: Add a test for CXL emulation.
tests/acpi: Add tables for CXL emulation.
hw/arm/virt: Basic CXL enablement onpci_expander_bridgeinstances
pxb-cxl
qtest/cxl: Add aarch64 virt test for CXL docs/cxl: Add initial Compute eXpress Link (CXL) documentation.MAINTAINERS | 7 + docs/system/device-emulation.rst | 1 + docs/system/devices/cxl.rst | 302 +++++++++++++++++ hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl-stub.c | 12 + hw/acpi/cxl.c | 231 +++++++++++++ hw/acpi/meson.build | 4 +- hw/arm/Kconfig | 1 + hw/arm/virt-acpi-build.c | 33 ++ hw/arm/virt.c | 40 ++- hw/core/machine.c | 28 ++ hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 284 ++++++++++++++++ hw/cxl/cxl-device-utils.c | 271 ++++++++++++++++ hw/cxl/cxl-host-stubs.c | 22 ++ hw/cxl/cxl-host.c | 262 +++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 483 ++++++++++++++++++++++++++++ hw/cxl/meson.build | 12 + hw/i386/acpi-build.c | 57 +++- hw/i386/pc.c | 57 +++- hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 353 ++++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 172 +++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci-host/gpex-acpi.c | 22 +- hw/pci/pci.c | 21 +- hw/pci/pcie_port.c | 25 ++ include/hw/acpi/cxl.h | 28 ++ include/hw/arm/virt.h | 1 + include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 51 +++ include/hw/cxl/cxl_component.h | 206 ++++++++++++ include/hw/cxl/cxl_device.h | 272 ++++++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 14 + include/hw/pci/pci_bridge.h | 20 ++ include/hw/pci/pci_bus.h | 7 + include/hw/pci/pci_ids.h | 1 + include/hw/pci/pcie_port.h | 2 + qapi/machine.json | 15 + qemu-options.hx | 37 +++ scripts/device-crash-test | 1 + softmmu/memory.c | 9 + softmmu/vl.c | 11 + tests/data/acpi/q35/CEDT.cxl | Bin 0 -> 184 bytes tests/data/acpi/q35/DSDT.cxl | Bin 0 -> 9627 bytes tests/qtest/bios-tables-test.c | 39 +++ tests/qtest/cxl-test.c | 181 +++++++++++ tests/qtest/meson.build | 5 + 55 files changed, 3996 insertions(+), 25 deletions(-) create mode 100644 docs/system/devices/cxl.rst create mode 100644 hw/acpi/cxl-stub.c create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-host-stubs.c create mode 100644 hw/cxl/cxl-host.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/q35/CEDT.cxl create mode 100644 tests/data/acpi/q35/DSDT.cxl create mode 100644 tests/qtest/cxl-test.c