CXl 2.0 emulation Support
v4 cover CXl 2.0 emulation Support
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.
Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.
Main changes:
* The CXL fixed memory windows are now instantiated via a -cxl-fixed-memory-window command line option. As they are host level entities, not associated with a particular hardware entity a top level parameter seems the most natural way to describe them. This is also much closer to how it works on a real host than the previous assignment of a physical address window to all components along the CXL path.
* Dynamic host memory physical address space allocation both for the CXL host bridge MMIO space and the CFMWS windows.
* Interleaving support (based loosely on Philippe Mathieu-Daudé's earlier work on an interleaved memory device). Note this is rudimentary and low performance but it may be sufficient for test purposes.
* Additional PCI and memory related utility functions needed for the interleaving.
* Various minor cleanup and increase in scope of tests.
* For now dropped the support for presenting CXL type 3 devices as memory devices in various QEMU interfaces.
* Dropped the patch letting UID be different from bus_nr. Whilst it may be a useful thing to have, we don't need it for this series and so should be handled separately.
I've called out patches with major changes by marking them as co-developed or introducing them as new patches. The original memory window code has been dropped
After discussions at plumbers and more recently on the mailing list it was clear that there was interest in getting emulation for CXL 2.0 upstream in QEMU. This version resolves many of the outstanding issues and enables the following features:
* Support on both x86/pc and ARM/virt with relevant ACPI tables generated in QEMU.
* Host bridge based on the existing PCI Expander Bridge PXB.
* CXL fixed memory windows, allowing host to describe interleaving across multiple CXL host bridges.
* pxb-cxl CXL host bridge support including MMIO region for control and HDM (Host manage device memory - basically interleaving / routing) decoder configuration.
* Basic CXL Root port support.
* CXL Type 3 device support with persistent memory regions (backed by hostmem backend).
* Pulled MAINTAINERS entry out to a separate patch and add myself as a co-maintainer at Ben's suggestion.
Big TODOs:
* Volatile memory devices (easy but it's more code so left for now).
* Switch support.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Main host bridge support (not a priority for me...)
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/
The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?
Example of new command line (with virt ITS patches ;)
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.
With Ben's CXL region patches (v3 shortly) plus fixes as discussed on list, Linux commands to bring up a 4 way interleave is:
cd /sys/bus/cxl/devices/
region=$(cat decoder0.1/create_region)
echo $region > decoder0.1/create_region
ls -lh
//Note the order of devices and adjust the following to make sure they //are in order across the 4 root ports. Easy to do in a tool, but //not easy to paste in a cover letter.
cd region0.1\:0
echo 4 > interleave_ways
echo mem2 > target0
echo mem3 > target1
echo mem0 > target2
echo mem1 > target3
echo $((1024<<20)) > size
echo 4096 > interleave_granularity
echo region0.1:0 > /sys/bus/cxl/drivers/cxl_region/bind
Tested with devmem2 and files with known content. Kernel tree was based on previous version of the region patches from Ben with various fixes. As Dan just posted an updated version next job on my list is to test that.
Thanks to Shameer for his help with reviewing the new stuff before posting.
I'll post a git tree shortly for any who prefer that to lots of emails ;)
Thanks,
Jonathan
Ben Widawsky (26):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory device utilities hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge) acpi/pci: Consolidate host bridge setup hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
hw/cxl/device: Add some trivial commands hw/cxl/device: Plumb real Label Storage Area (LSA) sizing hw/cxl/device: Implement get/set Label Storage Area (LSA) acpi/cxl: Introduce CFMWS structures in CEDT hw/cxl/component Add a dumb HDM decoder handler qtest/cxl: Add very basic sanity tests
Jonathan Cameron (16):
MAINTAINERS: Add entry for Compute Express Link Emulation tests/acpi: allow DSDT.viot table changes.
tests/acpi: Add update DSDT.viot
cxl: Machine level control on whether CXL support is enabled hw/cxl/component: Add utils for interleave parameter encoding/decoding hw/cxl/host: Add support for CXL Fixed Memory Windows. hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl pci/pcie_port: Add pci_find_port_by_pn() CXL/cxl_component: Add cxl_get_hb_cstate()
mem/cxl_type3: Add read and write functions for associated hostmem. cxl/cxl-host: Add memops for CFMWS region.
arm/virt: Allow virt/CEDT creation
hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
pxb-cxl
RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file i386/pc: Enable CXL fixed memory windows
qtest/acpi: Add reference CEDT tables.
MAINTAINERS | 7 + hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl.c | 232 +++++++++++++ hw/acpi/meson.build | 1 + hw/arm/Kconfig | 1 + hw/arm/virt-acpi-build.c | 30 ++ hw/arm/virt.c | 40 ++- hw/core/machine.c | 26 ++ hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 277 +++++++++++++++ hw/cxl/cxl-device-utils.c | 268 +++++++++++++++ hw/cxl/cxl-host-stubs.c | 22 ++ hw/cxl/cxl-host.c | 263 ++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 509 ++++++++++++++++++++++++++++ hw/cxl/meson.build | 9 + hw/i386/acpi-build.c | 97 +++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 57 +++- hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 353 +++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 179 +++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci-host/gpex-acpi.c | 22 +- hw/pci/pci.c | 21 +- hw/pci/pcie_port.c | 25 ++ hw/ppc/spapr.c | 1 + include/hw/acpi/cxl.h | 28 ++ include/hw/arm/virt.h | 1 + include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 51 +++ include/hw/cxl/cxl_component.h | 206 +++++++++++ include/hw/cxl/cxl_device.h | 266 +++++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 14 + include/hw/pci/pci_bridge.h | 20 ++ include/hw/pci/pci_bus.h | 7 + include/hw/pci/pci_ids.h | 1 + include/hw/pci/pcie_port.h | 2 + qapi/machine.json | 15 + qemu-options.hx | 37 ++ softmmu/memory.c | 9 + softmmu/vl.c | 11 + tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes tests/qtest/cxl-test.c | 151 +++++++++ tests/qtest/meson.build | 4 + 54 files changed, 3645 insertions(+), 40 deletions(-) create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-host-stubs.c create mode 100644 hw/cxl/cxl-host.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT create mode 100644 tests/data/acpi/virt/CEDT create mode 100644 tests/qtest/cxl-test.c
Signature · 1 line
2.32.0
PATCH v4 1/42 hw/pci/cxl: Add a CXL component type (interface)
From: Ben Widawsky <[email protected]>
A CXL component is a hardware entity that implements CXL component registers from the CXL 2.0 spec (8.2.3). Currently these represent 3 general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)
A CXL component can be conceptually thought of as a PCIe device with extra functionality when enumerated and enabled. For this reason, CXL does here, and will continue to add on to existing PCI code paths.
Host bridges will typically need to be handled specially and so they can implement this newly introduced interface or not. All other components should implement this interface. Implementing this interface allows the core PCI code to treat these devices as special where appropriate.
--- hw/pci/pci.c | 10 ++++++++++ include/hw/pci/pci.h | 8 ++++++++ 2 files changed, 18 insertions(+)
hw/pci/pci.c +10 -0
@@ -201,6 +201,11 @@ static const TypeInfo pci_bus_info = { .class_init = pci_bus_class_init, }; +static const TypeInfo cxl_interface_info = { + .name =INTERFACE_CXL_DEVICE , + .parent =TYPE_INTERFACE , +}; +static const TypeInfo pcie_interface_info = { .name =INTERFACE_PCIE_DEVICE , .parent =TYPE_INTERFACE ,
@@ -2128,6 +2133,10 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp) pci_dev->cap_present |=QEMU_PCI_CAP_EXPRESS ; } +if (object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE )) { + pci_dev->cap_present |=QEMU_PCIE_CAP_CXL ; + } + pci_dev =do_pci_register_device (pci_dev,object_get_typename (OBJECT (qdev)), pci_dev->devfn, errp);
@@ -2884,6 +2893,7 @@ static void pci_register_types(void)type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info);type_register_static (&conventional_pci_interface_info); +type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);type_register_static (&pci_device_type_info); }
include/hw/pci/pci.h +8 -0
@@ -195,6 +195,8 @@ enum { QEMU_PCIE_LNKSTA_DLLLA = (1 << QEMU_PCIE_LNKSTA_DLLLA_BITNR),#define QEMU_PCIE_EXTCAP_INIT_BITNR 9 QEMU_PCIE_EXTCAP_INIT = (1 << QEMU_PCIE_EXTCAP_INIT_BITNR), +#define QEMU_PCIE_CXL_BITNR 10 + QEMU_PCIE_CAP_CXL = (1 << QEMU_PCIE_CXL_BITNR), };#define TYPE_PCI_DEVICE "pci-device"
@@ -202,6 +204,12 @@ typedef struct PCIDeviceClass PCIDeviceClass;DECLARE_OBJ_CHECKERS (PCIDevice, PCIDeviceClass,PCI_DEVICE ,TYPE_PCI_DEVICE ) +/* +* Implemented by devices that can be plugged on CXL buses. In the spec, this is +* actually a "CXL Component, but we name it device to match the PCI naming. + */ +#define INTERFACE_CXL_DEVICE "cxl-device" +/* Implemented by devices that can be plugged on PCI Express buses */ #define INTERFACE_PCIE_DEVICE "pci-express-device"
Signature · 1 line
2.32.0
PATCH v4 2/42 hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The cxl_pci header manages bits associated with PCI, specifically the DVSEC and related fields. The cxl_component.h variant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] registers. Per spec, the registers are little endian.
None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
--- hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 212 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 3 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 16 +++ include/hw/cxl/cxl_component.h | 196 ++++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++ 8 files changed, 570 insertions(+)
hw/Kconfig +1 -0
@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfig
hw/cxl/Kconfig new file +3 -0
@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESS
hw/cxl/cxl-component-utils.c new file +212 -0
212 lines not on this page · raw message
hw/cxl/meson.build new file +3 -0
@@ -0,0 +1,3 @@ +softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', +))
hw/meson.build +1 -0
@@ -6,6 +6,7 @@ subdir('block') subdir('char') subdir('core') subdir('cpu') +subdir('cxl') subdir('display') subdir('dma') subdir('gpio')
include/hw/cxl/cxl.h new file +16 -0
@@ -0,0 +1,16 @@ +/* +* QEMU CXL Support +* +* Copyright (c) 2020 Intel +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#ifndef CXL_H +#define CXL_H + +#include "cxl_pci.h" +#include "cxl_component.h" + +#endif
include/hw/cxl/cxl_component.h new file +196 -0
196 lines not on this page · raw message
include/hw/cxl/cxl_pci.h new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v4 3/42 MAINTAINERS: Add entry for Compute Express Link Emulation
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The CXL emulation will be jointly maintained by Ben Widawsky and Jonathan Cameron. Broken out as a separate patch to improve visibility.
--- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
@@ -2519,6 +2519,13 @@ F: qapi/block*.json F: qapi/transaction.json T: git https://repo.or.cz/qemu/armbru.git block-next +Compute Express Link +M: Ben Widawsky <[email protected]> +M: Jonathan Cameron <[email protected]> +S: Supported +F: hw/cxl/ +F: include/hw/cxl/ + Dirty Bitmaps M: Eric Blake <[email protected]> M: Vladimir Sementsov-Ogievskiy <[email protected]>
Signature · 1 line
2.32.0
PATCH v4 4/42 hw/cxl/device: Introduce a CXL device (8.2.8)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.
This new device type is specifically catered towards the eventual implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.
--- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_device.h | 157 ++++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+)
include/hw/cxl/cxl.h +1 -0
@@ -12,5 +12,6 @@#include "cxl_pci.h" #include "cxl_component.h" +#include "cxl_device.h" #endif
include/hw/cxl/cxl_device.h new file +157 -0
157 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v4 5/42 hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This implements all device MMIO up to the first capability. That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
--- hw/cxl/cxl-device-utils.c | 105 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_device.h | 28 +++++++++- 3 files changed, 133 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c new file +105 -0
@@ -0,0 +1,105 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +/* +* Device registers have no restrictions per the spec, and so fall back to the +* default memory mapped register rules in 8.2: +* Software shall use CXL.io Memory Read and Write to access memory mapped +* register defined in this section. Unless otherwise specified, software +* shall restrict the accesses width based on the following: +* • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes +* quantity. +* • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 +* Bytes +* • The address shall be a multiple of the access width, e.g. when +* accessing a register as a 4 Byte quantity, the address shall be +* multiple of 4. +* • The accesses shall map to contiguous bytes.If these rules are not +* followed, the behavior is undefined +*/ + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +return cxl_dstate->caps_reg_state32[offset /4 ]; +} + +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL ,/* status register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL ,/* caps registers are read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =4 , + .max_access_size =4 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ +memory_region_init (&cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MMIO_SIZE )); + +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_CAPS_SIZE ); +memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate, +"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->device); +} + +static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =1 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); + +cxl_device_cap_init (cxl_dstate,DEVICE ,1 ); +device_reg_init_common (cxl_dstate); +}
hw/cxl/meson.build +1 -0
@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
include/hw/cxl/cxl_device.h +27 -1
@@ -63,6 +63,8 @@#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10/* Figure 138 */ #define CXL_DEVICE_CAP_REG_SIZE 0x10/* 8.2.8.2 */ #define CXL_DEVICE_CAPS_MAX 4/* 8.2.8.2.1 + 8.2.8.5 */ +#define CXL_CAPS_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE * (CXL_DEVICE_CAPS_MAX + 1))/* +1 for header */ #define CXL_DEVICE_REGISTERS_OFFSET 0x80/* Read comment above */ #define CXL_DEVICE_REGISTERS_LENGTH 0x8/* 8.2.8.3.1 */
@@ -75,11 +77,18 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) +#define CXL_MMIO_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ + CXL_MAILBOX_REGISTERS_LENGTH) + typedef struct cxl_device_state { MemoryRegion device_registers; /* mmio for device capabilities array - 8.2.8.2 */ - MemoryRegion caps; + struct { + MemoryRegion caps; + uint32_t caps_reg_state32[CXL_CAPS_SIZE / 4]; + }; /* mmio for the device status registers 8.2.8.3 */ MemoryRegion device;
@@ -128,6 +137,23 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +#define cxl_device_cap_init (dstate, reg, cap_id) \ + do { \ + uint32_t *cap_hdrs = dstate->caps_reg_state32; \ + int which = R_CXL_DEV_##reg##_CAP_HDR0; \ + cap_hdrs[which] = \ + FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, \ + CAP_ID, cap_id); \ + cap_hdrs[which] = FIELD_DP32( \ + cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1); \ + cap_hdrs[which + 1] = \ + FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1, \ + CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET); \ + cap_hdrs[which + 2] = \ + FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2, \ + CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH); \ + } while (0) + REG32(CXL_DEV_MAILBOX_CAP, 0) FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)FIELD (CXL_DEV_MAILBOX_CAP ,INT_CAP ,5 ,1 )
Signature · 1 line
2.32.0
PATCH v4 6/42 hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in cxl-mailbox-utils.c, and the hardware.
No commands are implemented yet.
--- hw/cxl/cxl-device-utils.c | 129 ++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 201 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 29 +++++- 5 files changed, 358 insertions(+), 5 deletions(-)
hw/cxl/cxl-device-utils.c +128 -1
@@ -40,6 +40,115 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return 0 ; } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +switch (size) { +case 1 : +return cxl_dstate->mbox_reg_state[offset]; +case 2 : +return cxl_dstate->mbox_reg_state16[offset /2 ]; +case 4 : +return cxl_dstate->mbox_reg_state32[offset /4 ]; +case 8 : +return cxl_dstate->mbox_reg_state64[offset /8 ]; +default : +g_assert_not_reached (); + } +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + + reg_state[offset /4 ] = value; +} + +static void mailbox_mem_writeq (uint64_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + + + reg_state[offset /8 ] = value; +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) { +memcpy (cxl_dstate->mbox_reg_state + offset, &value, size); +return ; + } + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +mailbox_mem_writeq (cxl_dstate->mbox_reg_state64, offset, value); +break ; +default : +g_assert_not_reached (); + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,/* status register is read only */
@@ -80,20 +189,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_CAPS_SIZE);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -102,4 +224,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
hw/cxl/cxl-mailbox-utils.c new file +201 -0
201 lines not on this page · raw message
hw/cxl/meson.build +1 -0
@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))
include/hw/cxl/cxl.h +3 -0
@@ -14,4 +14,7 @@#include "cxl_component.h" #include "cxl_device.h" +#define CXL_COMPONENT_REG_BAR_IDX 0 +#define CXL_DEVICE_REG_BAR_IDX 2 +#endif
include/hw/cxl/cxl_device.h +25 -4
@@ -85,16 +85,28 @@ typedef struct cxl_device_state {MemoryRegion device_registers;/* mmio for device capabilities array - 8.2.8.2 */ +MemoryRegion device;struct {MemoryRegion caps;uint32_t caps_reg_state32[CXL_CAPS_SIZE /4 ]; }; -/* mmio for the device status registers 8.2.8.3 */ -MemoryRegion device; -/* mmio for the mailbox registers 8.2.8.4 */ -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH /2 ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH /4 ]; +uint64_t mbox_reg_state64[CXL_MAILBOX_REGISTERS_LENGTH /8 ]; + }; +struct cel_log { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };/* memory region for persistent memory, HDM */ uint64_t pmem_size;
@@ -137,6 +149,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
@@ -165,6 +180,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )
Signature · 1 line
2.32.0
PATCH v4 7/42 hw/cxl/device: Add memory device utilities
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Memory devices implement extra capabilities on top of CXL devices. This adds support for that.
A large part of memory devices is the mailbox/command interface. All of the mailbox handling is done in the mailbox-utils library. Longer term, new CXL devices that are being emulated may want to handle commands differently, and therefore would need a mechanism to opt in/out of the specific generic handlers. As such, this is considered sufficient for now, but may need more depth in the future.
--- hw/cxl/cxl-device-utils.c | 38 ++++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 23 +++++++++++++++++++--- 2 files changed, 57 insertions(+), 4 deletions(-)
hw/cxl/cxl-device-utils.c +37 -1
@@ -134,6 +134,31 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,cxl_process_mailbox (cxl_dstate); } +static uint64_t mdev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +uint64_t retval =0 ; + + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MEDIA_STATUS ,1 ); + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MBOX_READY ,1 ); + +return retval; +} + +static const MemoryRegionOps mdev_ops = { + .read = mdev_reg_read, + .write =NULL ,/* memory device register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =8 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps mailbox_ops = { .read = mailbox_reg_read, .write = mailbox_reg_write,
@@ -191,6 +216,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "device-status", CXL_DEVICE_REGISTERS_LENGTH);memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->memory_device, obj, &mdev_ops, + cxl_dstate,"memory device caps" , +CXL_MEMORY_DEVICE_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);
@@ -200,6 +228,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)memory_region_add_subregion (&cxl_dstate->device_registers,CXL_MAILBOX_REGISTERS_OFFSET , &cxl_dstate->mailbox); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MEMORY_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->memory_device); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { }
@@ -212,10 +243,12 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; } +static void memdev_reg_init_common (CXLDeviceState *cxl_dstate) { } +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =2 ; +const int cap_count =3 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -228,5 +261,8 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,MAILBOX ,2 );mailbox_reg_init_common (cxl_dstate); +cxl_device_cap_init (cxl_dstate,MEMORY_DEVICE ,0x4000 ); +memdev_reg_init_common (cxl_dstate); +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
include/hw/cxl/cxl_device.h +20 -3
@@ -77,15 +77,21 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) -#define CXL_MMIO_SIZE \ - (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ - CXL_MAILBOX_REGISTERS_LENGTH) + +#define CXL_MEMORY_DEVICE_REGISTERS_OFFSET \ + (CXL_MAILBOX_REGISTERS_OFFSET + CXL_MAILBOX_REGISTERS_LENGTH) +#define CXL_MEMORY_DEVICE_REGISTERS_LENGTH 0x8 + +#define CXL_MMIO_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ + CXL_MAILBOX_REGISTERS_LENGTH + CXL_MEMORY_DEVICE_REGISTERS_LENGTH) typedef struct cxl_device_state { MemoryRegion device_registers; /* mmio for device capabilities array - 8.2.8.2 */ MemoryRegion device; + MemoryRegion memory_device; struct { MemoryRegion caps; uint32_t caps_reg_state32[CXL_CAPS_SIZE / 4];
@@ -148,6 +154,9 @@ REG32(CXL_DEV_CAP_ARRAY2, 4) /* We're going to pretend it's 64b */ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MEMORY_DEVICE , +CXL_DEVICE_CAP_HDR1_OFFSET + + CXL_DEVICE_CAP_REG_SIZE * 2)int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate);void cxl_process_mailbox (CXLDeviceState *cxl_dstate);
@@ -201,4 +210,12 @@ REG32(CXL_DEV_BG_CMD_STS, 0x18)REG32 (CXL_DEV_CMD_PAYLOAD ,0x20 ) +/* XXX: actually a 64b registers */ +REG32(CXL_MEM_DEV_STS, 0) + FIELD(CXL_MEM_DEV_STS, FATAL, 0, 1) + FIELD(CXL_MEM_DEV_STS, FW_HALT, 1, 1) + FIELD(CXL_MEM_DEV_STS, MEDIA_STATUS, 2, 2) + FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) + FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +#endif
Signature · 1 line
2.32.0
PATCH v4 8/42 hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Using the previously implemented stubbed helpers, it is now possible to easily add the missing, required commands to the implementation.
--- hw/cxl/cxl-mailbox-utils.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +22 -1
@@ -37,6 +37,14 @@* a register interface that already deals with it. */ +enum { + EVENTS = 0x01, + #define GET_RECORDS 0x0 +#define CLEAR_RECORDS 0x1 +#define GET_INTERRUPT_POLICY 0x2 +#define SET_INTERRUPT_POLICY 0x3 +}; +/* 8.2.8.4.5.1 Command Return Codes */ typedef enum { CXL_MBOX_SUCCESS = 0x0,
@@ -105,10 +113,23 @@ struct cxl_cmd {return CXL_MBOX_SUCCESS ; \ } +define_mailbox_handler_zeroed (EVENTS_GET_RECORDS ,0x20 ); +define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS ); +define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 ); +define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); + +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_LOG_CHANGE (1 << 4) +#define CXL_CMD (s, c, in, cel_effect) \ [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } -static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = { +CXL_CMD (EVENTS ,GET_RECORDS ,1 ,0 ), +CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ), +CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ), +CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +};#undef CXL_CMD
Signature · 1 line
2.32.0
PATCH v4 9/42 hw/cxl/device: Timestamp implementation (8.2.9.3)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.
--- hw/cxl/cxl-mailbox-utils.c | 53 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 59 insertions(+)
hw/cxl/cxl-mailbox-utils.c +53 -0
@@ -43,6 +43,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (TIMESTAMP_GET ); +declare_mailbox_handler (TIMESTAMP_SET );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)#define CXL_CMD (s, c, in, cel_effect) \
@@ -129,10 +135,57 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (TIMESTAMP ,GET ,0 ,0 ), + CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE), };#undef CXL_CMD +/* +* 8.2.9.3.1 +*/ +define_mailbox_handler (TIMESTAMP_GET ) +{ +struct timespec ts; +uint64_t delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done; + } + +/* First find the delta from the last time the host set the time. */ +clock_gettime (CLOCK_REALTIME , &ts); + delta = (ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec) - + cxl_dstate->timestamp.last_set; + +/* Then adjust the actual time */ +stq_le_p (cmd->payload, cxl_dstate->timestamp.host_set + delta); + +done: + *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* +* 8.2.9.3.2 +*/ +define_mailbox_handler (TIMESTAMP_SET ) +{ +struct timespec ts; + +clock_gettime (CLOCK_REALTIME , &ts); + + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set = + ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec; + + cxl_dstate->timestamp.host_set =le64_to_cpu (*(uint64_t *)cmd->payload); + + *len =0 ; +return CXL_MBOX_SUCCESS ; +} +QemuUUID cel_uuid;void cxl_process_mailbox (CXLDeviceState *cxl_dstate)
include/hw/cxl/cxl_device.h +6 -0
@@ -114,6 +114,12 @@ typedef struct cxl_device_state {size_t cel_size; }; +struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +/* memory region for persistent memory, HDM */ uint64_t pmem_size; } CXLDeviceState;
Signature · 1 line
2.32.0
PATCH v4 10/42 hw/cxl/device: Add log commands (8.2.9.4) + CEL
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
--- hw/cxl/cxl-mailbox-utils.c | 67 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
hw/cxl/cxl-mailbox-utils.c +67 -0
@@ -46,6 +46,9 @@ enum { TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1 + LOGS = 0x04, +#define GET_SUPPORTED 0x0 +#define GET_LOG 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -122,6 +125,8 @@ define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY );declare_mailbox_handler (TIMESTAMP_GET );declare_mailbox_handler (TIMESTAMP_SET ); +declare_mailbox_handler (LOGS_GET_SUPPORTED ); +declare_mailbox_handler (LOGS_GET_LOG );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -137,6 +142,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ),CXL_CMD (TIMESTAMP ,GET ,0 ,0 ),CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ), +CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ), + CXL_CMD(LOGS, GET_LOG, 0x18, 0), };#undef CXL_CMD
@@ -188,6 +195,66 @@ define_mailbox_handler(TIMESTAMP_SET)QemuUUID cel_uuid; +/* 8.2.9.4.1 */ +define_mailbox_handler (LOGS_GET_SUPPORTED ) +{ +struct { +uint16_t entries; +uint8_t rsvd[6 ]; +struct { +QemuUUID uuid; +uint32_t size; + } log_entries[1 ]; + } __attribute__((packed)) *supported_logs = (void *)cmd->payload; +_Static_assert (sizeof (*supported_logs) ==0x1c ,"Bad supported log size" ); + + supported_logs->entries =1 ; + supported_logs->log_entries[0 ].uuid = cel_uuid; + supported_logs->log_entries[0 ].size =4 * cxl_dstate->cel_size; + + *len =sizeof (*supported_logs); +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.4.2 */ +define_mailbox_handler (LOGS_GET_LOG ) +{ +struct { +QemuUUID uuid; +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (16 ))) *get_log = (void *)cmd->payload; + +/* +* 8.2.9.4.2 +* The device shall return Invalid Parameter if the Offset or Length +* fields attempt to access beyond the size of the log as reported by Get +* Supported Logs. +* +* XXX: Spec is wrong, "Invalid Parameter" isn't a thing. +* XXX: Spec doesn't address incorrect UUID incorrectness. +* +* The CEL buffer is large enough to fit all commands in the emulation, so +* the only possible failure would be if the mailbox itself isn't big +* enough. +*/ +if (get_log->offset + get_log->length > cxl_dstate->payload_size) { +return CXL_MBOX_INVALID_INPUT ; + } + +if (!qemu_uuid_is_equal (&get_log->uuid, &cel_uuid)) { +return CXL_MBOX_UNSUPPORTED ; + } + +/* Store off everything to local variables so we can wipe out the payload */ + *len = get_log->length; + +memmove (cmd->payload, cxl_dstate->cel_log + get_log->offset, + get_log->length); + +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
2.32.0
PATCH v4 11/42 hw/pxb: Use a type for realizing expanders
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This opens up the possibility for more types of expanders (other than PCI and PCIe). We'll need this to create a CXL expander.
--- hw/pci-bridge/pci_expander_bridge.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +7 -4
@@ -24,6 +24,8 @@#include "hw/boards.h" #include "qom/object.h" +enum BusType {PCI ,PCIE }; +#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ; DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
@@ -221,7 +223,8 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)0 ; } -static void pxb_dev_realize_common (PCIDevice *dev,bool pcie,Error **errp) +static void pxb_dev_realize_common (PCIDevice *dev,enum BusType type, +Error **errp) {PXBDev *pxb =convert_to_pxb (dev);DeviceState *ds, *bds =NULL ;
@@ -246,7 +249,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp) } ds =qdev_new (TYPE_PXB_HOST ); -if (pcie) { +if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS );
@@ -295,7 +298,7 @@ static void pxb_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, false, errp); +pxb_dev_realize_common (dev,PCI , errp); }static void pxb_dev_exitfn (PCIDevice *pci_dev)
@@ -348,7 +351,7 @@ static void pxb_pcie_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, true, errp); +pxb_dev_realize_common (dev,PCIE , errp); }static void pxb_pcie_dev_class_init (ObjectClass *klass,void *data)
Signature · 1 line
2.32.0
PATCH v4 12/42 hw/pci/cxl: Create a CXL bus type
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The easiest way to differentiate a CXL bus, and a PCIE bus is using a flag. A CXL bus, in hardware, is backward compatible with PCIE, and therefore the code tries pretty hard to keep them in sync as much as possible.
The other way to implement this would be to try to cast the bus to the correct type. This is less code and useful for debugging via simply looking at the flags.
--- hw/pci-bridge/pci_expander_bridge.c | 9 ++++++++- include/hw/pci/pci_bus.h | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +8 -1
@@ -24,7 +24,7 @@#include "hw/boards.h" #include "qom/object.h" -enum BusType {PCI ,PCIE }; +enum BusType {PCI ,PCIE ,CXL };#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ;
@@ -35,6 +35,10 @@ DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,DECLARE_INSTANCE_CHECKER (PXBBus,PXB_PCIE_BUS , TYPE_PXB_PCIE_BUS) +#define TYPE_PXB_CXL_BUS "pxb-cxl-bus" +DECLARE_INSTANCE_CHECKER (PXBBus,PXB_CXL_BUS , + TYPE_PXB_CXL_BUS) +struct PXBBus {/*< private >*/ PCIBus parent_obj;
@@ -251,6 +255,9 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, ds =qdev_new (TYPE_PXB_HOST );if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); + }else if (type ==CXL ) { + bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); + bus->flags |=PCI_BUS_CXL ; }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
include/hw/pci/pci_bus.h +7 -0
@@ -24,6 +24,8 @@ enum PCIBusFlags {PCI_BUS_IS_ROOT =0x0001 ,/* PCIe extended configuration space is accessible on this bus */ PCI_BUS_EXTENDED_CONFIG_SPACE =0x0002 , +/* This is a CXL Type BUS */ + PCI_BUS_CXL = 0x0004, }; struct PCIBus {
@@ -53,6 +55,11 @@ struct PCIBus {Notifier machine_done; }; +static inline bool pci_bus_is_cxl (PCIBus *bus) +{ +return !!(bus->flags &PCI_BUS_CXL ); +} +static inline bool pci_bus_is_root (PCIBus *bus) {return !!(bus->flags &PCI_BUS_IS_ROOT );
Signature · 1 line
2.32.0
PATCH v4 13/42 hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1
A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in _SB named ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.
--- hw/pci-bridge/pci_expander_bridge.c | 98 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 ++ 3 files changed, 109 insertions(+), 2 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +96 -2
@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h"
@@ -56,6 +57,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +struct PXBDev {/*< private >*/ PCIDevice parent_obj;
@@ -66,8 +71,19 @@ struct PXBDev {bool bypass_iommu; }; +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }
@@ -76,6 +92,9 @@ static GList *pxb_dev_list;#define TYPE_PXB_HOST "pxb-host" +#define TYPE_PXB_CXL_HOST "pxb-cxl-host" +#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);
@@ -112,11 +131,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;
@@ -218,6 +246,16 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); } +static void pxb_dev_reset (DeviceState *dev) +{ +CXLHost *cxl =PXB_CXL_HOST (dev); +CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT ,8 ); +} +static gint pxb_compare (gconstpointer a,gconstpointer b) {const PXBDev *pxb_a = a, *pxb_b = b;
@@ -290,6 +328,11 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,pci_config_set_class (dev->config,PCI_CLASS_BRIDGE_HOST ); pxb_dev_list =g_list_insert_sorted (pxb_dev_list, pxb, pxb_compare); + +if (type ==CXL ) { +pxb_dev_reset (ds); + } +return ; err_register_bus:
@@ -338,6 +381,12 @@ static void pxb_dev_class_init(ObjectClass *klass, void *data)device_class_set_props (dc, pxb_dev_properties); dc->hotpluggable = false;set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* +* Reset doesn't seem to actually be called, but maybe it will in the +* future? +*/ + dc->reset = pxb_dev_reset; }static const TypeInfo pxb_dev_info = {
@@ -389,13 +438,58 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)
hw/pci/pci.c +7 -0
@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);
@@ -2892,6 +2898,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);
include/hw/pci/pci.h +6 -0
@@ -409,6 +409,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"typedef void (*pci_bus_dev_fn )(PCIBus *b,PCIDevice *d,void *opaque);typedef void (*pci_bus_fn )(PCIBus *b,void *opaque);
@@ -768,6 +769,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
Signature · 1 line
2.32.0
PATCH v4 14/42 tests/acpi: allow DSDT.viot table changes.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The next patch unifies some of the PCI host bridge DSDT generation code and results in some minor changes to this file.
--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)
tests/qtest/bios-tables-test-allowed-diff.h +1 -0
@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.viot",
Signature · 1 line
2.32.0
PATCH v4 15/42 acpi/pci: Consolidate host bridge setup
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This cleanup will make it easier to add support for CXL to the mix.
--- hw/i386/acpi-build.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
hw/i386/acpi-build.c +21 -17
@@ -1398,6 +1398,23 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } +enum {PCI ,PCIE }; +static void init_pci_acpi (Aml *dev,int uid,int type,bool native_pcie_hp) +{ +if (type ==PCI ) { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +/* Expander bridges do not have ACPI PCI Hot-plug enabled */ +aml_append (dev,build_q35_osc_method (native_pcie_hp)); + } +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1429,9 +1446,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,if (misc->is_piix4) { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); -aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (pcmc->pci_root_uid))); +init_pci_acpi (dev, pcmc->pci_root_uid,PCI , false);aml_append (sb_scope, dev);aml_append (dsdt, sb_scope);
@@ -1447,11 +1462,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, }else { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); -aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (pcmc->pci_root_uid))); -aml_append (dev,build_q35_osc_method (!pm->pcihp_bridge_en)); +init_pci_acpi (dev, pcmc->pci_root_uid,PCIE , !pm->pcihp_bridge_en);aml_append (sb_scope, dev);if (mcfg_valid) {aml_append (sb_scope,build_q35_dram_controller (&mcfg));
@@ -1562,17 +1573,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" ); dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -if (pci_bus_is_express (bus)) { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); -/* Expander bridges do not have ACPI PCI Hot-plug enabled */ -aml_append (dev,build_q35_osc_method (true)); - }else { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); - } +init_pci_acpi (dev, bus_num, +pci_bus_is_express (bus) ?PCIE :PCI , true);if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
Signature · 1 line
2.32.0
PATCH v4 16/42 tests/acpi: Add update DSDT.viot
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The consolidation of DSDT AML generation for PCI host bridges lead to some minor ordering changes and the addition of _ADR with a default of 0 for those case that didn't already have it. Only DSDT.viot test is affected.
Changes all similar to:
Scope (\_SB)
{
Device (PC30)
{
- Name (_UID, 0x30) // _UID: Unique ID
Name (_BBN, 0x30) // _BBN: BIOS Bus Number
Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID
Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID
+ Name (_ADR, Zero) // _ADR: Address
+ Name (_UID, 0x30) // _UID: Unique ID Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities
--- tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes tests/qtest/bios-tables-test-allowed-diff.h | 1 - 2 files changed, 1 deletion(-)
tests/data/acpi/q35/DSDT.viot binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -1
@@ -1,2 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/q35/DSDT.viot",
Signature · 1 line
2.32.0
PATCH v4 17/42 cxl: Machine level control on whether CXL support is enabled
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
There are going to be some potential overheads to CXL enablement, for example the host bridge region reserved in memory maps. Add a machine level control so that CXL is disabled by default.
--- hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++++++++++++++++++++++++++ hw/i386/microvm.c | 1 + hw/i386/pc.c | 1 + hw/ppc/spapr.c | 1 + include/hw/boards.h | 2 ++ include/hw/cxl/cxl.h | 4 ++++ 7 files changed, 36 insertions(+)
hw/arm/virt.c +1 -0
@@ -2856,6 +2856,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->unplug = virt_machine_device_unplug_cb; mc->nvdimm_supported = true; mc->smp_props.clusters_supported = true; + mc->cxl_supported = false; mc->auto_enable_numa_with_memhp = true; mc->auto_enable_numa_with_memdev = true; mc->default_ram_id ="mach-virt.ram" ;
hw/core/machine.c +26 -0
@@ -31,6 +31,7 @@#include "sysemu/qtest.h" #include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "migration/global_state.h" #include "migration/vmstate.h" #include "exec/confidential-guest-support.h"
@@ -545,6 +546,20 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value, nvdimms_state->persistence_string =g_strdup (value); } +static bool machine_get_cxl (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->cxl_devices_state->is_enabled; +} + +static void machine_set_cxl (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->cxl_devices_state->is_enabled = value; +} +void machine_class_allow_dynamic_sysbus_dev (MachineClass *mc,const char *type) {QAPI_LIST_PREPEND (mc->allowed_dynamic_sysbus_devices,g_strdup (type));
@@ -922,6 +937,16 @@ static void machine_initfn(Object *obj) "Valid values are cpu, mem-ctrl"); } +if (mc->cxl_supported) { +Object *obj =OBJECT (ms); + + ms->cxl_devices_state =g_new0 (CXLState,1 ); +object_property_add_bool (obj,"cxl" , machine_get_cxl, machine_set_cxl); +object_property_set_description (obj,"cxl" , +"Set on/off to enable/disable " +"CXL instantiation" ); + } +if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) { ms->numa_state =g_new0 (NumaState,1 ); object_property_add_bool(obj, "hmat",
@@ -956,6 +981,7 @@ static void machine_finalize(Object *obj)g_free (ms->device_memory);g_free (ms->nvdimms_state);g_free (ms->numa_state); +g_free (ms->cxl_devices_state); }bool machine_usb (MachineState *machine)
hw/i386/microvm.c +1 -0
@@ -685,6 +685,7 @@ static void microvm_class_init(ObjectClass *oc, void *data) mc->auto_enable_numa_with_memdev = false; mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = false; + mc->cxl_supported = false; mc->default_ram_id ="microvm.ram" ;/* Avoid relying too much on kernel components */
hw/i386/pc.c +1 -0
@@ -1739,6 +1739,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = true; mc->smp_props.dies_supported = true; + mc->cxl_supported = true; mc->default_ram_id ="pc.ram" ; object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
hw/ppc/spapr.c +1 -0
@@ -4569,6 +4569,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =POWERPC_CPU_TYPE_NAME ("power9_v2.0" ); mc->has_hotpluggable_cpus = true; mc->nvdimm_supported = true; + mc->cxl_supported = false; smc->resize_hpt_default =SPAPR_RESIZE_HPT_ENABLED ; fwc->get_dev_path = spapr_get_fw_dev_path; nc->nmi_monitor_handler = spapr_nmi;
include/hw/boards.h +2 -0
@@ -269,6 +269,7 @@ struct MachineClass {bool ignore_boot_device_suffixes;bool smbus_no_migration_support;bool nvdimm_supported; +bool cxl_supported;bool numa_mem_supported;bool auto_enable_numa;SMPCompatProps smp_props;
@@ -360,6 +361,7 @@ struct MachineState {CPUArchIdList *possible_cpus;CpuTopology smp;struct NVDIMMState *nvdimms_state; +struct CXLState *cxl_devices_state;struct NumaState *numa_state; };
include/hw/cxl/cxl.h +4 -0
@@ -17,4 +17,8 @@#define CXL_COMPONENT_REG_BAR_IDX 0#define CXL_DEVICE_REG_BAR_IDX 2 +typedef struct CXLState { +bool is_enabled; +}CXLState ; +#endif
Signature · 1 line
2.32.0
PATCH v4 18/42 hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL host bridges themselves may have MMIO. Since host bridges don't have a BAR they are treated as special for MMIO. This patch includes i386/pc support.
--- hw/i386/acpi-build.c | 26 +++++++++++--- hw/i386/pc.c | 27 ++++++++++++++- hw/pci-bridge/pci_expander_bridge.c | 53 ++++++++++++++++++++++++++++- include/hw/cxl/cxl.h | 4 +++ 4 files changed, 104 insertions(+), 6 deletions(-)
hw/i386/acpi-build.c +22 -4
@@ -28,6 +28,7 @@#include "qemu/bitmap.h" #include "qemu/error-report.h" #include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" #include "hw/core/cpu.h" #include "target/i386/cpu.h" #include "hw/misc/pvpanic.h"
@@ -1398,7 +1399,7 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } -enum {PCI ,PCIE }; +enum {PCI ,PCIE ,CXL };static void init_pci_acpi (Aml *dev,int uid,int type,bool native_pcie_hp) {if (type ==PCI ) {
@@ -1561,22 +1562,30 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +int32_t uid = bus_num;/* TODO: Explicit uid */ +int type;/* look only for expander root buses */ if (!pci_bus_is_root (bus)) {continue ; } + type =pci_bus_is_cxl (bus) ?CXL : +pci_bus_is_express (bus) ?PCIE :PCI ; +if (bus_num < root_bus_limit) { root_bus_limit = bus_num -1 ; } scope =aml_scope ("\\_SB" ); - dev =aml_device ("PC%.02X" , bus_num); +if (type ==CXL ) { + dev =aml_device ("CL%.02X" , uid); + }else { + dev =aml_device ("PC%.02X" , bus_num); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -init_pci_acpi (dev, bus_num, -pci_bus_is_express (bus) ?PCIE :PCI , true); +init_pci_acpi (dev, uid, type, true);if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
@@ -1588,6 +1597,15 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (dev,aml_name_decl ("_CRS" , crs));aml_append (scope, dev);aml_append (dsdt, scope); + +/* Handle the ranges for the PXB expanders */ +if (type ==CXL ) { +MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +uint64_t base = mr->addr; + +crs_range_insert (crs_range_set.mem_ranges, base, + base +memory_region_size (mr) -1 ); + } } }
hw/i386/pc.c +26 -1
@@ -75,6 +75,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/qapi-visit-machine.h"
@@ -815,6 +816,7 @@ void pc_memory_init(PCMachineState *pcms,MachineClass *mc =MACHINE_GET_CLASS (machine);PCMachineClass *pcmc =PC_MACHINE_GET_CLASS (pcms);X86MachineState *x86ms =X86_MACHINE (pcms); +hwaddr cxl_base;assert (machine->ram_size == x86ms->below_4g_mem_size + x86ms->above_4g_mem_size);
@@ -904,6 +906,26 @@ void pc_memory_init(PCMachineState *pcms, &machine->device_memory->mr); } +if (machine->cxl_devices_state->is_enabled) { +MemoryRegion *mr = &machine->cxl_devices_state->host_mr; +hwaddr cxl_size = MiB; + +if (pcmc->has_reserved_memory && machine->device_memory->base) { + cxl_base = machine->device_memory->base; +if (!pcmc->broken_reserved_end) { + cxl_base +=memory_region_size (&machine->device_memory->mr); + } + }else if (pcms->sgx_epc.size !=0 ) { + cxl_base =sgx_epc_above_4g_end (&pcms->sgx_epc); + }else { + cxl_base =0x100000000ULL + x86ms->above_4g_mem_size; + } + +e820_add_entry (cxl_base, cxl_size,E820_RESERVED ); +memory_region_init (mr,OBJECT (machine),"cxl_host_reg" , cxl_size); +memory_region_add_subregion (system_memory, cxl_base, mr); + } +/* Initialize PC system firmware */ pc_system_firmware_init (pcms, rom_memory);
@@ -964,7 +986,10 @@ uint64_t pc_pci_hole64_start(void)X86MachineState *x86ms =X86_MACHINE (pcms);uint64_t hole64_start =0 ; -if (pcmc->has_reserved_memory && ms->device_memory->base) { +if (ms->cxl_devices_state->host_mr.addr) { + hole64_start = ms->cxl_devices_state->host_mr.addr + +memory_region_size (&ms->cxl_devices_state->host_mr); + }else if (pcmc->has_reserved_memory && ms->device_memory->base) { hole64_start = ms->device_memory->base;if (!pcmc->broken_reserved_end) { hole64_start +=memory_region_size (&ms->device_memory->mr);
hw/pci-bridge/pci_expander_bridge.c +52 -1
71 lines not on this page · raw message
include/hw/cxl/cxl.h +4 -0
@@ -17,8 +17,12 @@#define CXL_COMPONENT_REG_BAR_IDX 0#define CXL_DEVICE_REG_BAR_IDX 2 +#define CXL_WINDOW_MAX 10 +typedef struct CXLState {bool is_enabled; +MemoryRegion host_mr; +unsigned int next_mr_idx; }CXLState ;#endif
Signature · 1 line
2.32.0
PATCH v4 19/42 hw/cxl/rp: Add a root port
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4
Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.
--- hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 4 +- 5 files changed, 245 insertions(+), 2 deletions(-)
hw/pci-bridge/Kconfig +5 -0
@@ -27,3 +27,8 @@ config DEC_PCI config SIMBA bool + +config CXL + bool + default y if PCI_EXPRESS && PXB + depends on PCI_EXPRESS && MSI_NONBROKEN && PXB
hw/pci-bridge/cxl_root_port.c new file +231 -0
231 lines not on this page · raw message
hw/pci-bridge/meson.build +1 -0
@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c')) pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c')) pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c')) pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c')) +pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c')) # NewWorld PowerMac pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))
hw/pci-bridge/pcie_root_port.c +5 -1
@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)int rc;pci_config_set_interrupt_pin (d->config,1 ); -pci_bridge_initfn (d,TYPE_PCIE_BUS ); +if (d->cap_present &QEMU_PCIE_CAP_CXL ) { +pci_bridge_initfn (d,TYPE_CXL_BUS ); + }else { +pci_bridge_initfn (d,TYPE_PCIE_BUS ); + }pcie_port_init_reg (d); rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,
hw/pci/pci.c +3 -1
@@ -2708,7 +2708,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)object_class_dynamic_cast (klass,INTERFACE_CONVENTIONAL_PCI_DEVICE );ObjectClass *pcie =object_class_dynamic_cast (klass,INTERFACE_PCIE_DEVICE ); -assert (conventional || pcie); +ObjectClass *cxl = +object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE ); +assert (conventional || pcie || cxl); } }
Signature · 1 line
2.32.0
PATCH v4 20/42 hw/cxl/device: Add a memory device (8.2.8.5)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Although the memory device is configured like a normal PCIe device, the memory traffic is on an entirely separate bus conceptually (using the same physical wires as PCIe, but different protocol).
Once the CXL topology is fully configure and address decoders committed, the guest physical address for the memory device is part of a larger window which is owned by the platform. The creation of these windows is later in this series.
The following example will create a 256M device in a 512M window:
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
Note: Dropped PCDIMM info interfaces for now. They can be added if appropriate at a later date.
--- hw/cxl/cxl-mailbox-utils.c | 44 ++++++++++ hw/mem/Kconfig | 5 ++ hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_pci.h | 22 +++++ include/hw/pci/pci_ids.h | 1 + 7 files changed, 244 insertions(+)
hw/cxl/cxl-mailbox-utils.c +44 -0
@@ -49,6 +49,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG ); +declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ), + CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0), };#undef CXL_CMD
@@ -255,6 +259,46 @@ define_mailbox_handler(LOGS_GET_LOG)return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +define_mailbox_handler (IDENTIFY_MEMORY_DEVICE ) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + + id->total_capacity = size / (256 <<20 ); + id->persistent_capacity = size / (256 <<20 ); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
hw/mem/Kconfig +5 -0
@@ -11,3 +11,8 @@ config NVDIMM config SPARSE_MEM bool + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICE
hw/mem/cxl_type3.c new file +170 -0
170 lines not on this page · raw message
hw/mem/meson.build +1 -0
@@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
include/hw/cxl/cxl.h +1 -0
@@ -17,6 +17,7 @@#define CXL_COMPONENT_REG_BAR_IDX 0#define CXL_DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3"#define CXL_WINDOW_MAX 10 typedef struct CXLState {
include/hw/cxl/cxl_pci.h +22 -0
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct cxl_dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct cxl_dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct cxl_dvsec_port_extensions {struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06
Signature · 1 line
2.32.0
PATCH v4 21/42 hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
A device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. Whilst the implementation currently does no validity checks on the encoder set up, future work will add sanity checking specific to the type of cxl component.
--- hw/mem/cxl_type3.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
hw/mem/cxl_type3.c +54 -0
@@ -61,6 +61,56 @@ static void build_dvsecs(CXLType3Dev *ct3d) REG_LOC_DVSEC_REVID, dvsec); } +static void hdm_decoder_commit (CXLType3Dev *ct3d,int which) +{ +ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; +uint32_t *cache_mem = cregs->cache_mem_registers; + +assert (which ==0 ); + +/* TODO: Sanity checks that the decoder is possible */ +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERROR ,0 ); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); +} + +static void ct3d_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; +CXLType3Dev *ct3d =container_of (cxl_cstate, CXLType3Dev, cxl_cstate); +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; +int which_hdm =-1 ; + +assert (size ==4 ); + +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); + which_hdm =0 ; +break ; +default : +break ; + } + +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) { +hdm_decoder_commit (ct3d, which_hdm); + } +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; + +g_free ((void *)regs->special_ops); +} +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) {MemoryRegion *mr;
@@ -103,6 +153,9 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) ct3d->cxl_cstate.pdev = pci_dev;build_dvsecs (ct3d); + regs->special_ops =g_new0 (MemoryRegionOps,1 ); + regs->special_ops->write = ct3d_reg_write; +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate,TYPE_CXL_TYPE3_DEV );
@@ -155,6 +208,7 @@ static const TypeInfo ct3d_info = { .parent = TYPE_PCI_DEVICE, .class_init = ct3_class_init, .instance_size = sizeof(CXLType3Dev), + .instance_finalize = ct3_finalize, .interfaces = (InterfaceInfo []) { {INTERFACE_CXL_DEVICE }, {INTERFACE_PCIE_DEVICE },
Signature · 1 line
2.32.0
PATCH v4 22/42 acpi/cxl: Add _OSC implementation (9.14.2)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
CXL 2.0 specification adds 2 new dwords to the existing _OSC definition from PCIe. The new dwords are accessed with a new uuid. This implementation supports what is in the specification.
--- hw/acpi/Kconfig | 5 ++ hw/acpi/cxl.c | 104 ++++++++++++++++++++++++++++++++++++++++++ hw/acpi/meson.build | 1 + hw/i386/acpi-build.c | 14 +++++- include/hw/acpi/cxl.h | 23 ++++++++++ 5 files changed, 146 insertions(+), 1 deletion(-)
hw/acpi/Kconfig +5 -0
@@ -5,6 +5,7 @@ config ACPI_X86 bool select ACPI select ACPI_NVDIMM + select ACPI_CXL select ACPI_CPU_HOTPLUG select ACPI_MEMORY_HOTPLUG select ACPI_HMAT
@@ -60,3 +61,7 @@ config ACPI_HW_REDUCED select ACPI select ACPI_MEMORY_HOTPLUG select ACPI_NVDIMM + +config ACPI_CXL + bool + depends on ACPI
hw/acpi/cxl.c new file +104 -0
104 lines not on this page · raw message
hw/acpi/meson.build +1 -0
@@ -13,6 +13,7 @@ acpi_ss.add(when: 'CONFIG_ACPI_MEMORY_HOTPLUG', if_false: files('acpi-mem-hotplu acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_true: files('nvdimm.c')) acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_false: files('acpi-nvdimm-stub.c')) acpi_ss.add(when: 'CONFIG_ACPI_PCI', if_true: files('pci.c')) +acpi_ss.add(when: 'CONFIG_ACPI_CXL', if_true: files('cxl.c')) acpi_ss.add(when: 'CONFIG_ACPI_VMGENID', if_true: files('vmgenid.c')) acpi_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device.c')) acpi_ss.add(when: 'CONFIG_ACPI_HMAT', if_true: files('hmat.c'))
hw/i386/acpi-build.c +13 -1
@@ -65,6 +65,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/utils.h" #include "hw/acpi/pci.h" +#include "hw/acpi/cxl.h" #include "qom/qom-qobject.h" #include "hw/i386/amd_iommu.h"
@@ -1406,13 +1407,24 @@ static void init_pci_acpi(Aml *dev, int uid, int type, bool native_pcie_hp)aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); - }else { + }else if (type ==PCIE ) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +/* Expander bridges do not have ACPI PCI Hot-plug enabled */ aml_append (dev,build_q35_osc_method (native_pcie_hp)); + }else {/* CXL */ +struct Aml *pkg =aml_package (2 ); + +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +build_cxl_osc_method (dev); } }
include/hw/acpi/cxl.h new file +23 -0
@@ -0,0 +1,23 @@ +/* +* Copyright (C) 2020 Intel Corporation +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. + +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. + +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef HW_ACPI_CXL_H +#define HW_ACPI_CXL_H + +void build_cxl_osc_method (Aml *dev); + +#endif
Signature · 1 line
2.32.0
PATCH v4 23/42 tests/acpi: allow CEDT table addition
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Following patches will add a new ACPI table, the
CXL Early Discovery Table (CEDT).
--- tests/data/acpi/pc/CEDT | 0 tests/data/acpi/q35/CEDT | 0 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++ 3 files changed, 2 insertions(+)
tests/data/acpi/pc/CEDT new file +0 -0
tests/data/acpi/q35/CEDT new file +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +2 -0
@@ -1 +1,3 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/pc/CEDT" , +"tests/data/acpi/q35/CEDT",
Signature · 1 line
2.32.0
PATCH v4 24/42 acpi/cxl: Create the CEDT (9.14.1)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The CXL Early Discovery Table is defined in the CXL 2.0 specification as a way for the OS to get CXL specific information from the system firmware.
CXL 2.0 specification adds an _HID, ACPI0016, for CXL capable host bridges, with a _CID of PNP0A08 (PCIe host bridge). CXL aware software is able to use this initiate the proper _OSC method, and get the _UID which is referenced by the CEDT. Therefore the existence of an ACPI0016 device allows a CXL aware driver perform the necessary actions. For a CXL capable OS, this works. For a CXL unaware OS, this works.
CEDT awaremess requires more. The motivation for ACPI0017 is to provide the possibility of having a Linux CXL module that can work on a legacy Linux kernel. Linux core PCI/ACPI which won't be built as a module, will see the _CID of PNP0A08 and bind a driver to it. If we later loaded a driver for ACPI0016, Linux won't be able to bind it to the hardware because it has already bound the PNP0A08 driver. The ACPI0017 device is an opportunity to have an object to bind a driver will be used by a Linux driver to walk the CXL topology and do everything that we would have preferred to do with ACPI0016.
There is another motivation for an ACPI0017 device which isn't implemented here. An operating system needs an attach point for a non-volatile region provider that understands cross-hostbridge interleaving. Since QEMU emulation doesn't support interleaving yet, this is more important on the OS side, for now.
As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge Structure (CHBS) which is primarily useful for telling the OS exactly where the MMIO for the host bridge is.
--- hw/acpi/cxl.c | 69 +++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 27 +++++++++++ hw/pci-bridge/pci_expander_bridge.c | 15 +------ include/hw/acpi/cxl.h | 5 +++ include/hw/pci/pci_bridge.h | 20 +++++++++ 5 files changed, 122 insertions(+), 14 deletions(-)
hw/acpi/cxl.c +69 -0
@@ -18,14 +18,83 @@*/ #include "qemu/osdep.h" +#include "hw/sysbus.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" +#include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h" #include "hw/acpi/bios-linker-loader.h" #include "hw/acpi/cxl.h" +#include "hw/acpi/cxl.h" #include "qapi/error.h" #include "qemu/uuid.h" +static void cedt_build_chbs (GArray *table_data,PXBDev *cxl) +{ +SysBusDevice *sbd =SYS_BUS_DEVICE (cxl->cxl.cxl_host_bridge); +struct MemoryRegion *mr = sbd->mmio[0 ].memory; + +/* Type */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Record Length */ +build_append_int_noprefix (table_data,32 ,2 ); + +/* UID - currently equal to bus number */ +build_append_int_noprefix (table_data, cxl->bus_nr,4 ); + +/* Version */ +build_append_int_noprefix (table_data,1 ,4 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,4 ); + +/* Base - subregion within a container that is in PA space */ +build_append_int_noprefix (table_data, mr->container->addr + mr->addr,8 ); + +/* Length */ +build_append_int_noprefix (table_data,memory_region_size (mr),8 ); +} + +static int cxl_foreach_pxb_hb (Object *obj,void *opaque) +{ +Aml *cedt = opaque; + +if (object_dynamic_cast (obj,TYPE_PXB_CXL_DEVICE )) { +cedt_build_chbs (cedt->buf,PXB_CXL_DEV (obj)); + } + +return 0 ; +} + +void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +BIOSLinker *linker,const char *oem_id, +const char *oem_table_id) +{ +Aml *cedt; +AcpiTable table = { .sig ="CEDT" , .rev =1 , .oem_id = oem_id, + .oem_table_id = oem_table_id }; + +acpi_add_table (table_offsets, table_data); +acpi_table_begin (&table, table_data); + cedt =init_aml_allocator (); + +/* reserve space for CEDT header */ + +object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); + +/* copy AML table into ACPI tables blob and patch header there */ +g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len); +free_aml_allocator (); + +acpi_table_end (linker, &table); +} +static Aml *__build_cxl_osc_method (void ) {Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
hw/i386/acpi-build.c +27 -0
@@ -76,6 +76,8 @@#include "hw/acpi/hmat.h" #include "hw/acpi/viot.h" +#include "hw/acpi/cxl.h" +/* These are used to size the ACPI tables for -M pc-i440fx-1.7 and * -M pc-i440fx-2.0. Even if the actual amount of AML generated grows * a little bit, there should be plenty of free space since the DSDT
@@ -1428,6 +1430,22 @@ static void init_pci_acpi(Aml *dev, int uid, int type, bool native_pcie_hp) } } +static void build_acpi0017 (Aml *table) +{ +Aml *dev, *scope, *method; + + scope =aml_scope ("_SB" ); + dev =aml_device ("CXLM" ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + + method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED ); +aml_append (method,aml_return (aml_int (0x01 ))); +aml_append (dev, method); + +aml_append (scope, dev); +aml_append (table, scope); +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1447,6 +1465,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,#ifdef CONFIG_TPM TPMIf *tpm =tpm_find ();#endif +bool cxl_present = false;int i;VMBusBridge *vmbus_bridge =vmbus_bridge_find ();AcpiTable table = { .sig ="DSDT" , .rev =1 , .oem_id = x86ms->oem_id,
@@ -1615,12 +1634,17 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,MemoryRegion *mr = &machine->cxl_devices_state->host_mr;uint64_t base = mr->addr; + cxl_present = true;crs_range_insert (crs_range_set.mem_ranges, base, base +memory_region_size (mr) -1 ); } } } +if (cxl_present) { +build_acpi0017 (dsdt); + } +/* * At this point crs_range_set has all the ranges used by pci * busses *other* than PCI0. These ranges will be excluded from
@@ -2674,6 +2698,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine) x86ms->oem_id, x86ms->oem_table_id); } +cxl_build_cedt (machine, table_offsets, tables_blob, tables->linker, + x86ms->oem_id, x86ms->oem_table_id); +acpi_add_table (table_offsets, tables_blob);build_waet (tables_blob, tables->linker, x86ms->oem_id, x86ms->oem_table_id);
hw/pci-bridge/pci_expander_bridge.c +1 -14
@@ -57,20 +57,6 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) -#define TYPE_PXB_CXL_DEVICE "pxb-cxl" -DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , - TYPE_PXB_CXL_DEVICE) - -struct PXBDev { -/*< private >*/ -PCIDevice parent_obj; -/*< public >*/ - -uint8_t bus_nr; -uint16_t numa_node; -bool bypass_iommu; -}; - typedef struct CXLHost { PCIHostState parent_obj;
@@ -346,6 +332,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, }else if (type ==CXL ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); bus->flags |=PCI_BUS_CXL ; +PXB_CXL_DEV (dev)->cxl.cxl_host_bridge =PXB_CXL_HOST (ds); }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
include/hw/acpi/cxl.h +5 -0
@@ -18,6 +18,11 @@#ifndef HW_ACPI_CXL_H #define HW_ACPI_CXL_H +#include "hw/acpi/bios-linker-loader.h" + +void cxl_build_cedt (MachineState *ms,GArray *table_offsets,GArray *table_data, +BIOSLinker *linker,const char *oem_id, +const char *oem_table_id);void build_cxl_osc_method (Aml *dev);#endif
include/hw/pci/pci_bridge.h +20 -0
@@ -28,6 +28,7 @@#include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" +#include "hw/cxl/cxl.h" #include "qom/object.h" typedef struct PCIBridgeWindows PCIBridgeWindows ;
@@ -80,6 +81,25 @@ struct PCIBridge {#define PCI_BRIDGE_DEV_PROP_CHASSIS_NR "chassis_nr"#define PCI_BRIDGE_DEV_PROP_MSI "msi"#define PCI_BRIDGE_DEV_PROP_SHPC "shpc" +typedef struct CXLHost CXLHost ; + +struct PXBDev { +/*< private >*/ +PCIDevice parent_obj; +/*< public >*/ + +uint8_t bus_nr; +uint16_t numa_node; +bool bypass_iommu; +struct cxl_dev { +CXLHost *cxl_host_bridge;/* Pointer to a CXLHost */ + } cxl; +}; + +typedef struct PXBDev PXBDev ; +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) int pci_bridge_ssvid_init(PCIDevice *dev, uint8_t offset, uint16_t svid, uint16_t ssid,
Signature · 1 line
2.32.0
PATCH v4 25/42 hw/cxl/device: Add some trivial commands
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
GET_FW_INFO and GET_PARTITION_INFO, for this emulation, is equivalent to info already returned in the IDENTIFY command. To have a more robust implementation, add those.
--- hw/cxl/cxl-mailbox-utils.c | 66 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
hw/cxl/cxl-mailbox-utils.c +66 -0
@@ -43,6 +43,8 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + FIRMWARE_UPDATE = 0x02, +#define GET_INFO 0x0 TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1
@@ -51,6 +53,8 @@ enum {#define GET_LOG 0x1 IDENTIFY = 0x40,#define MEMORY_DEVICE 0x0 + CCLS = 0x41, +#define GET_PARTITION_INFO 0x0 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -125,11 +129,13 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (FIRMWARE_UPDATE_GET_INFO );declare_mailbox_handler (TIMESTAMP_GET );declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG );declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE ); +declare_mailbox_handler (CCLS_GET_PARTITION_INFO );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -143,15 +149,50 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (FIRMWARE_UPDATE ,GET_INFO ,0 ,0 ),CXL_CMD (TIMESTAMP ,GET ,0 ,0 ),CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ),CXL_CMD (IDENTIFY ,MEMORY_DEVICE ,0 ,0 ), + CXL_CMD(CCLS, GET_PARTITION_INFO, 0, 0), };#undef CXL_CMD +/* +* 8.2.9.2.1 +*/ +define_mailbox_handler (FIRMWARE_UPDATE_GET_INFO ) +{ +struct { +uint8_t slots_supported; +uint8_t slot_info; +uint8_t caps; +uint8_t rsvd[0xd ]; +char fw_rev1[0x10 ]; +char fw_rev2[0x10 ]; +char fw_rev3[0x10 ]; +char fw_rev4[0x10 ]; + } __attribute__((packed)) *fw_info; +_Static_assert (sizeof (*fw_info) ==0x50 ,"Bad firmware info size" ); + +if (cxl_dstate->pmem_size < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + fw_info = (void *)cmd->payload; +memset (fw_info,0 ,sizeof (*fw_info)); + + fw_info->slots_supported =2 ; + fw_info->slot_info =BIT (0 ) |BIT (3 ); + fw_info->caps =0 ; +snprintf (fw_info->fw_rev1,0x10 ,"BWFW VERSION %02d" ,0 ); + + *len =sizeof (*fw_info); +return CXL_MBOX_SUCCESS ; +} +/* * 8.2.9.3.1 */
@@ -299,6 +340,31 @@ define_mailbox_handler(IDENTIFY_MEMORY_DEVICE)return CXL_MBOX_SUCCESS ; } +define_mailbox_handler (CCLS_GET_PARTITION_INFO ) +{ +struct { +uint64_t active_vmem; +uint64_t active_pmem; +uint64_t next_vmem; +uint64_t next_pmem; + } __attribute__((packed)) *part_info = (void *)cmd->payload; +_Static_assert (sizeof (*part_info) ==0x20 ,"Bad get partition info size" ); +uint64_t size = cxl_dstate->pmem_size; + +if (!QEMU_IS_ALIGNED (size,256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + +/* PMEM only */ + part_info->active_vmem =0 ; + part_info->next_vmem =0 ; + part_info->active_pmem = size / (256 <<20 ); + part_info->next_pmem = part_info->active_pmem; + + *len =sizeof (*part_info); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
2.32.0
PATCH v4 26/42 hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
This should introduce no change. Subsequent work will make use of this new class member.
--- hw/cxl/cxl-mailbox-utils.c | 3 +++ hw/mem/cxl_type3.c | 24 +++++++++--------------- include/hw/cxl/cxl_device.h | 29 +++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-)
hw/cxl/cxl-mailbox-utils.c +3 -0
@@ -321,6 +321,8 @@ define_mailbox_handler(IDENTIFY_MEMORY_DEVICE) } __attribute__((packed)) *id;_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d);uint64_t size = cxl_dstate->pmem_size;if (!QEMU_IS_ALIGNED (size,256 <<20 )) {
@@ -335,6 +337,7 @@ define_mailbox_handler(IDENTIFY_MEMORY_DEVICE) id->total_capacity = size / (256 <<20 ); id->persistent_capacity = size / (256 <<20 ); + id->lsa_size = cvc->get_lsa_size (ct3d); *len =sizeof (*id);return CXL_MBOX_SUCCESS ;
hw/mem/cxl_type3.c +9 -15
@@ -13,21 +13,6 @@#include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" -typedef struct cxl_type3_dev { -/* Private */ -PCIDevice parent_obj; - -/* Properties */ -uint64_t size; -HostMemoryBackend *hostmem; - -/* State */ -CXLComponentState cxl_cstate; -CXLDeviceState cxl_dstate; -}CXLType3Dev ; - -#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) -static void build_dvsecs (CXLType3Dev *ct3d) {CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
@@ -186,10 +171,16 @@ static Property ct3_props[] = { DEFINE_PROP_END_OF_LIST(), }; +static uint64_t get_lsa_size (CXLType3Dev *ct3d) +{ +return 0 ; +} +static void ct3_class_init (ObjectClass *oc,void *data) {DeviceClass *dc =DEVICE_CLASS (oc);PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +CXLType3Class *cvc =CXL_TYPE3_DEV_CLASS (oc); pc->realize = ct3_realize; pc->class_id =PCI_CLASS_STORAGE_EXPRESS ;
@@ -201,11 +192,14 @@ static void ct3_class_init(ObjectClass *oc, void *data) dc->desc ="CXL PMEM Device (Type 3)" ; dc->reset = ct3d_reset;device_class_set_props (dc, ct3_props); + + cvc->get_lsa_size = get_lsa_size; }static const TypeInfo ct3d_info = { .name =TYPE_CXL_TYPE3_DEV , .parent =TYPE_PCI_DEVICE , + .class_size =sizeof (struct CXLType3Class ), .class_init = ct3_class_init, .instance_size =sizeof (CXLType3Dev), .instance_finalize = ct3_finalize,
include/hw/cxl/cxl_device.h +29 -0
@@ -224,4 +224,33 @@ REG32(CXL_MEM_DEV_STS, 0) FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; +HostMemoryBackend *lsa; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#ifndef TYPE_CXL_TYPE3_DEV +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#endif + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) +OBJECT_DECLARE_TYPE(CXLType3Device, CXLType3Class, CXL_TYPE3_DEV) + +struct CXLType3Class { +/* Private */ +PCIDeviceClass parent_class; + +/* public */ +uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); +}; +#endif
Signature · 1 line
2.32.0
PATCH v4 27/42 hw/cxl/device: Implement get/set Label Storage Area (LSA)
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
--- hw/cxl/cxl-mailbox-utils.c | 54 +++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 56 ++++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 5 ++++ 3 files changed, 114 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +54 -0
@@ -55,6 +55,8 @@ enum {#define MEMORY_DEVICE 0x0 CCLS = 0x41,#define GET_PARTITION_INFO 0x0 +#define GET_LSA 0x2 +#define SET_LSA 0x3 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -136,8 +138,11 @@ declare_mailbox_handler(LOGS_GET_SUPPORTED);declare_mailbox_handler (LOGS_GET_LOG );declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );declare_mailbox_handler (CCLS_GET_PARTITION_INFO ); +declare_mailbox_handler (CCLS_GET_LSA ); +declare_mailbox_handler (CCLS_SET_LSA );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -156,6 +161,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ),CXL_CMD (IDENTIFY ,MEMORY_DEVICE ,0 ,0 ),CXL_CMD (CCLS ,GET_PARTITION_INFO ,0 ,0 ), +CXL_CMD (CCLS ,GET_LSA ,0 ,0 ), + CXL_CMD(CCLS, SET_LSA, ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE), };#undef CXL_CMD
@@ -368,6 +375,53 @@ define_mailbox_handler(CCLS_GET_PARTITION_INFO)return CXL_MBOX_SUCCESS ; } +define_mailbox_handler (CCLS_GET_LSA ) +{ +struct { +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (8 ))) *get_lsa; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint32_t offset, length; + + get_lsa = (void *)cmd->payload; + offset = get_lsa->offset; + length = get_lsa->length; + + *len =0 ; +if (offset + length > cvc->get_lsa_size (ct3d)) { +return CXL_MBOX_INVALID_INPUT ; + } + + *len = cvc->get_lsa (ct3d, get_lsa, length, offset); +return CXL_MBOX_SUCCESS ; +} + +define_mailbox_handler (CCLS_SET_LSA ) +{ +struct { +uint32_t offset; +uint32_t rsvd; + } __attribute__((packed,__aligned__ (8 ))) *set_lsa = (void *)cmd->payload; +CXLType3Dev *ct3d =container_of (cxl_dstate, CXLType3Dev, cxl_dstate); +CXLType3Class *cvc =CXL_TYPE3_DEV_GET_CLASS (ct3d); +uint16_t plen = *len; + + *len =0 ; +if (!plen) { +return CXL_MBOX_SUCCESS ; + } + +if (set_lsa->offset + plen > cvc->get_lsa_size (ct3d) +sizeof (*set_lsa)) { +return CXL_MBOX_INVALID_INPUT ; + } + + cvc->set_lsa (ct3d, (void *)set_lsa +sizeof (*set_lsa), + plen -sizeof (*set_lsa), set_lsa->offset); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
hw/mem/cxl_type3.c +55 -1
85 lines not on this page · raw message
include/hw/cxl/cxl_device.h +5 -0
@@ -251,6 +251,11 @@ struct CXLType3Class {/* public */ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d); + +uint64_t (*get_lsa)(CXLType3Dev *ct3d,void *buf,uint64_t size, +uint64_t offset); +void (*set_lsa)(CXLType3Dev *ct3d,const void *buf,uint64_t size, + uint64_t offset); };#endif
Signature · 1 line
2.32.0
PATCH v4 28/42 hw/cxl/component: Add utils for interleave parameter encoding/decoding
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Both registers and the CFMWS entries in CDAT use simple encodings for the number of interleave ways and the interleave granularity. Introduce simple conversion functions to/from the unencoded number / size. So far the iw decode has not been needed so is it not implemented.
--- hw/cxl/cxl-component-utils.c | 34 ++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 8 ++++++++ 2 files changed, 42 insertions(+)
hw/cxl/cxl-component-utils.c +34 -0
@@ -9,6 +9,7 @@#include "qemu/osdep.h" #include "qemu/log.h" +#include "qapi/error.h" #include "hw/pci/pci.h" #include "hw/cxl/cxl.h"
@@ -210,3 +211,36 @@ void cxl_component_create_dvsec(CXLComponentState *cxl, uint16_t length,range_init_nofail (&cxl->dvsecs[type], cxl->dvsec_offset, length); cxl->dvsec_offset += length; } + +uint8_t cxl_interleave_ways_enc (int iw,Error **errp) +{ +switch (iw) { +case 1 :return 0x0 ; +case 2 :return 0x1 ; +case 4 :return 0x2 ; +case 8 :return 0x3 ; +case 16 :return 0x4 ; +case 3 :return 0x8 ; +case 6 :return 0x9 ; +case 12 :return 0xa ; +default : +error_setg (errp,"Interleave ways: %d not supported" , iw); +return 0 ; + } +} + +uint8_t cxl_interleave_granularity_enc (uint64_t gran,Error **errp) +{ +switch (gran) { +case 256 :return 0 ; +case 512 :return 1 ; +case 1024 :return 2 ; +case 2048 :return 3 ; +case 4096 :return 4 ; +case 8192 :return 5 ; +case 16384 :return 6 ; +default : +error_setg (errp,"Interleave granularity: %lu invalid" , gran); +return 0 ; + } +}
include/hw/cxl/cxl_component.h +8 -0
@@ -193,4 +193,12 @@ void cxl_component_register_init_common(uint32_t *reg_state,void cxl_component_create_dvsec (CXLComponentState *cxl_cstate,uint16_t length,uint16_t type,uint8_t rev,uint8_t *body); +uint8_t cxl_interleave_ways_enc (int iw,Error **errp); +uint8_t cxl_interleave_granularity_enc (uint64_t gran,Error **errp); + +static inline hwaddr cxl_decode_ig (int ig) +{ +return 1 << (ig +8 ); +} +#endif
Signature · 1 line
2.32.0
PATCH v4 29/42 hw/cxl/host: Add support for CXL Fixed Memory Windows.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions.
ExampleS:
-cxl-fixed-memory-window targets=cxl.0,size=128G
-cxl-fixed-memory-window targets=cxl.1,size=128G
-cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
Specifies
* 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity.
When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added.
Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
--- hw/cxl/cxl-host-stubs.c | 22 +++++++ hw/cxl/cxl-host.c | 138 ++++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 12 ++-- include/hw/cxl/cxl.h | 20 ++++++ qapi/machine.json | 15 +++++ qemu-options.hx | 37 +++++++++++ softmmu/vl.c | 11 ++++ 7 files changed, 251 insertions(+), 4 deletions(-)
hw/cxl/cxl-host-stubs.c new file +22 -0
@@ -0,0 +1,22 @@ +/* +* CXL host parameter parsing routine stubs +* +* Copyright (c) 2022 Huawei +*/ +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/option.h" +#include "hw/cxl/cxl.h" + +QemuOptsList qemu_cxl_fixed_window_opts = { + .name ="cxl-fixed-memory-window" , + .implied_opt_name ="type" , + .head =QTAILQ_HEAD_INITIALIZER (qemu_cxl_fixed_window_opts.head), + .desc = { {0 } } +}; + +void parse_cxl_fixed_memory_window_opts (MachineState *ms) {}; + +void cxl_fixed_memory_window_link_targets (Error **errp) {}; + +const MemoryRegionOps cfmws_ops;
hw/cxl/cxl-host.c new file +138 -0
138 lines not on this page · raw message
hw/cxl/meson.build +8 -4
@@ -1,5 +1,9 @@ -softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( - 'cxl-component-utils.c', - 'cxl-device-utils.c', - 'cxl-mailbox-utils.c', +specific_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', + 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', + 'cxl-host.c', +)) +specific_ss.add(when: 'CONFIG_CXL', if_false: files( + 'cxl-host-stubs.c', ))
include/hw/cxl/cxl.h +20 -0
@@ -10,6 +10,9 @@#ifndef CXL_H #define CXL_H +#include "qapi/qapi-types-machine.h" +#include "qemu/option.h" +#include "hw/pci/pci_bridge.h" #include "cxl_pci.h" #include "cxl_component.h" #include "cxl_device.h"
@@ -20,10 +23,27 @@#define TYPE_CXL_TYPE3_DEV "cxl-type3"#define CXL_WINDOW_MAX 10 +typedef struct CXLFixedWindow { +uint64_t size; +char **targets; +struct PXBDev *target_hbs[8 ]; +uint8_t num_targets; +uint8_t enc_int_ways; +uint8_t enc_int_gran; +/* Todo: XOR based interleaving */ +MemoryRegion mr; +hwaddr base; +}CXLFixedWindow ; +typedef struct CXLState {bool is_enabled;MemoryRegion host_mr;unsigned int next_mr_idx; +GList *fixed_windows; }CXLState ; +extern QemuOptsList qemu_cxl_fixed_window_opts; +void parse_cxl_fixed_memory_window_opts (MachineState *ms); +void cxl_fixed_memory_window_link_targets (Error **errp); +#endif
qapi/machine.json +15 -0
@@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDs +# +# Since X.X //fixme +## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: #
qemu-options.hx +37 -0
@@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)
softmmu/vl.c +11 -0
@@ -91,6 +91,7 @@#include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif
@@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)qemu_init_board ();qemu_create_cli_devices (); +cxl_fixed_memory_window_link_targets (errp);qemu_machine_creation_done ();if (loadvm) {
@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_add_opts (&qemu_msg_opts);qemu_add_opts (&qemu_name_opts);qemu_add_opts (&qemu_numa_opts); +qemu_add_opts (&qemu_cxl_fixed_window_opts);qemu_add_opts (&qemu_icount_opts);qemu_add_opts (&qemu_semihosting_config_opts);qemu_add_opts (&qemu_fw_cfg_opts);
@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)exit (1 ); }break ; +case QEMU_OPTION_cxl_fixed_memory_window: + opts =qemu_opts_parse_noisily (qemu_find_opts ("cxl-fixed-memory-window" ), + optarg, true); +if (!opts) { +exit (1 ); + } +break ;case QEMU_OPTION_display:parse_display (optarg);break ;
@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)qemu_resolve_machine_memdev ();parse_numa_opts (current_machine); +parse_cxl_fixed_memory_window_opts (current_machine);if (vmstate_dump_file) {/* dump and exit */
Signature · 1 line
2.32.0
PATCH v4 30/42 acpi/cxl: Introduce CFMWS structures in CEDT
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
The CEDT CXL Fixed Window Memory Window Structures (CFMWs) define regions of the host phyiscal address map which (via an impdef means) are configured such that they have a particular interleave setup across one or more CXL Host Bridges.
--- hw/acpi/cxl.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+)
hw/acpi/cxl.c +59 -0
@@ -61,6 +61,64 @@ static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)build_append_int_noprefix (table_data,memory_region_size (mr),8 ); } +/* +* CFMWS entries in CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. +* Interleave ways encoding in CXL 2.0 ECN: 3, 6, 12 and 16-way memory +* interleaving. +*/ +static void cedt_build_cfmws (GArray *table_data,MachineState *ms) +{ +CXLState *cxls = ms->cxl_devices_state; +GList *it; + +for (it = cxls->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; +int i; + +/* Type */ +build_append_int_noprefix (table_data,1 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Record Length */ +build_append_int_noprefix (table_data,36 +4 * fw->num_targets,2 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,4 ); + +/* Base HPA */ +build_append_int_noprefix (table_data, fw->mr.addr,8 ); + +/* Window Size */ +build_append_int_noprefix (table_data, fw->size,8 ); + +/* Host Bridge Interleave Ways */ +build_append_int_noprefix (table_data, fw->enc_int_ways,1 ); + +/* Host Bridge Interleave Arithmetic */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,2 ); + +/* Host Bridge Interleave Granularity */ +build_append_int_noprefix (table_data, fw->enc_int_gran,4 ); + +/* Window Restrictions */ +build_append_int_noprefix (table_data,0x0f ,2 );/* No restrictions */ + +/* QTG ID */ +build_append_int_noprefix (table_data,0 ,2 ); + +/* Host Bridge List (list of UIDs - currently bus_nr) */ +for (i =0 ; i < fw->num_targets; i++) { +g_assert (fw->target_hbs[i]); +build_append_int_noprefix (table_data, fw->target_hbs[i]->bus_nr,4 ); + } + } +} +static int cxl_foreach_pxb_hb (Object *obj,void *opaque) {Aml *cedt = opaque;
@@ -87,6 +145,7 @@ void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,/* reserve space for CEDT header */ object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); +cedt_build_cfmws (cedt->buf, ms);/* copy AML table into ACPI tables blob and patch header there */ g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len);
Signature · 1 line
2.32.0
PATCH v4 31/42 hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
In reply to Jonathan CameronThis adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
--- hw/arm/Kconfig | 1 + hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)
hw/arm/Kconfig +1 -0
@@ -29,6 +29,7 @@ config ARM_VIRT select ACPI_APEI select ACPI_VIOT select VIRTIO_MEM_SUPPORTED + select ACPI_CXL config CHEETAH bool
hw/pci-host/gpex-acpi.c +19 -3
@@ -5,6 +5,7 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" +#include "hw/acpi/cxl.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq) {
@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;
@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); + dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); +if (is_cxl) { +struct Aml *pkg =aml_package (2 ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_STR" ,aml_unicode ("pxb Device" )));
@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs)); -acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev); + }else { +acpi_dsdt_add_pci_osc (dev); + }aml_append (scope, dev); }
Signature · 1 line
2.32.0
PATCH v4 32/42 pci/pcie_port: Add pci_find_port_by_pn()
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Simple function to search a PCIBus to find a port by it's port number.
CXL interleave decoding uses the port number as a target so it is necessary to locate the port when doing interleave decoding.
--- hw/pci/pcie_port.c | 25 +++++++++++++++++++++++++ include/hw/pci/pcie_port.h | 2 ++ 2 files changed, 27 insertions(+)
hw/pci/pcie_port.c +25 -0
@@ -136,6 +136,31 @@ static void pcie_port_class_init(ObjectClass *oc, void *data)device_class_set_props (dc, pcie_port_props); } +PCIDevice *pcie_find_port_by_pn (PCIBus *bus,uint8_t pn) +{ +int devfn; + +for (devfn =0 ; devfn <ARRAY_SIZE (bus->devices); devfn++) { +PCIDevice *d = bus->devices[devfn]; +PCIEPort *port; + +if (!d || !pci_is_express (d) || !d->exp.exp_cap) { +continue ; + } + +if (!object_dynamic_cast (OBJECT (d),TYPE_PCIE_PORT )) { +continue ; + } + + port =PCIE_PORT (d); +if (port->port == pn) { +return d; + } + } + +return NULL ; +} +static const TypeInfo pcie_port_type_info = { .name =TYPE_PCIE_PORT , .parent =TYPE_PCI_BRIDGE ,
include/hw/pci/pcie_port.h +2 -0
@@ -39,6 +39,8 @@ struct PCIEPort {void pcie_port_init_reg (PCIDevice *d); +PCIDevice *pcie_find_port_by_pn (PCIBus *bus,uint8_t pn); +#define TYPE_PCIE_SLOT "pcie-slot"OBJECT_DECLARE_SIMPLE_TYPE (PCIESlot,PCIE_SLOT )
Signature · 1 line
2.32.0
PATCH v4 33/42 CXL/cxl_component: Add cxl_get_hb_cstate()
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Accessor to get hold of the cxl state for a CXL host bridge without exposing the internals of the implementation.
--- hw/pci-bridge/pci_expander_bridge.c | 7 +++++++ include/hw/cxl/cxl_component.h | 2 ++ 2 files changed, 9 insertions(+)
hw/pci-bridge/pci_expander_bridge.c +7 -0
@@ -81,6 +81,13 @@ static GList *pxb_dev_list;#define TYPE_PXB_CXL_HOST "pxb-cxl-host"#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb) +{ +CXLHost *host =PXB_CXL_HOST (hb); + +return &host->cxl_cstate; +} +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);
include/hw/cxl/cxl_component.h +2 -0
@@ -201,4 +201,6 @@ static inline hwaddr cxl_decode_ig(int ig) return 1 << (ig + 8); } +CXLComponentState *cxl_get_hb_cstate (PCIHostState *hb); +#endif
Signature · 1 line
2.32.0
PATCH v4 34/42 mem/cxl_type3: Add read and write functions for associated hostmem.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Once a read or write reaches a CXL type 3 device, the HDM decoders on the device are used to establish the Device Physical Address which should be accessed. These functions peform the required maths and then directly access the hostmem->mr to fullfil the actual operation. Note that failed writes are silent, but failed reads return poison. Note this is based loosely on:
https://lore.kernel.org/qemu-devel/[email protected]/
[RFC PATCH 0/9] hw/misc: Add support for interleaved memory accesses
Only lightly tested so far. More complex test cases yet to be written.
--- hw/mem/cxl_type3.c | 81 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 5 +++ 2 files changed, 86 insertions(+)
hw/mem/cxl_type3.c +81 -0
@@ -161,6 +161,87 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) &ct3d->cxl_dstate.device_registers); } +/* TODO: Support multiple HDM decoders and DPA skip */ +static bool cxl_type3_dpa (CXLType3Dev *ct3d,hwaddr host_addr,uint64_t *dpa) +{ +uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers; +uint64_t decoder_base, decoder_size, hpa_offset; +uint32_t hdm0_ctrl; +int ig, iw; + + decoder_base = (((uint64_t )cache_mem[R_CXL_HDM_DECODER0_BASE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_BASE_LO ]); +if ((uint64_t )host_addr < decoder_base) { +return false; + } + + hpa_offset = (uint64_t )host_addr - decoder_base; + + decoder_size = ((uint64_t )cache_mem[R_CXL_HDM_DECODER0_SIZE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_SIZE_LO ]; +if (hpa_offset >= decoder_size) { +return false; + } + + hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL ]; + iw =FIELD_EX32 (hdm0_ctrl,CXL_HDM_DECODER0_CTRL ,IW ); + ig =FIELD_EX32 (hdm0_ctrl,CXL_HDM_DECODER0_CTRL ,IG ); + + *dpa = (MAKE_64BIT_MASK (0 ,8 + ig) & hpa_offset) | + ((MAKE_64BIT_MASK (8 + ig + iw,64 -8 - ig - iw) & hpa_offset) >> iw); + +return true; +} + +MemTxResult cxl_type3_read (PCIDevice *d,hwaddr host_addr,uint64_t *data, +unsigned size,MemTxAttrs attrs) +{ +CXLType3Dev *ct3d =CT3 (d); +uint64_t dpa_offset; +MemoryRegion *mr; + +/* TODO support volatile region */ + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return MEMTX_ERROR ; + } + +if (!cxl_type3_dpa (ct3d, host_addr, &dpa_offset)) { +return MEMTX_ERROR ; + } + +if (dpa_offset > mr->size) { +return MEMTX_ERROR ; + } + +return memory_region_dispatch_read (mr, dpa_offset, data, +size_memop (size), attrs); +} + +MemTxResult cxl_type3_write (PCIDevice *d,hwaddr host_addr,uint64_t data, +unsigned size,MemTxAttrs attrs) +{ +CXLType3Dev *ct3d =CT3 (d); +uint64_t dpa_offset; +MemoryRegion *mr; + + mr =host_memory_backend_get_memory (ct3d->hostmem); +if (!mr) { +return MEMTX_OK ; + } + +if (!cxl_type3_dpa (ct3d, host_addr, &dpa_offset)) { +return MEMTX_OK ; + } + +if (dpa_offset > mr->size) { +return MEMTX_OK ; + } + +return memory_region_dispatch_write (mr, dpa_offset, data, +size_memop (size), attrs); +} +static void ct3d_reset (DeviceState *dev) {CXLType3Dev *ct3d =CT3 (dev);
include/hw/cxl/cxl_device.h +5 -0
@@ -258,4 +258,9 @@ struct CXLType3Class { uint64_t offset); }; +MemTxResult cxl_type3_read (PCIDevice *d,hwaddr host_addr,uint64_t *data, +unsigned size,MemTxAttrs attrs); +MemTxResult cxl_type3_write (PCIDevice *d,hwaddr host_addr,uint64_t data, +unsigned size,MemTxAttrs attrs); +#endif
Signature · 1 line
2.32.0
PATCH v4 35/42 cxl/cxl-host: Add memops for CFMWS region.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
These memops perform interleave decoding, walking down the CXL topology from CFMWS described host interleave decoder via CXL host bridge HDM decoders, through the CXL root ports and finally call CXL type 3 specific read and write functions.
Note that, whilst functional the current implementation does not support:
* switches
* multiple HDM decoders at a given level.
* unaligned accesses across the interleave boundaries
--- hw/cxl/cxl-host.c | 125 +++++++++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl.h | 2 + 2 files changed, 127 insertions(+)
hw/cxl/cxl-host.c +125 -0
@@ -136,3 +136,128 @@ void cxl_fixed_memory_window_link_targets(Error **errp) } } } + +/* TODO: support, multiple hdm decoders */ +static bool cxl_hdm_find_target (uint32_t *cache_mem,hwaddr addr, +uint8_t *target) +{ +uint32_t ctrl; +uint32_t ig_enc; +uint32_t iw_enc; +uint32_t target_reg; +uint32_t target_idx; + + ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL ]; +if (!FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,COMMITTED )) { +return false; + } + + ig_enc =FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,IG ); + iw_enc =FIELD_EX32 (ctrl,CXL_HDM_DECODER0_CTRL ,IW ); + target_idx = (addr /cxl_decode_ig (ig_enc)) % (1 << iw_enc); + +if (target_idx >4 ) { + target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO ]; + target_reg >>= target_idx *8 ; + }else { + target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO ]; + target_reg >>= (target_idx -4 ) *8 ; + } + *target = target_reg &0xff ; + +return true; +} + +static PCIDevice *cxl_cfmws_find_device (CXLFixedWindow *fw,hwaddr addr) +{ +CXLComponentState *hb_cstate; +PCIHostState *hb; +int rb_index; +uint32_t *cache_mem; +uint8_t target; +bool target_found; +PCIDevice *rp, *d; + +/* Address is relative to memory region. Convert to HPA */ + addr += fw->base; + + rb_index = (addr /cxl_decode_ig (fw->enc_int_gran)) % fw->num_targets; + hb =PCI_HOST_BRIDGE (fw->target_hbs[rb_index]->cxl.cxl_host_bridge); +if (!hb || !hb->bus || !pci_bus_is_cxl (hb->bus)) { +return NULL ; + } + + hb_cstate =cxl_get_hb_cstate (hb); +if (!hb_cstate) { +return NULL ; + } + + cache_mem = hb_cstate->crb.cache_mem_registers; + + target_found =cxl_hdm_find_target (cache_mem, addr, &target); +if (!target_found) { +return NULL ; + } + + rp =pcie_find_port_by_pn (hb->bus, target); +if (!rp) { +return NULL ; + } + + d =pci_bridge_get_sec_bus (PCI_BRIDGE (rp))->devices[0 ]; + +if (!d || !object_dynamic_cast (OBJECT (d),TYPE_CXL_TYPE3_DEV )) { +return NULL ; + } + +return d; +} + +static MemTxResult cxl_read_cfmws (void *opaque,hwaddr addr,uint64_t *data, +unsigned size,MemTxAttrs attrs) +{ +CXLFixedWindow *fw = opaque; +PCIDevice *d; + + d =cxl_cfmws_find_device (fw, addr); +if (d ==NULL ) { + *data =0 ; +/* Reads to invalid address return poison */ +return MEMTX_ERROR ; + } + +return cxl_type3_read (d, addr + fw->base, data, size, attrs); +} + +static MemTxResult cxl_write_cfmws (void *opaque,hwaddr addr, +uint64_t data,unsigned size, +MemTxAttrs attrs) +{ +CXLFixedWindow *fw = opaque; +PCIDevice *d; + + d =cxl_cfmws_find_device (fw, addr); +if (d ==NULL ) { +/* Writes to invalid address are silent */ +return MEMTX_OK ; + } + +return cxl_type3_write (d, addr + fw->base, data, size, attrs); +} + +const MemoryRegionOps cfmws_ops = { + .read_with_attrs = cxl_read_cfmws, + .write_with_attrs = cxl_write_cfmws, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = true, + }, + .impl = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = true, + }, +}; +
include/hw/cxl/cxl.h +2 -0
@@ -46,4 +46,6 @@ extern QemuOptsList qemu_cxl_fixed_window_opts;void parse_cxl_fixed_memory_window_opts (MachineState *ms);void cxl_fixed_memory_window_link_targets (Error **errp); +extern const MemoryRegionOps cfmws_ops; +#endif
Signature · 1 line
2.32.0
PATCH v4 36/42 arm/virt: Allow virt/CEDT creation
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Allow for the creation of the CEDT ACPI table without qtest fails due to the unknown ACPI tables.
--- tests/data/acpi/virt/CEDT | 0 tests/qtest/bios-tables-test-allowed-diff.h | 1 + 2 files changed, 1 insertion(+)
tests/data/acpi/virt/CEDT new file +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +1 -0
@@ -1,3 +1,4 @@/* List of comma-separated changed AML files to ignore */ "tests/data/acpi/pc/CEDT" ,"tests/data/acpi/q35/CEDT" , +"tests/data/acpi/virt/CEDT",
Signature · 1 line
2.32.0
PATCH v4 37/42 hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl
In reply to Jonathan CameronCode based on i386/pc enablement.
The memory layout places space for 16 host bridge register regions after the GIC_REDIST2 in the extended memmap.
The CFMWs are placed above the extended memmap.
--- hw/arm/virt-acpi-build.c | 30 +++++++++++++++++++++++++++++ hw/arm/virt.c | 41 ++++++++++++++++++++++++++++++++++++++-- include/hw/arm/virt.h | 1 + 3 files changed, 70 insertions(+), 2 deletions(-)
hw/arm/virt-acpi-build.c +30 -0
@@ -39,6 +39,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/utils.h" #include "hw/acpi/pci.h" +#include "hw/acpi/cxl.h" #include "hw/acpi/memory_hotplug.h" #include "hw/acpi/generic_event_device.h" #include "hw/acpi/tpm.h"
@@ -157,10 +158,29 @@ static void acpi_dsdt_add_virtio(Aml *scope, } } +/* Uses local definition of AcpiBuildState so can't easily be common code */ +static void build_acpi0017 (Aml *table) +{ +Aml *dev, *scope, *method; + + scope =aml_scope ("_SB" ); + dev =aml_device ("CXLM" ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + + method =aml_method ("_STA" ,0 ,AML_NOTSERIALIZED ); +aml_append (method,aml_return (aml_int (0x01 ))); +aml_append (dev, method); + +aml_append (scope, dev); +aml_append (table, scope); +} +static void acpi_dsdt_add_pci (Aml *scope,const MemMapEntry *memmap,uint32_t irq,VirtMachineState *vms) {int ecam_id =VIRT_ECAM_ID (vms->highmem_ecam); +bool cxl_present = false; +PCIBus *bus = vms->bus;struct GPEXConfig cfg = { .mmio32 = memmap[VIRT_PCIE_MMIO ], .pio = memmap[VIRT_PCIE_PIO ],
@@ -174,6 +194,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, }acpi_dsdt_add_gpex (scope, &cfg); + QLIST_FOREACH(bus, &vms->bus->child, sibling) { +if (pci_bus_is_cxl (bus)) { + cxl_present = true; + } + } +if (cxl_present) { +build_acpi0017 (scope); + } } static void acpi_dsdt_add_gpio(Aml *scope, const MemMapEntry *gpio_memmap,
@@ -991,6 +1019,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables) vms->oem_table_id); } } +cxl_build_cedt (ms, table_offsets, tables_blob, tables->linker, + vms->oem_id, vms->oem_table_id);if (ms->nvdimms_state->is_enabled) { nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
hw/arm/virt.c +39 -2
89 lines not on this page · raw message
include/hw/arm/virt.h +1 -0
@@ -92,6 +92,7 @@ enum {/* indices of IO regions located after the RAM */ enum {VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_CXL_HOST ,VIRT_HIGH_PCIE_ECAM ,VIRT_HIGH_PCIE_MMIO , };
Signature · 1 line
2.32.0
PATCH v4 38/42 RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Inorder to implement memory interleaving we need a means to proxy the calls. Adding mem_ops allows such proxying.
Note should have no impact on use cases not using _dispatch_read/write. For now, only file backed hostmem is considered to seek feedback on the approach before considering other hostmem backends.
--- softmmu/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+)
softmmu/memory.c +9 -0
@@ -1606,6 +1606,15 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,Error *err =NULL ;memory_region_init (mr, owner, name, size); mr->ram = true; + +/* +* ops used only when directly accessing via +* - memory_region_dispatch_read() +* - memory_region_dispatch_write() +*/ + mr->ops = &ram_device_mem_ops; + mr->opaque = mr; + mr->readonly = readonly; mr->terminates = true; mr->destructor = memory_region_destructor_ram;
Signature · 1 line
2.32.0
PATCH v4 39/42 hw/cxl/component Add a dumb HDM decoder handler
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Add a trivial handler for now to cover the root bridge where we could do some error checking in future.
* Ben. I lifted this one from you but you didn't sign off on it.
--- hw/cxl/cxl-component-utils.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
hw/cxl/cxl-component-utils.c +31 -0
@@ -28,6 +28,31 @@ static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset, } } +static void dumb_hdm_handler (CXLComponentState *cxl_cstate,hwaddr offset, +uint32_t value) +{ +ComponentRegisters *cregs = &cxl_cstate->crb; +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; + +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); +break ; +default : +break ; + } + +memory_region_transaction_begin (); +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) { +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERROR ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); + } +memory_region_transaction_commit (); +} +static void cxl_cache_mem_write_reg (void *opaque,hwaddr offset,uint64_t value,unsigned size) {
@@ -38,6 +63,12 @@ static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value,if (cregs->special_ops && cregs->special_ops->write) { cregs->special_ops->write (cxl_cstate, offset, value, size); +return ; + } + +if (offset >=A_CXL_HDM_DECODER_CAPABILITY && + offset <=A_CXL_HDM_DECODER0_TARGET_LIST_HI ) { +dumb_hdm_handler (cxl_cstate, offset, value); }else { cregs->cache_mem_registers[offset /4 ] = value; }
Signature · 1 line
2.32.0
PATCH v4 40/42 i386/pc: Enable CXL fixed memory windows
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
Add the CFMWs memory regions to the memorymap and adjust the PCI window to avoid hitting the same memory.
--- hw/i386/pc.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
hw/i386/pc.c +30 -1
@@ -816,7 +816,7 @@ void pc_memory_init(PCMachineState *pcms,MachineClass *mc =MACHINE_GET_CLASS (machine);PCMachineClass *pcmc =PC_MACHINE_GET_CLASS (pcms);X86MachineState *x86ms =X86_MACHINE (pcms); -hwaddr cxl_base; +hwaddr cxl_base, cxl_resv_end =0 ;assert (machine->ram_size == x86ms->below_4g_mem_size + x86ms->above_4g_mem_size);
@@ -924,6 +924,24 @@ void pc_memory_init(PCMachineState *pcms,e820_add_entry (cxl_base, cxl_size,E820_RESERVED );memory_region_init (mr,OBJECT (machine),"cxl_host_reg" , cxl_size);memory_region_add_subregion (system_memory, cxl_base, mr); + cxl_resv_end = cxl_base + cxl_size; +if (machine->cxl_devices_state->fixed_windows) { +hwaddr cxl_fmw_base; +GList *it; + + cxl_fmw_base =ROUND_UP (cxl_base + cxl_size,256 * MiB); +for (it = machine->cxl_devices_state->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; + + fw->base = cxl_fmw_base; +memory_region_init_io (&fw->mr,OBJECT (machine), &cfmws_ops, fw, +"cxl-fixed-memory-region" , fw->size); +memory_region_add_subregion (system_memory, fw->base, &fw->mr); +e820_add_entry (fw->base, fw->size,E820_RESERVED ); + cxl_fmw_base += fw->size; + cxl_resv_end = cxl_fmw_base; + } + } }/* Initialize PC system firmware */
@@ -953,6 +971,10 @@ void pc_memory_init(PCMachineState *pcms,if (!pcmc->broken_reserved_end) { res_mem_end +=memory_region_size (&machine->device_memory->mr); } + +if (machine->cxl_devices_state->is_enabled) { + res_mem_end = cxl_resv_end; + } *val =cpu_to_le64 (ROUND_UP (res_mem_end,1 * GiB));fw_cfg_add_file (fw_cfg,"etc/reserved-memory-end" , val,sizeof (*val)); }
@@ -989,6 +1011,13 @@ uint64_t pc_pci_hole64_start(void)if (ms->cxl_devices_state->host_mr.addr) { hole64_start = ms->cxl_devices_state->host_mr.addr +memory_region_size (&ms->cxl_devices_state->host_mr); +if (ms->cxl_devices_state->fixed_windows) { +GList *it; +for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { +CXLFixedWindow *fw = it->data; + hole64_start = fw->mr.addr +memory_region_size (&fw->mr); + } + } }else if (pcmc->has_reserved_memory && ms->device_memory->base) { hole64_start = ms->device_memory->base;if (!pcmc->broken_reserved_end) {
Signature · 1 line
2.32.0
PATCH v4 41/42 qtest/acpi: Add reference CEDT tables.
In reply to Jonathan CameronFrom: Jonathan Cameron <[email protected]>
More sophisticated tests will come later, but for now deal with the NULL case.
--- tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes tests/qtest/bios-tables-test-allowed-diff.h | 3 --- 4 files changed, 3 deletions(-)
tests/data/acpi/pc/CEDT binary +0 -0
tests/data/acpi/q35/CEDT binary +0 -0
tests/data/acpi/virt/CEDT binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -3
@@ -1,4 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/pc/CEDT" , -"tests/data/acpi/q35/CEDT" , -"tests/data/acpi/virt/CEDT",
Signature · 1 line
2.32.0
PATCH v4 42/42 qtest/cxl: Add very basic sanity tests
In reply to Jonathan CameronFrom: Ben Widawsky <[email protected]>
Simple 'does it boot tests' with up to
2x PXB host bridge, each with 2x CXL RP and each of those with a Type 3 memory device. Single CFMWS to interleave across the two HBs and ultimate the 4 devices.
More complete tests may be possible but CXL interleave setup is complex so a lot of steps will be needed.
--- tests/qtest/cxl-test.c | 151 ++++++++++++++++++++++++++++++++++++++++ tests/qtest/meson.build | 4 ++ 2 files changed, 155 insertions(+)
tests/qtest/cxl-test.c new file +151 -0
@@ -0,0 +1,151 @@ +/* +* QTest testcase for CXL +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "libqtest-single.h" + +#define QEMU_PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=22 " \ + "-cxl-fixed-memory-window targets=cxl.0,size=4G " + +#define QEMU_2PXB_CMD "-machine q35,cxl=on " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=22 " \ + "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53,uid=122 " \ + "-cxl-fixed-memory-window targets=cxl.0,targets=cxl.1,size=4G " + +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " + +/* Dual ports on first pxb */ +#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " + +/* Dual ports on each of the pxb instances */ +#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \ + "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \ + "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ + "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " + +#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " + +#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " + +#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,memdev=cxl-mem2,id=cxl-pmem2,size=256M " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,memdev=cxl-mem3,id=cxl-pmem3,size=256M " + +static void cxl_basic_hb (void ) +{ +qtest_start ("-machine q35,cxl=on" ); +qtest_end (); +} + +static void cxl_basic_pxb (void ) +{ +qtest_start ("-machine q35,cxl=on -device pxb-cxl,bus=pcie.0,uid=22" ); +qtest_end (); +} + +static void cxl_pxb_with_window (void ) +{ +qtest_start (QEMU_PXB_CMD ); +qtest_end (); +} + +static void cxl_2pxb_with_window (void ) +{ +qtest_start (QEMU_2PXB_CMD ); +qtest_end (); +} + +static void cxl_root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_RP); +qtest_end (); +} + +static void cxl_2root_port (void ) +{ + qtest_start(QEMU_PXB_CMD QEMU_2RP); +qtest_end (); +} + +static void cxl_t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_1pxb_2rp_2t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_2RP QEMU_2T3D, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_2pxb_4rp_4t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); + g_string_printf(cmdline, QEMU_2PXB_CMD QEMU_4RP QEMU_4T3D, + tmpfs, tmpfs, tmpfs, tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +int main (int argc,char **argv) +{ +g_test_init (&argc, &argv,NULL ); + +qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb); +qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); +qtest_add_func ("/pci/cxl/pxb_x2_with_window" , cxl_2pxb_with_window); +qtest_add_func ("/pci/cxl/rp" , cxl_root_port); +qtest_add_func ("/pci/cxl/rp_x2" , cxl_2root_port); +qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); +qtest_add_func ("/pci/cxl/rp_x2_type3_x2" , cxl_1pxb_2rp_2t3d); +qtest_add_func ("/pci/cxl/pxb_x2_root_port_x4_type3_x4" , cxl_2pxb_4rp_4t3d); +return g_test_run (); +}
tests/qtest/meson.build +4 -0
@@ -41,6 +41,9 @@ qtests_pci = \ (config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : []) + \ (config_all_devices.has_key('CONFIG_IVSHMEM_DEVICE') ? ['ivshmem-test'] : []) +qtests_cxl = \ + (config_all_devices.has_key('CONFIG_CXL') ? ['cxl-test'] : []) + qtests_i386 = \ (slirp.found() ? ['pxe-test', 'test-netfilter'] : []) + \ (config_host.has_key('CONFIG_POSIX') ? ['test-filter-mirror'] : []) + \
@@ -74,6 +77,7 @@ qtests_i386 = \ slirp.found() ? ['virtio-net-failover'] : []) + \ (unpack_edk2_blobs ? ['bios-tables-test'] : []) + \ qtests_pci + \ + qtests_cxl + \ ['fdc-test', 'ide-test', 'hd-geo-test',
Signature · 1 line
2.32.0
Quoted 3 lines · On Mon, 24 Jan 2022 17:16:23 +0000 Jonathan Cameron…
I'll post a git tree shortly for any who prefer that to lots of emails ;)
https://github.com/hisilicon/qemu.git cxl-v4
I've included the pci swizzle fix on the tree as it avoids some really annoying waits for interrupts to get masked when testing on ARM.
I've also put the basic DOE patch set on top (CDAT + compliance) and the ARM GIC ITS support patch for virt as who wouldn't want that?
Thanks,
Jonathan
Quoted 127 lines
Thanks,
Jonathan
Ben Widawsky (26):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory device utilities hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge) acpi/pci: Consolidate host bridge setup hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add_OSCimplementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
hw/cxl/device: Add some trivial commands hw/cxl/device: Plumb real Label Storage Area (LSA) sizing hw/cxl/device: Implement get/set Label Storage Area (LSA) acpi/cxl: Introduce CFMWS structures in CEDT hw/cxl/component Add a dumb HDM decoder handler qtest/cxl: Add very basic sanity testsJonathan Cameron (16):
MAINTAINERS: Add entry for Compute Express Link Emulation tests/acpi: allow DSDT.viot table changes.
tests/acpi: Add update DSDT.viot
cxl: Machine level control on whether CXL support is enabled hw/cxl/component: Add utils for interleave parameter encoding/decoding hw/cxl/host: Add support for CXL Fixed Memory Windows. hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl pci/pcie_port: Addpci_find_port_by_pn()CXL/cxl_component: Addcxl_get_hb_cstate()
mem/cxl_type3: Add read and write functions for associated hostmem. cxl/cxl-host: Add memops for CFMWS region.
arm/virt: Allowvirt/CEDTcreation
hw/arm/virt: Basic CXL enablement onpci_expander_bridgeinstances
pxb-cxl
RFC: softmmu/memory: Add ops tomemory_region_ram_init_from_filei386/pc: Enable CXL fixed memory windows
qtest/acpi: Add reference CEDT tables.MAINTAINERS | 7 + hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl.c | 232 +++++++++++++ hw/acpi/meson.build | 1 + hw/arm/Kconfig | 1 + hw/arm/virt-acpi-build.c | 30 ++ hw/arm/virt.c | 40 ++- hw/core/machine.c | 26 ++ hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 277 +++++++++++++++ hw/cxl/cxl-device-utils.c | 268 +++++++++++++++ hw/cxl/cxl-host-stubs.c | 22 ++ hw/cxl/cxl-host.c | 263 ++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 509 ++++++++++++++++++++++++++++ hw/cxl/meson.build | 9 + hw/i386/acpi-build.c | 97 +++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 57 +++- hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 353 +++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 179 +++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci-host/gpex-acpi.c | 22 +- hw/pci/pci.c | 21 +- hw/pci/pcie_port.c | 25 ++ hw/ppc/spapr.c | 1 + include/hw/acpi/cxl.h | 28 ++ include/hw/arm/virt.h | 1 + include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 51 +++ include/hw/cxl/cxl_component.h | 206 +++++++++++ include/hw/cxl/cxl_device.h | 266 +++++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 14 + include/hw/pci/pci_bridge.h | 20 ++ include/hw/pci/pci_bus.h | 7 + include/hw/pci/pci_ids.h | 1 + include/hw/pci/pcie_port.h | 2 + qapi/machine.json | 15 + qemu-options.hx | 37 ++ softmmu/memory.c | 9 + softmmu/vl.c | 11 + tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes tests/qtest/cxl-test.c | 151 +++++++++ tests/qtest/meson.build | 4 + 54 files changed, 3645 insertions(+), 40 deletions(-) create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-host-stubs.c create mode 100644 hw/cxl/cxl-host.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT create mode 100644 tests/data/acpi/virt/CEDT create mode 100644 tests/qtest/cxl-test.c
Re: [PATCH v4 01/42] hw/pci/cxl: Add a CXL component type (interface)
In reply to Jonathan CameronQuoted 13 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL component is a hardware entity that implements CXL component registers from the CXL 2.0 spec (8.2.3). Currently these represent 3 general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)A CXL component can be conceptually thought of as a PCIe device with extra functionality when enumerated and enabled. For this reason, CXL does here, and will continue to add on to existing PCI code paths.
Quoted 7 lines
Host bridges will typically need to be handled specially and so they can implement this newly introduced interface or not. All other components should implement this interface. Implementing this interface allows the core PCI code to treat these devices as special where appropriate.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Quoted 14 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.
<snip>
Quoted 11 lines
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Is the core CXL support already in the upstream kernel or do you need a patched one?
Quoted 16 lines
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Quoted 8 lines
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
Ah right...
Quoted 7 lines
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?Example of new command line (with virt ITS patches ;)
One thing I think is missing in this series is some documentation. We've been historically bad at adding it for new devices but given the complexity of CXL I think we should certainly try to improve. I think a reasonable stab could be made from the commit messages in the series. I would suggest:
docs/system/devices/cxl.rst
And include:
- an brief overview of CXL
- kernel config options
and an some example command lines, like bellow:
Quoted 18 lines
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
Quoted 8 lines
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window
targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
So AIUI the above creates some CXL pmem devices that are part of the CXL root bus which itself is on the PCIe bus? Is the intention that reads/writes into the pmem by the guest end up visible in various forms in the memory backend files? Are memory backends required or can the address space be treated as volatile RAM that doesn't persist beyond a reset/reboot?
Maybe a simple diagram will help make things clearer?
Quoted 5 lines
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.
<snip>
Signature · 1 line
Alex Bennée
On Tue, 25 Jan 2022 13:55:29 +0000
Alex Bennée <[email protected]> wrote:
Hi Alex,
Thanks for taking a look so quickly!
Quoted 29 lines
Quoted 14 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.<snip>
Quoted 11 lines
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Quoted 2 lines
Is the core CXL support already in the upstream kernel or do you need a patched one?
Most of support is upstream for those features we are emulating so far, but a few elements are still work in progress.
The interleave feature has had a couple of revisions on list and Dan Williams posted a new version of that yesterday.
I haven't tested that version yet but will get to that shortly, this was done against the previous version on list.
I would expect this feature to go in this kernel cycle.
Quoted 42 lines
Quoted 17 lines
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Quoted 8 lines
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
Ah right...
Quoted 7 lines
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?Example of new command line (with virt ITS patches ;)
One thing I think is missing in this series is some documentation. We've been historically bad at adding it for new devices but given the complexity of CXL I think we should certainly try to improve. I think a reasonable stab could be made from the commit messages in the series. I would suggest:
Quoted 6 lines
docs/system/devices/cxl.rstAnd include:
- an brief overview of CXL
- kernel config options
Sure. Good idea, I'll write something up.
Quoted 30 lines
and an some example command lines, like bellow:
Quoted 18 lines
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \Quoted 8 lines
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window
targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
Quoted 2 lines
So AIUI the above creates some CXL pmem devices that are part of the CXL root bus which itself is on the PCIe bus?
That is possibly because of the 'hack' that pxb (pci-expander-bridge) does of pretending to be a root bus "on2 the pci bus (which I'm fairly sure you can't actually do in real PCI). Reality is that is just convenience for QEMU rather than anything you'd see on a real system. It's just easier to use PXB for this as it works on various architectures. From an OS point of view there isn't a driver associated with the PXB device, instead its just seen via ACPI description just like any other root bus.
The CXL root bus, in the sense of the one below which you can conceive of CXL host bridges sitting is host specific and not visible on the PCI bus. It's effectively part of the system interconnect routing the CXL memory read/write to the CXL root bridges. That configuration is considered static by the time any generic software sees it (early boot firmware may do the actual setup in a similar fashion to a system address map routing for multiple socket systems which is configured very early in boot and isn't something we'd want to emulate). The Fixed memory windows (CFMW) provide a static description of a particularly region of Physical Address space which will do interleaving across a predefined set of host bridges with a particular interleave granularity. They can also have QoS values, but so far I've skipped that in the emulation so they are all in QOS group 0. On real hardware you'd likely have quite a lot of CFMWs to cover combinations the OS might want to use - spanning a huge part of the physical address space.
Those CXL root bridges have spec defined controls over some features ( such as the interleave across the root ports below a particular root bridge) and an existence in ACPI that is an extension of what is done for PCI root bridges.
The CXL root ports are visible as PCI topology as are the CXL devices below them, including switches (which this patch set doesn't currently support)
From a Linux point of view we end up with two parallel topologies for CXL and PCI with cross points where the two line up (there end up being quite a few elements in CXL that don't exist in the PCI topology representation).
Quoted 3 lines
Is the intention that
reads/writes into the pmem by the guest end up visible in various forms in the memory backend files?
Yes. That's how I've been testing it so far. It's very nice to be able to prefill the files and hence know you are reading the location you expect.
Quoted 3 lines
Are memory backends required or can the
address space be treated as volatile RAM that doesn't persist beyond a reset/reboot?
We could potentially do that though it would limit testing somewhat, particularly when we come to label storage area (LSA) based setup which will "describe" the topology of a previous boot. It's hard to test something that is pretending to be persistent memory without being able to have the contents persist across boot.
Quoted 2 lines
Maybe a simple diagram will help make things clearer?
Sure - I'll give it a go though it won't be particularly simple!
Comments welcome as I would expect this will end up as part of the documentation.
Memory Address Map for CXL elements. Note where exactly these regions appear is Arch and platform dependent.
Base somewhere far up in the Host PA map.
_______________________________
| |
| CXL Host Bridge 0 Registers |
| CXL Host Bridge 1 Registers |
| ... | This bit is normal MMIO register space.
| CXL Host bridge N registers | including programmable interleave decoders
|______________________________| for interleave across root ports.
| |
....
| |
|______________________________|
| |
| CFMW 0, | Note that there can be multiple regions
| Interleave 2 way, targets | of memory within this 1TB which can be
| Hostbridge 0, Hostbridge 1 | interleaved differently: in the host bridges
| Granularity 16KiB, 1TB | across root ports or in switches below the root.
|______________________________| ports
| |
| CFMW 1, |
| Interleave 1 way, target |
| Hostbridge 0, 512GiB |
|______________________________|etc for all interleave combinations
configured, or built in to the
system before any generic software
sees it.
System Topology considering CFMW 0 only to keep this simple. x marks the match in each decoder level
Switches have more interleave decoders and other features that we haven't implemented yet in QEMU.
Address Read to CFMW0 base + N
_________________|________________
| |
| Host interconnect |
| Configured to route CFM |
| memory access to particular HB |
|_____x____________________________|
| |
Interleave Decoder |
Matches this HB |
| |
_______|__________ _____|____________
| | | |
| CXL HB 0 | | CXL HB 1 | Only exist in PCI (mostly)
| HB IntLv Decoder | | HB IntLv Decoder | via ACPI description
| PCI Root Bus 0c | | PCI Root Bus 0d |
|x_________________| |__________________| In CXL have MMIO
| | | | at location given in CEDT
| | | | CHBS entry (ACPI)
____________|___ __________|__ __|_________ ___|_________
| Root Port 0 | | Root Port 1 | | Root Port 2| | Root Port 3 |
| Appears in | | Appears in | | Appears in | | Appear in |
| PCI topology | | PCI Topology| | PCI Topo | | PCI Topo |
| As 0c:00.0 | | as 0c:01.0 | | as de:00.0 | | as de:01.0 |
|_______________| |_____________| |____________| |_____________|
| | | |
| | | |
_____|_________ ______|______ ______|_____ ______|_______
| x | | | | | | |
| CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 |
| | | | | | | |
| PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) |
| Decoder to go | | | | | | |
| from host PA | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0 |
| to device PA | | | | | | |
| PCI as 0d:00.0| | | | | | |
|_______________| |_____________| |____________| |______________| Backed by Backed by Backed by Backed by
file 0 file 1 file 2 file 3
LSA backed by additional files for each device (not yet supported)
So currently we have decoders as follows for each interleaved access.
1) CFMW decoder - fixed config so forms part of qemu command line.
2) Host bridge decoders - programmable decoders that the system software will program either based on user command or based on info from the Label Storage Area (not yet emulated)
3) Type 3 device decoders. Down to here the address used is the Host PA. These decoders convert to the local device PA (in simple case - drop some bits in the middle of the address)
Future patches will add decoders in switch upstream ports making the above diagram have another layer between root ports and the memory devices.
Note, we've focused for now on Persistent Memory devices as they are seen as an early and important usecase (and are the most complex one). But it should be straight forward to add volatile memory support and indeed that would be backed by RAM.
lspci -tv for above shows
-+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host Bridge (this is the cxl PXB)f
| \-OTHER STUFF
+-[0000:0c]-+-00.0-[0d]----00.0 Intel Corporation Device 0d93
| \-01.0-[0e]----00.0 Intel Corporation Device 0d93
\-[0000:de]-+-00.0-[df]----00.0 Intel Corporation Device 0d93
\-01.0-[e0]----00.0 Intel Corporation Device 0d93Where those Intel parts are the type 3 devices.
So everything should now be as clear as mud.
Thanks,
Jonathan
Quoted 8 lines
Quoted 6 lines
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.<snip>
Re: [PATCH v4 29/42] hw/cxl/host: Add support for CXL Fixed Memory Windows.
In reply to Jonathan CameronQuoted 12 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
This breaks the linux-user builds...
Quoted 8 lines
hw/cxl/meson.build +0
-3@@ -1,5 +1,9 @@ -softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( - 'cxl-component-utils.c', - 'cxl-device-utils.c',
hw/cxl/meson.build line 9
Quoted 10 lines
hw/cxl/meson.build +8
-1- 'cxl-mailbox-utils.c', +specific_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', + 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', + 'cxl-host.c', +)) +specific_ss.add(when: 'CONFIG_CXL', if_false: files( + 'cxl-host-stubs.c', ))
I think you want this:
softmmu_ss.add(when: 'CONFIG_CXL',
if_true: files(
'cxl-component-utils.c',
'cxl-device-utils.c',
'cxl-mailbox-utils.c',
'cxl-host.c'),
if_false: files(
'cxl-host-stubs.c',
)) # This is required as well:
softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))
Signature · 1 line
Alex Bennée
Re: [PATCH v4 31/42] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
In reply to Jonathan CameronQuoted 4 lines · Jonathan Cameron
This adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
This fails to build on all machines:
FAILED: qemu-system-mips64el
c++ -m64 -mcx16 -o qemu-system-mips64el qemu-system-mips64el.p/softmmu_main.c.o <snip> subprojects/libvhost-user/libvhost-user.a libmigration.fa libhwcore.fa libqom.fa libio.fa libcrypto.fa libauthz.fa libblockdev.fa libblock.fa libchardev.fa libqmp.fa @block.syms @qemu.syms /usr/lib/x86_64-linux-gnu/libpixman-1.so -lepoxy /usr/lib/x86_64-linux-gnu/libcapstone.so /usr/lib/x86_64-linux-gnu/libspice-server.so -Xlinker --dynamic-list=/home/alex/lsrc/qemu.git/plugins/qemu-plugins.symbols /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libpng16.so /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/x86_64-linux-gnu/libgnutls.so -lsasl2 /usr/lib/x86_64-linux-gnu/libpmem.so /usr/lib/x86_64-linux-gnu/libseccomp.so -lnuma -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lrdmacm -libverbs -libumad /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libzstd.so /usr/lib/x86_64-linux-gnu/libslirp.so /usr/lib/x86_64-linux-gnu/libglib-2.0.so -lvdeplug /usr/lib/x86_64-linux-gnu/libudev.so /usr/lib/x86_64-linux-gnu/libbpf.so /usr/lib/x86_64-linux-gnu/libasound.so /usr/lib/x86_64-linux-gnu/libpulse.so -lbrlapi /usr/lib/x86_64-linux-gnu/libvirglrenderer.so /usr/lib/x86_64-linux-gnu/libusb-1.0.so /usr/lib/x86_64-linux-gnu/libusbredirparser.so /usr/lib/x86_64-linux-gnu/libcacard.so -lncursesw -ltinfo -pthread -lgmodule-2.0 -lglib-2.0 /usr/lib/x86_64-linux-gnu/libgbm.so /usr/lib/x86_64-linux-gnu/libgtk-3.so /usr/lib/x86_64-linux-gnu/libgdk-3.so /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so /usr/lib/x86_64-linux-gnu/libpango-1.0.so /usr/lib/x86_64-linux-gnu/libharfbuzz.so /usr/lib/x86_64-linux-gnu/libatk-1.0.so /usr/lib/x86_64-linux-gnu/libcairo-gobject.so /usr/lib/x86_64-linux-gnu/libcairo.so /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so /usr/lib/x86_64-linux-gnu/libgio-2.0.so /usr/lib/x86_64-linux-gnu/libgobject-2.0.so /usr/lib/x86_64-linux-gnu/libvte-2.91.so /usr/lib/x86_64-linux-gnu/libX11.so @block.syms -lutil -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgmodule-2.0 -lglib-2.0 -lm -lgmodule-2.0 -lglib-2.0 /usr/lib/x86_64-linux-gnu/libiscsi.so -laio /usr/lib/x86_64-linux-gnu/libcurl.so -lgmodule-2.0 -lglib-2.0 -lbz2 /usr/lib/x86_64-linux-gnu/libacl.so /usr/lib/x86_64-linux-gnu/libgfapi.so /usr/lib/x86_64-linux-gnu/libglusterfs.so /usr/lib/x86_64-linux-gnu/libgfrpc.so /usr/lib/x86_64-linux-gnu/libgfxdr.so /usr/lib/x86_64-linux-gnu/libuuid.so /usr/lib/x86_64-linux-gnu/libnfs.so -lrbd -lrados /usr/lib/x86_64-linux-gnu/libssh.so -lstdc++ -Wl,--end-group /usr/bin/ld: /lib/x86_64-linux-gnu/libtirpc.so.3: warning: common of `rpc_createerr@@GLIBC_2.2.5' overridden by definition from /lib/x86_64-linux-gnu/libc.so.6
/usr/bin/ld: libcommon.fa.p/hw_pci-host_gpex-acpi.c.o: in function `acpi_dsdt_add_gpex':
/home/alex/lsrc/qemu.git/builds/all/../../hw/pci-host/gpex-acpi.c:191: undefined reference to `build_cxl_osc_method'
collect2: error: ld returned 1 exit status
[1600/2203] Linking target qemu-system-or1k
Quoted 63 lines
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Ben Widawsky <[email protected]>--- hw/arm/Kconfig | 1 + hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)hw/arm/Kconfig +1
-0@@ -29,6 +29,7 @@ config ARM_VIRT select ACPI_APEI select ACPI_VIOT select VIRTIO_MEM_SUPPORTED + select ACPI_CXL config CHEETAH boolhw/pci-host/gpex-acpi.c +14
-2@@ -5,6 +5,7 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" +#include "hw/acpi/cxl.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq) {@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); + dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); +if (is_cxl) { +struct Aml *pkg =aml_package (2 ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_STR" ,aml_unicode ("pxb Device" )));
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) · line 191
Quoted 6 lines
hw/pci-host/gpex-acpi.c +2
-1@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs)); -acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev);
Either we need an #ifdef gate on CONFIG_ACPI_CXL or possibly a stub implementation (with a g_assert_not_reached()).
Quoted 6 lines
hw/pci-host/gpex-acpi.c +3
-0+ }else { +acpi_dsdt_add_pci_osc (dev); + }aml_append (scope, dev); }
Signature · 1 line
Alex Bennée
Re: [PATCH v4 29/42] hw/cxl/host: Add support for CXL Fixed Memory Windows.
In reply to Alex BennéeQuoted 47 lines · On Tue, 25 Jan 2022 17:02:32 +0000 Alex Bennée <[email protected]> wrote:
Quoted 12 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
This breaks the linux-user builds...
Quoted 8 lines
hw/cxl/meson.build +0
-3@@ -1,5 +1,9 @@ -softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( - 'cxl-component-utils.c', - 'cxl-device-utils.c',hw/cxl/meson.build line 9
Quoted 10 lines
hw/cxl/meson.build +8
-1- 'cxl-mailbox-utils.c', +specific_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', + 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', + 'cxl-host.c', +)) +specific_ss.add(when: 'CONFIG_CXL', if_false: files( + 'cxl-host-stubs.c', ))I think you want this:
softmmu_ss.add(when: '
CONFIG_CXL',if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c'), if_false: files( 'cxl-host-stubs.c', ))
Quoted 2 lines
# This is required as well:
softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))
Ah. I'd started with the first block and got incomprehensible error message. Second block fixes it.
I'll not pretend to understand the 'why' :)
Thanks,
Jonathan
Quoted 2 lines
Re: [PATCH v4 31/42] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
In reply to Alex BennéeQuoted 84 lines · On Tue, 25 Jan 2022 17:15:58 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
This adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
This fails to build on all machines:
FAILED: qemu-system-mips64el
c++ -m64 -mcx16 -o qemu-system-mips64elqemu-system-mips64el.p/softmmu_main.c.o<snip> subprojects/libvhost-user/libvhost-user.a libmigration.fa libhwcore.fa libqom.fa libio.fa libcrypto.fa libauthz.fa libblockdev.fa libblock.fa libchardev.fa libqmp.fa @block.syms @qemu.syms/usr/lib/x86_64-linux-gnu/libpixman-1.so-lepoxy/usr/lib/x86_64-linux-gnu/libcapstone.so/usr/lib/x86_64-linux-gnu/libspice-server.so-Xlinker --dynamic-list=/home/alex/lsrc/qemu.git/plugins/qemu-plugins.symbols/usr/lib/x86_64-linux-gnu/libz.so/usr/lib/x86_64-linux-gnu/libpng16.so/usr/lib/x86_64-linux-gnu/libjpeg.so/usr/lib/x86_64-linux-gnu/libgnutls.so-lsasl2/usr/lib/x86_64-linux-gnu/libpmem.so/usr/lib/x86_64-linux-gnu/libseccomp.so-lnuma -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lrdmacm -libverbs -libumad/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libzstd.so/usr/lib/x86_64-linux-gnu/libslirp.so/usr/lib/x86_64-linux-gnu/libglib-2.0.so-lvdeplug/usr/lib/x86_64-linux-gnu/libudev.so/usr/lib/x86_64-linux-gnu/libbpf.so/usr/lib/x86_64-linux-gnu/libasound.so/usr/lib/x86_64-linux-gnu/libpulse.so-lbrlapi/usr/lib/x86_64-linux-gnu/libvirglrenderer.so/usr/lib/x86_64-linux-gnu/libusb-1.0.so/usr/lib/x86_64-linux-gnu/libusbredirparser.so/usr/lib/x86_64-linux-gnu/libcacard.so-lncursesw -ltinfo -pthread -lgmodule-2.0 -lglib-2.0/usr/lib/x86_64-linux-gnu/libgbm.so/usr/lib/x86_64-linux-gnu/libgtk-3.so/usr/lib/x86_64-linux-gnu/libgdk-3.so/usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so/usr/lib/x86_64-linux-gnu/libpango-1.0.so/usr/lib/x86_64-linux-gnu/libharfbuzz.so/usr/lib/x86_64-linux-gnu/libatk-1.0.so/usr/lib/x86_64-linux-gnu/libcairo-gobject.so/usr/lib/x86_64-linux-gnu/libcairo.so/usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so/usr/lib/x86_64-linux-gnu/libgio-2.0.so/usr/lib/x86_64-linux-gnu/libgobject-2.0.so/usr/lib/x86_64-linux-gnu/libvte-2.91.so/usr/lib/x86_64-linux-gnu/libX11.so@block.syms -lutil -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgmodule-2.0 -lglib-2.0 -lm -lgmodule-2.0 -lglib-2.0/usr/lib/x86_64-linux-gnu/libiscsi.so-laio/usr/lib/x86_64-linux-gnu/libcurl.so-lgmodule-2.0 -lglib-2.0 -lbz2/usr/lib/x86_64-linux-gnu/libacl.so/usr/lib/x86_64-linux-gnu/libgfapi.so/usr/lib/x86_64-linux-gnu/libglusterfs.so/usr/lib/x86_64-linux-gnu/libgfrpc.so/usr/lib/x86_64-linux-gnu/libgfxdr.so/usr/lib/x86_64-linux-gnu/libuuid.so/usr/lib/x86_64-linux-gnu/libnfs.so-lrbd -lrados/usr/lib/x86_64-linux-gnu/libssh.so-lstdc++ -Wl,--end-group/usr/bin/ld:/lib/x86_64-linux-gnu/libtirpc.so.3: warning: common of `rpc_createerr@@GLIBC_2.2.5' overridden by definition from/lib/x86_64-linux-gnu/libc.so.6
/usr/bin/ld:libcommon.fa.p/hw_pci-host_gpex-acpi.c.o: in function `acpi_dsdt_add_gpex':
/home/alex/lsrc/qemu.git/builds/all/../../hw/pci-host/gpex-acpi.c:191: undefined reference to `build_cxl_osc_method'
collect2: error: ld returned 1 exit status
[1600/2203] Linking target qemu-system-or1kQuoted 66 lines
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Ben Widawsky <[email protected]>--- hw/arm/Kconfig | 1 + hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)hw/arm/Kconfig +1
-0@@ -29,6 +29,7 @@ config ARM_VIRT select ACPI_APEI select ACPI_VIOT select VIRTIO_MEM_SUPPORTED + select ACPI_CXL config CHEETAH boolhw/pci-host/gpex-acpi.c +14
-2@@ -5,6 +5,7 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" +#include "hw/acpi/cxl.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq) {@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); + dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); +if (is_cxl) { +struct Aml *pkg =aml_package (2 ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_STR" ,aml_unicode ("pxb Device" )));@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs));
Quoted 5 lines
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) · line 191
Quoted 3 lines
hw/pci-host/gpex-acpi.c +2
-1-acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev);Either we need an #ifdef gate on
CONFIG_ACPI_CXL
error: attempt to use poisoned "CONFIG_ACPI_CXL"
Quoted 2 lines
or possibly a stub
implementation (with ag_assert_not_reached()).
That works.
Turns out I was too lazy thinking riscv was enough to exercise the not CXL support case.
Lesson learned - mips64el now in my standard config :) I'd not realized there were ACPI supporting MIPS machines...
Thanks,
Jonathan
Quoted 9 lines
Quoted 7 lines
hw/pci-host/gpex-acpi.c +3
-0+ }else { +acpi_dsdt_add_pci_osc (dev); + }aml_append (scope, dev); }
Re: [PATCH v4 31/42] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
Quoted 102 lines · Jonathan Cameron
Quoted 84 lines · On Tue, 25 Jan 2022 17:15:58 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
This adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
This fails to build on all machines:
FAILED: qemu-system-mips64el
c++ -m64 -mcx16 -o qemu-system-mips64elqemu-system-mips64el.p/softmmu_main.c.o<snip> subprojects/libvhost-user/libvhost-user.a libmigration.fa libhwcore.fa libqom.fa libio.fa libcrypto.fa libauthz.fa libblockdev.fa libblock.fa libchardev.fa libqmp.fa @block.syms @qemu.syms/usr/lib/x86_64-linux-gnu/libpixman-1.so-lepoxy/usr/lib/x86_64-linux-gnu/libcapstone.so/usr/lib/x86_64-linux-gnu/libspice-server.so-Xlinker --dynamic-list=/home/alex/lsrc/qemu.git/plugins/qemu-plugins.symbols/usr/lib/x86_64-linux-gnu/libz.so/usr/lib/x86_64-linux-gnu/libpng16.so/usr/lib/x86_64-linux-gnu/libjpeg.so/usr/lib/x86_64-linux-gnu/libgnutls.so-lsasl2/usr/lib/x86_64-linux-gnu/libpmem.so/usr/lib/x86_64-linux-gnu/libseccomp.so-lnuma -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lrdmacm -libverbs -libumad/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libzstd.so/usr/lib/x86_64-linux-gnu/libslirp.so/usr/lib/x86_64-linux-gnu/libglib-2.0.so-lvdeplug/usr/lib/x86_64-linux-gnu/libudev.so/usr/lib/x86_64-linux-gnu/libbpf.so/usr/lib/x86_64-linux-gnu/libasound.so/usr/lib/x86_64-linux-gnu/libpulse.so-lbrlapi/usr/lib/x86_64-linux-gnu/libvirglrenderer.so/usr/lib/x86_64-linux-gnu/libusb-1.0.so/usr/lib/x86_64-linux-gnu/libusbredirparser.so/usr/lib/x86_64-linux-gnu/libcacard.so-lncursesw -ltinfo -pthread -lgmodule-2.0 -lglib-2.0/usr/lib/x86_64-linux-gnu/libgbm.so/usr/lib/x86_64-linux-gnu/libgtk-3.so/usr/lib/x86_64-linux-gnu/libgdk-3.so/usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so/usr/lib/x86_64-linux-gnu/libpango-1.0.so/usr/lib/x86_64-linux-gnu/libharfbuzz.so/usr/lib/x86_64-linux-gnu/libatk-1.0.so/usr/lib/x86_64-linux-gnu/libcairo-gobject.so/usr/lib/x86_64-linux-gnu/libcairo.so/usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so/usr/lib/x86_64-linux-gnu/libgio-2.0.so/usr/lib/x86_64-linux-gnu/libgobject-2.0.so/usr/lib/x86_64-linux-gnu/libvte-2.91.so/usr/lib/x86_64-linux-gnu/libX11.so@block.syms -lutil -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgmodule-2.0 -lglib-2.0 -lm -lgmodule-2.0 -lglib-2.0/usr/lib/x86_64-linux-gnu/libiscsi.so-laio/usr/lib/x86_64-linux-gnu/libcurl.so-lgmodule-2.0 -lglib-2.0 -lbz2/usr/lib/x86_64-linux-gnu/libacl.so/usr/lib/x86_64-linux-gnu/libgfapi.so/usr/lib/x86_64-linux-gnu/libglusterfs.so/usr/lib/x86_64-linux-gnu/libgfrpc.so/usr/lib/x86_64-linux-gnu/libgfxdr.so/usr/lib/x86_64-linux-gnu/libuuid.so/usr/lib/x86_64-linux-gnu/libnfs.so-lrbd -lrados/usr/lib/x86_64-linux-gnu/libssh.so-lstdc++ -Wl,--end-group/usr/bin/ld:/lib/x86_64-linux-gnu/libtirpc.so.3: warning: common of `rpc_createerr@@GLIBC_2.2.5' overridden by definition from/lib/x86_64-linux-gnu/libc.so.6
/usr/bin/ld:libcommon.fa.p/hw_pci-host_gpex-acpi.c.o: in function `acpi_dsdt_add_gpex':
/home/alex/lsrc/qemu.git/builds/all/../../hw/pci-host/gpex-acpi.c:191: undefined reference to `build_cxl_osc_method'
collect2: error: ld returned 1 exit status
[1600/2203] Linking target qemu-system-or1kQuoted 66 lines
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Ben Widawsky <[email protected]>--- hw/arm/Kconfig | 1 + hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)hw/arm/Kconfig +1
-0@@ -29,6 +29,7 @@ config ARM_VIRT select ACPI_APEI select ACPI_VIOT select VIRTIO_MEM_SUPPORTED + select ACPI_CXL config CHEETAH boolhw/pci-host/gpex-acpi.c +14
-2@@ -5,6 +5,7 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie_host.h" +#include "hw/acpi/cxl.h" static void acpi_dsdt_add_pci_route_table (Aml *dev,uint32_t irq) {@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +bool is_cxl;if (!pci_bus_is_root (bus)) {continue ;@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) nr_pcie_buses = bus_num; } + is_cxl =pci_bus_is_cxl (bus); + dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); +if (is_cxl) { +struct Aml *pkg =aml_package (2 ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_string ("PNP0A03" ))); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_STR" ,aml_unicode ("pxb Device" )));@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs));Quoted 5 lines
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) · line 191
Quoted 3 lines
hw/pci-host/gpex-acpi.c +2
-1-acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev);Either we need an #ifdef gate on
CONFIG_ACPI_CXLerror: attempt to use poisoned "
CONFIG_ACPI_CXL"Quoted 2 lines
or possibly a stub
implementation (with ag_assert_not_reached()).That works.
Turns out I was too lazy thinking riscv was enough to exercise the not CXL support case.
Quoted 6 lines
Lesson learned - mips64el now in my standard config :) I'd not realized there were ACPI supporting MIPS machines...
Thanks,
Jonathan
Fine for development but before you post you really should pass make check with all machine types. Subtle interactions are not uncommon.
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
hw/pci-host/gpex-acpi.c +3
-0+ }else { +acpi_dsdt_add_pci_osc (dev); + }aml_append (scope, dev); }
Really awesome work Jonathan. Dan and I are wrapping up some of the kernel bits, so all I'll do for now is try to run this, but I hope to be able to review the parts I'm familiar with at least.
Quoted 14 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.
Quoted 8 lines
Main changes:
* The CXL fixed memory windows are now instantiated via a -cxl-fixed-memory-window command line option. As they are host level entities, not associated with a particular hardware entity a top level parameter seems the most natural way to describe them. This is also much closer to how it works on a real host than the previous assignment of a physical address window to all components along the CXL path.
Excellent.
Quoted 2 lines
* Dynamic host memory physical address space allocation both for the CXL host bridge MMIO space and the CFMWS windows.
I thought I had done the host bridge MMIO, but perhaps I was mistaken. Either way, this is an important step to support all platforms more generally.
Quoted 3 lines
* Interleaving support (based loosely on Philippe Mathieu-Daudé's earlier work on an interleaved memory device). Note this is rudimentary and low performance but it may be sufficient for test purposes.
I'll have to look at this further. I had some thoughts about how we might make this fast, but it would be more of fake interleaving. How low is "low"?
Quoted 5 lines
* Additional PCI and memory related utility functions needed for the interleaving.
* Various minor cleanup and increase in scope of tests.
* For now dropped the support for presenting CXL type 3 devices as memory devices in various QEMU interfaces.
What are the downsides to this? I only used the memory interface originally because it seemed like a natural fit, but looking back I'm not sure we gain much (though my memory is very lossy).
Quoted 27 lines
* Dropped the patch letting UID be different from
bus_nr. Whilst it may be a useful thing to have, we don't need it for this series and so should be handled separately.I've called out patches with major changes by marking them as co-developed or introducing them as new patches. The original memory window code has been dropped
After discussions at plumbers and more recently on the mailing list it was clear that there was interest in getting emulation for CXL 2.0 upstream in QEMU. This version resolves many of the outstanding issues and enables the following features:
* Support on both x86/pc and ARM/virt with relevant ACPI tables generated in QEMU.
* Host bridge based on the existing PCI Expander Bridge PXB.
* CXL fixed memory windows, allowing host to describe interleaving across multiple CXL host bridges.
* pxb-cxl CXL host bridge support including MMIO region for control and HDM (Host manage device memory - basically interleaving / routing) decoder configuration.
* Basic CXL Root port support.
* CXL Type 3 device support with persistent memory regions (backed by hostmem backend).
* PulledMAINTAINERSentry out to a separate patch and add myself as a co-maintainer at Ben's suggestion.
Quoted 8 lines
Big TODOs:
* Volatile memory devices (easy but it's more code so left for now).
* Switch support.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Main host bridge support (not a priority for me...)
I originally cared about this for the sake of making a system more realistic. I now believe we should drop this entirely.
Quoted 40 lines
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
Quoted 5 lines
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?
Sure.
Quoted 14 lines
Example of new command line (with virt ITS patches ;)
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \
Is align actually necessary here?
Quoted 1 line
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
Did you mean to put this in there? Is it somehow used internally?
Quoted 4 lines
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
Quoted 8 lines
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
I assume interleave-ways is based on the number of targets. For testing purposes it might be nice to add the flags as well (perhaps it's there).
Quoted 35 lines
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.With Ben's CXL region patches (v3 shortly) plus fixes as discussed on list, Linux commands to bring up a 4 way interleave is:
cd
/sys/bus/cxl/devices/
region=$(cat decoder0.1/create_region)
echo $region > decoder0.1/create_region
ls -lh//Note the order of devices and adjust the following to make sure they //are in order across the 4 root ports. Easy to do in a tool, but //not easy to paste in a cover letter.
cd region0.1\:0
echo 4 >interleave_ways
echo mem2 > target0
echo mem3 > target1
echo mem0 > target2
echo mem1 > target3
echo $((1024<<20)) > size
echo 4096 >interleave_granularity
echo region0.1:0 >/sys/bus/cxl/drivers/cxl_region/bindTested with devmem2 and files with known content. Kernel tree was based on previous version of the region patches from Ben with various fixes. As Dan just posted an updated version next job on my list is to test that.
Thanks to Shameer for his help with reviewing the new stuff before posting.
Quoted 6 lines
I'll post a git tree shortly for any who prefer that to lots of emails ;)
Thanks,
Jonathan
Thanks again!
Ben
[snip]
Re: [PATCH v4 29/42] hw/cxl/host: Add support for CXL Fixed Memory Windows.
In reply to Jonathan CameronQuoted 50 lines · Jonathan Cameron
Quoted 47 lines · On Tue, 25 Jan 2022 17:02:32 +0000 Alex Bennée <[email protected]> wrote:
Quoted 12 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges.
This breaks the linux-user builds...
Quoted 8 lines
hw/cxl/meson.build +0
-3@@ -1,5 +1,9 @@ -softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( - 'cxl-component-utils.c', - 'cxl-device-utils.c',hw/cxl/meson.build line 9
Quoted 10 lines
hw/cxl/meson.build +8
-1- 'cxl-mailbox-utils.c', +specific_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', + 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', + 'cxl-host.c', +)) +specific_ss.add(when: 'CONFIG_CXL', if_false: files( + 'cxl-host-stubs.c', ))I think you want this:
softmmu_ss.add(when: '
CONFIG_CXL',if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c'), if_false: files( 'cxl-host-stubs.c', ))
Quoted 5 lines
Quoted 2 lines
# This is required as well:
softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))Ah. I'd started with the first block and got incomprehensible error message. Second block fixes it.
https://qemu.readthedocs.io/en/latest/devel/build-system.html#stage-2-meson
documents some of this although we could probably do a better job of sign posting the bits in the root meson file.
Quoted 9 lines
I'll not pretend to understand the 'why' :)
Thanks,
Jonathan
Quoted 2 lines
Signature · 1 line
Alex Bennée
Quoted 174 lines · Ben Widawsky
Really awesome work Jonathan. Dan and I are wrapping up some of the kernel bits, so all I'll do for now is try to run this, but I hope to be able to review the parts I'm familiar with at least.
Quoted 14 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.Quoted 8 lines
Main changes:
* The CXL fixed memory windows are now instantiated via a -cxl-fixed-memory-window command line option. As they are host level entities, not associated with a particular hardware entity a top level parameter seems the most natural way to describe them. This is also much closer to how it works on a real host than the previous assignment of a physical address window to all components along the CXL path.Excellent.
Quoted 2 lines
* Dynamic host memory physical address space allocation both for the CXL host bridge MMIO space and the CFMWS windows.
I thought I had done the host bridge MMIO, but perhaps I was mistaken. Either way, this is an important step to support all platforms more generally.
Quoted 3 lines
* Interleaving support (based loosely on Philippe Mathieu-Daudé's earlier work on an interleaved memory device). Note this is rudimentary and low performance but it may be sufficient for test purposes.
I'll have to look at this further. I had some thoughts about how we might make this fast, but it would be more of fake interleaving. How low is "low"?
Quoted 5 lines
* Additional PCI and memory related utility functions needed for the interleaving.
* Various minor cleanup and increase in scope of tests.
* For now dropped the support for presenting CXL type 3 devices as memory devices in various QEMU interfaces.What are the downsides to this? I only used the memory interface originally because it seemed like a natural fit, but looking back I'm not sure we gain much (though my memory is very lossy).
Quoted 27 lines
* Dropped the patch letting UID be different from
bus_nr. Whilst it may be a useful thing to have, we don't need it for this series and so should be handled separately.I've called out patches with major changes by marking them as co-developed or introducing them as new patches. The original memory window code has been dropped
After discussions at plumbers and more recently on the mailing list it was clear that there was interest in getting emulation for CXL 2.0 upstream in QEMU. This version resolves many of the outstanding issues and enables the following features:
* Support on both x86/pc and ARM/virt with relevant ACPI tables generated in QEMU.
* Host bridge based on the existing PCI Expander Bridge PXB.
* CXL fixed memory windows, allowing host to describe interleaving across multiple CXL host bridges.
* pxb-cxl CXL host bridge support including MMIO region for control and HDM (Host manage device memory - basically interleaving / routing) decoder configuration.
* Basic CXL Root port support.
* CXL Type 3 device support with persistent memory regions (backed by hostmem backend).
* PulledMAINTAINERSentry out to a separate patch and add myself as a co-maintainer at Ben's suggestion.Quoted 8 lines
Big TODOs:
* Volatile memory devices (easy but it's more code so left for now).
* Switch support.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Main host bridge support (not a priority for me...)I originally cared about this for the sake of making a system more realistic. I now believe we should drop this entirely.
Quoted 40 lines
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
Quoted 5 lines
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?Sure.
Quoted 14 lines
Example of new command line (with virt ITS patches ;)
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \Is align actually necessary here?
Quoted 1 line
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
Did you mean to put this in there? Is it somehow used internally?
Quoted 4 lines
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \Quoted 8 lines
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
Quoted 3 lines
I assume interleave-ways is based on the number of targets. For testing purposes it might be nice to add the flags as well (perhaps it's there).
This requires cxl=on machine arg now btw.
Quoted 48 lines
Quoted 35 lines
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.With Ben's CXL region patches (v3 shortly) plus fixes as discussed on list, Linux commands to bring up a 4 way interleave is:
cd
/sys/bus/cxl/devices/
region=$(cat decoder0.1/create_region)
echo $region > decoder0.1/create_region
ls -lh//Note the order of devices and adjust the following to make sure they //are in order across the 4 root ports. Easy to do in a tool, but //not easy to paste in a cover letter.
cd region0.1\:0
echo 4 >interleave_ways
echo mem2 > target0
echo mem3 > target1
echo mem0 > target2
echo mem1 > target3
echo $((1024<<20)) > size
echo 4096 >interleave_granularity
echo region0.1:0 >/sys/bus/cxl/drivers/cxl_region/bindTested with devmem2 and files with known content. Kernel tree was based on previous version of the region patches from Ben with various fixes. As Dan just posted an updated version next job on my list is to test that.
Thanks to Shameer for his help with reviewing the new stuff before posting.
Quoted 6 lines
I'll post a git tree shortly for any who prefer that to lots of emails ;)
Thanks,
Jonathan
Thanks again!
Ben[snip]
Quoted 29 lines · On Tue, 25 Jan 2022 15:55:03 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 28 lines · Ben Widawsky
Really awesome work Jonathan. Dan and I are wrapping up some of the kernel bits, so all I'll do for now is try to run this, but I hope to be able to review the parts I'm familiar with at least.
Quoted 22 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.Looking in particular for:
* Review of the PCI interactions
* x86 and ARM machine interactions (particularly the memory maps)
* Review of the interleaving approach - is the basic idea acceptable?
* Review of the command line interface.
* CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially.Main changes:
* The CXL fixed memory windows are now instantiated via a -cxl-fixed-memory-window command line option. As they are host level entities, not associated with a particular hardware entity a top level parameter seems the most natural way to describe them. This is also much closer to how it works on a real host than the previous assignment of a physical address window to all components along the CXL path.
Quoted 7 lines
Quoted 7 lines
Excellent.
Quoted 2 lines
* Dynamic host memory physical address space allocation both for the CXL host bridge MMIO space and the CFMWS windows.
I thought I had done the host bridge MMIO, but perhaps I was mistaken. Either way, this is an important step to support all platforms more generally.
That is comment probably more general that it needs to be ;) I can't remember how much of this was done using fixed addresses but it all got rewritten anyway. As you've probably noticed I got lazy on change logs because lots of changes had minor influence on a large set of patches making them fiddly to document.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
* Interleaving support (based loosely on Philippe Mathieu-Daudé's earlier work on an interleaved memory device). Note this is rudimentary and low performance but it may be sufficient for test purposes.
I'll have to look at this further. I had some thoughts about how we might make this fast, but it would be more of fake interleaving. How low is "low"?
The question becomes what is the purpose. We aren't doing this emulation to provide a realistic system, but rather to just have something we can test kernel etc tooling against.
I'm not yet in a position to do perf tests, but given it's walking the decoders ever time it is never going to be great. We could look at caching the walks, but then a bunch of locking comes into play. Mind you right now I suspect we'll have all sorts of nasty issues if devices are hot removed whilst transactions are in flight. Tidying that up was one of the TODOs I forgot to list.
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
* Additional PCI and memory related utility functions needed for the interleaving.
* Various minor cleanup and increase in scope of tests.
* For now dropped the support for presenting CXL type 3 devices as memory devices in various QEMU interfaces.What are the downsides to this? I only used the memory interface originally because it seemed like a natural fit, but looking back I'm not sure we gain much (though my memory is very lossy).
Main downside is simply that people might expect to see all their memory devices in some of the info commands and right now the CXL ones don't show up at all. That doesn't necessarily mean we need to use the existing framework, but it might make sense to extend the tools a little to include the CXL attached memories.
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
* Dropped the patch letting UID be different from
bus_nr. Whilst it may be a useful thing to have, we don't need it for this series and so should be handled separately.I've called out patches with major changes by marking them as co-developed or introducing them as new patches. The original memory window code has been dropped
After discussions at plumbers and more recently on the mailing list it was clear that there was interest in getting emulation for CXL 2.0 upstream in QEMU. This version resolves many of the outstanding issues and enables the following features:
* Support on both x86/pc and ARM/virt with relevant ACPI tables generated in QEMU.
* Host bridge based on the existing PCI Expander Bridge PXB.
* CXL fixed memory windows, allowing host to describe interleaving across multiple CXL host bridges.
* pxb-cxl CXL host bridge support including MMIO region for control and HDM (Host manage device memory - basically interleaving / routing) decoder configuration.
* Basic CXL Root port support.
* CXL Type 3 device support with persistent memory regions (backed by hostmem backend).
* PulledMAINTAINERSentry out to a separate patch and add myself as a co-maintainer at Ben's suggestion.Big TODOs:
* Volatile memory devices (easy but it's more code so left for now).
* Switch support.
* Hotplug? May not need much but it's not tested yet!
* More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check.
* Main host bridge support (not a priority for me...)
Quoted 2 lines
Quoted 2 lines
I originally cared about this for the sake of making a system more realistic. I now believe we should drop this entirely.
Cool. That avoids some mess around the type of a CXL host bridge that I hadn't figured a clean way around (short of checking against all implemented possibilities)
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
* Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation.
Why do we want QEMU emulation of CXL?
As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches.
Other features on the qemu-list that build on these include PCI-DOE
/CDAT support from the Avery Design team further showing how this
code is useful. Whilst not directly related this is also the test
platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
utilizes and extends those technologies and is likely to be an early
adopter.
Refs:
CMA Kernel: https://lore.kernel.org/all/[email protected]/
CMA Qemu: https://lore.kernel.org/qemu-devel/[email protected]/
DOE Qemu: https://lore.kernel.org/qemu-devel/[email protected]/As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :)
Ben mentioned a few other good reasons in v3:
https://lore.kernel.org/qemu-devel/[email protected]/The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
All comments welcome.
Ben - I lifted one patch from your git tree that didn't have a
Sign-off. hw/cxl/component Add a dumb HDM decoder handler
Could you confirm you are happy for one to be added?Sure.
Cool. I'll put that in for v5.
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Example of new command line (with virt ITS patches ;)
qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
-m 4g,maxmem=8G,slots=8 \
...
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M,align=256M \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M,align=256M \
Quoted 1 line
Quoted 1 line
Is align actually necessary here?
Err. That's been in my config a long time. IIRC I ran into problems with your earlier versions when I didn't provide it, but might not be necessary any more. Good spot.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
-object memory-backend-file,id=tt,share=on,mem-path=/tmp/tt.raw,size=1g \
Did you mean to put this in there? Is it somehow used internally?
Oops. Nope - that is bad editing on my part - was part of a nvdimm test I was running to make sure I didn't accidentally break anything in the normal file backed hostmem paths.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
-device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
-device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
-device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
-device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
-device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
-device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
-cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
-cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
Quoted 2 lines
Quoted 2 lines
I assume interleave-ways is based on the number of targets. For testing purposes it might be nice to add the flags as well (perhaps it's there).
Good point, though not implemented yet. Easy thing to add as a later step.
Quoted 3 lines
Quoted 1 line
This requires cxl=on machine arg now btw.
Absolutely. Though I wonder if that is worth bothering with. We could just always
reserve the memory for the host bridge MMIO and run through building empty CEDT (or add sanity checks in the CEDT build code on no HB and no CFMWS == no CEDT).
I'd be interested on what various machine maintainers think on this.
Longer term it would be nice to generally cleanup the machine PA space code to use some sort of allocator because we just added another layer of if/else to some already deep trees based on all the optional parts.
For ARM there is one suitable for the MMIO regions (given the hack to just allocated space for 16) but the CFMWs / device memory etc are just as nasty to deal with as on i386/pc
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
First CFMWS suitable for 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge).
targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.With Ben's CXL region patches (v3 shortly) plus fixes as discussed on list, Linux commands to bring up a 4 way interleave is:
cd
/sys/bus/cxl/devices/
region=$(cat decoder0.1/create_region)
echo $region > decoder0.1/create_region
ls -lh//Note the order of devices and adjust the following to make sure they //are in order across the 4 root ports. Easy to do in a tool, but //not easy to paste in a cover letter.
cd region0.1\:0
echo 4 >interleave_ways
echo mem2 > target0
echo mem3 > target1
echo mem0 > target2
echo mem1 > target3
echo $((1024<<20)) > size
echo 4096 >interleave_granularity
echo region0.1:0 >/sys/bus/cxl/drivers/cxl_region/bindTested with devmem2 and files with known content. Kernel tree was based on previous version of the region patches from Ben with various fixes. As Dan just posted an updated version next job on my list is to test that.
Thanks to Shameer for his help with reviewing the new stuff before posting.
I'll post a git tree shortly for any who prefer that to lots of emails ;)
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Thanks,
Jonathan
Thanks again!
Ben
You are welcome.
Been an interesting learning curve as all my past QEMU work was rather more superficial than this.
Jonathan
Quoted 4 lines
Quoted 4 lines
[snip]
Re: [PATCH v4 31/42] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
In reply to Jonathan CameronQuoted 14 lines · Jonathan Cameron
Quoted 11 lines · On Tue, 25 Jan 2022 17:15:58 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
This adds code to instantiate the slightly extended ACPI root port description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
This fails to build on all machines:
FAILED: qemu-system-mips64el
c++ -m64 -mcx16 -o qemu-system-mips64el
<snip>
Quoted 6 lines
Quoted 6 lines
/usr/lib/x86_64-linux-gnu/libssh.so-lstdc++ -Wl,--end-group
/usr/bin/ld:/lib/x86_64-linux-gnu/libtirpc.so.3: warning: common ofrpc_createerr@@GLIBC_2.2.5' overridden by definition from /lib/x86_64-linux-gnu/libc.so.6 /usr/bin/ld: libcommon.fa.p/hw_pci-host_gpex-acpi.c.o: in functionacpi_dsdt_add_gpex':
/home/alex/lsrc/qemu.git/builds/all/../../hw/pci-host/gpex-acpi.c:191: undefined reference to `build_cxl_osc_method'
collect2: error: ld returned 1 exit status
[1600/2203] Linking target qemu-system-or1k
<snip>
Quoted 11 lines
Quoted 9 lines
hw/pci-host/gpex-acpi.c void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) · line 191
Quoted 7 lines
hw/pci-host/gpex-acpi.c +2
-1@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) cfg->pio.base, 0, 0, 0);aml_append (dev,aml_name_decl ("_CRS" , crs)); -acpi_dsdt_add_pci_osc (dev); +if (is_cxl) { +build_cxl_osc_method (dev);Either we need an #ifdef gate on
CONFIG_ACPI_CXLerror: attempt to use poisoned "
CONFIG_ACPI_CXL"
Hmm I'm not sure why that happened. We generally poison symbols to stop making configuration changes objects that are shared between binaries. I guess softmmu_ss must be shared then. Paolo?
Quoted 10 lines
Quoted 3 lines
or possibly a stub
implementation (with ag_assert_not_reached()).That works.
Turns out I was too lazy thinking riscv was enough to exercise the not CXL support case.
Lesson learned - mips64el now in my standard config :) I'd not realized there were ACPI supporting MIPS machines...
It's worth making sure your gitlab account is setup to run the CI loop because it's the first thing I do when I review a series ;-)
https://gitlab.com/stsquad/qemu/-/pipelines/456700583
Signature · 1 line
Alex Bennée
Re: [PATCH v4 02/42] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
In reply to Jonathan CameronQuoted 75 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The
cxl_pciheader manages bits associated with PCI, specifically the DVSEC and related fields. Thecxl_component.hvariant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] registers. Per spec, the registers are little endian.None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 212 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 3 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 16 +++ include/hw/cxl/cxl_component.h | 196 ++++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++ 8 files changed, 570 insertions(+)hw/Kconfig +1
-0@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfighw/cxl/Kconfig new file +3
-0@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESShw/cxl/cxl-component-utils.c new file +11
-0@@ -0,0 +1,212 @@ +/* +* CXL Utility library for components +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h"
hw/cxl/cxl-component-utils.c line 21
Quoted 10 lines
hw/cxl/cxl-component-utils.c new file +10
-0+#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" + +static uint64_t cxl_cache_mem_read_reg (void *opaque,hwaddr offset, +unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; + +assert (size ==4 );
You assert here but bellow:
Quoted 18 lines
hw/cxl/cxl-component-utils.c new file +18
-0+ +/* +* 8.2.3 +* The access restrictions specified in Section 8.2.2 also apply to CXL 2.0 +* Component Registers. +* +* 8.2.2 +* • A 32 bit register shall be accessed as a 4 Bytes quantity. Partial +* reads are not permitted. +* • A 64 bit register shall be accessed as a 8 Bytes quantity. Partial +* reads are not permitted. +* +* As of the spec defined today, only 4 byte registers exist. +*/ +static const MemoryRegionOps cache_mem_ops = { + .read = cxl_cache_mem_read_reg, + .write = cxl_cache_mem_write_reg, + .endianness =DEVICE_LITTLE_ENDIAN ,
hw/cxl/cxl-component-utils.c line 71
Quoted 10 lines
hw/cxl/cxl-component-utils.c new file +10
-0+ .valid = { + .min_access_size = 4, + .max_access_size = 8, + .unaligned = false, + }, + .impl = { + .min_access_size = 4, + .max_access_size = 4, + }, +};
You have constrained the access to 4 so you will only see 4 bytes accesses. If it is valid for the guest to access 64bit words then it would be better to no-op that case and maybe LOG_UNIMP the fact.
Otherwise the rest looks ok to me:
Signature · 1 line
Alex Bennée
Re: [PATCH v4 03/42] MAINTAINERS: Add entry for Compute Express Link Emulation
In reply to Jonathan CameronQuoted 7 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The CXL emulation will be jointly maintained by Ben Widawsky and Jonathan Cameron. Broken out as a separate patch to improve visibility.
Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v4 04/42] hw/cxl/device: Introduce a CXL device (8.2.8)
In reply to Jonathan CameronQuoted 86 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.
This new device type is specifically catered towards the eventual implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_device.h | 157 ++++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+)include/hw/cxl/cxl.h +1
-0@@ -12,5 +12,6 @@#include "cxl_pci.h" #include "cxl_component.h" +#include "cxl_device.h" #endif include/hw/cxl/cxl_device.h new file +51
-0@@ -0,0 +1,157 @@ +/* +* QEMU CXL Devices +* +* Copyright (c) 2020 Intel +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#ifndef CXL_DEVICE_H +#define CXL_DEVICE_H + +#include "hw/register.h" + +/* +* The following is how a CXL device's MMIO space is laid out. The only +* requirement from the spec is that the capabilities array and the capability +* headers start at offset 0 and are contiguously packed. The headers themselves +* provide offsets to the register fields. For this emulation, registers will +* start at offset 0x80 (m == 0x80). No secondary mailbox is implemented which +* means that n = m + sizeof(mailbox registers) + sizeof(device registers). +* +* This is roughly described in 8.2.8 Figure 138 of the CXL 2.0 spec. +* +* +---------------------------------+ +* | | +* | Memory Device Registers | +* | | +* n + PAYLOAD_SIZE_MAX ----------------------------------- +* ^ | | +* | | | +* | | | +* | | | +* | | | +* | | Mailbox Payload | +* | | | +* | | | +* | | | +* | ----------------------------------- +* | | Mailbox Registers | +* | | | +* n ----------------------------------- +* ^ | | +* | | Device Registers | +* | | | +* m ----------------------------------> +* ^ | Memory Device Capability Header| +* | ----------------------------------- +* | | Mailbox Capability Header | +* | -------------- -------------------- +* | | Device Capability Header |
include/hw/cxl/cxl_device.h line 61
Quoted 10 lines
include/hw/cxl/cxl_device.h new file +10
-0+* | ----------------------------------- +* | | | +* | | | +* | | Device Cap Array[0..n] | +* | | | +* | | | +* | | +* 0 +---------------------------------+ +* +*/
Excellent diagram ;-)
Quoted 30 lines
include/hw/cxl/cxl_device.h new file +30
-0+ +#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10/* Figure 138 */ +#define CXL_DEVICE_CAP_REG_SIZE 0x10/* 8.2.8.2 */ +#define CXL_DEVICE_CAPS_MAX 4/* 8.2.8.2.1 + 8.2.8.5 */ + +#define CXL_DEVICE_REGISTERS_OFFSET 0x80/* Read comment above */ +#define CXL_DEVICE_REGISTERS_LENGTH 0x8/* 8.2.8.3.1 */ + +#define CXL_MAILBOX_REGISTERS_OFFSET \ + (CXL_DEVICE_REGISTERS_OFFSET + CXL_DEVICE_REGISTERS_LENGTH) +#define CXL_MAILBOX_REGISTERS_SIZE 0x20/* 8.2.8.4, Figure 139 */ +#define CXL_MAILBOX_PAYLOAD_SHIFT 11 +#define CXL_MAILBOX_MAX_PAYLOAD_SIZE (1 << CXL_MAILBOX_PAYLOAD_SHIFT) +#define CXL_MAILBOX_REGISTERS_LENGTH \ + (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) + +typedef struct cxl_device_state { + MemoryRegion device_registers; + + /* mmio for device capabilities array - 8.2.8.2 */ + MemoryRegion caps; + + /* mmio for the device status registers 8.2.8.3 */ + MemoryRegion device; + + /* mmio for the mailbox registers 8.2.8.4 */ + MemoryRegion mailbox; + + /* memory region for persistent memory, HDM */ + uint64_t pmem_size;
include/hw/cxl/cxl_device.h line 101
Quoted 10 lines
include/hw/cxl/cxl_device.h new file +10
-0+} CXLDeviceState; + +/* Initialize the register block for a device */ +void cxl_device_register_block_init (Object *obj,CXLDeviceState *dev); + +/* Set up default values for the register block */ +void cxl_device_register_init_common (CXLDeviceState *dev); + +/* CXL 2.0 - 8.2.8.1 */ +REG32 (CXL_DEV_CAP_ARRAY ,0 )/* 48b!?!?! */
What does this comment mean? A 48 bit register stuffed in a 32 bit one? Doesn't seem right.
include/hw/cxl/cxl_device.h line 105
Quoted 4 lines
include/hw/cxl/cxl_device.h new file +4
-0+ FIELD(CXL_DEV_CAP_ARRAY, CAP_ID, 0, 16) + FIELD(CXL_DEV_CAP_ARRAY, CAP_VERSION, 16, 8) +REG32(CXL_DEV_CAP_ARRAY2, 4) /* We're going to pretend it's 64b */ +FIELD (CXL_DEV_CAP_ARRAY2 ,CAP_COUNT ,0 ,16 )
I'm confused why you don't treat it as a single register it is in the spec.
Quoted 38 lines
include/hw/cxl/cxl_device.h new file +38
-0+ +/* +* Helper macro to initialize capability headers for CXL devices. +* +* In the 8.2.8.2, this is listed as a 128b register, but in 8.2.8, it says: +* > No registers defined in Section 8.2.8 are larger than 64-bits wide so that +* > is the maximum access size allowed for these registers. If this rule is not +* > followed, the behavior is undefined +* +* Here we've chosen to make it 4 dwords. The spec allows any pow2 multiple +* access to be used for a register (2 qwords, 8 words, 128 bytes). +*/ +#define CXL_DEVICE_CAPABILITY_HEADER_REGISTER (n, offset) \ + REG32(CXL_DEV_##n##_CAP_HDR0, offset) \ + FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_ID, 0, 16) \ + FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_VERSION, 16, 8) \ + REG32(CXL_DEV_##n##_CAP_HDR1, offset + 4) \ + FIELD(CXL_DEV_##n##_CAP_HDR1, CAP_OFFSET, 0, 32) \ + REG32(CXL_DEV_##n##_CAP_HDR2, offset + 8) \ + FIELD(CXL_DEV_##n##_CAP_HDR2, CAP_LENGTH, 0, 32) + +CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET) +CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ + CXL_DEVICE_CAP_REG_SIZE) + +REG32(CXL_DEV_MAILBOX_CAP, 0) + FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5) + FIELD(CXL_DEV_MAILBOX_CAP, INT_CAP, 5, 1) + FIELD(CXL_DEV_MAILBOX_CAP, BG_INT_CAP, 6, 1) + FIELD(CXL_DEV_MAILBOX_CAP, MSI_N, 7, 4) + +REG32(CXL_DEV_MAILBOX_CTRL, 4) + FIELD(CXL_DEV_MAILBOX_CTRL, DOORBELL, 0, 1) + FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1) +FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) + +/* XXX: actually a 64b register */ +REG32 (CXL_DEV_MAILBOX_STS ,0x10 )
include/hw/cxl/cxl_device.h line 153
Quoted 10 lines
include/hw/cxl/cxl_device.h new file +10
-0+ FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1) + FIELD(CXL_DEV_MAILBOX_STS, ERRNO, 32, 16) +FIELD (CXL_DEV_MAILBOX_STS ,VENDOR_ERRNO ,48 ,16 ) + +/* XXX: actually a 64b register */ +REG32(CXL_DEV_BG_CMD_STS, 0x18) + FIELD(CXL_DEV_BG_CMD_STS, BG, 0, 16) + FIELD(CXL_DEV_BG_CMD_STS, DONE, 16, 7) + FIELD(CXL_DEV_BG_CMD_STS, ERRNO, 32, 16) +FIELD (CXL_DEV_BG_CMD_STS ,VENDOR_ERRNO ,48 ,16 )
Again is there a reason not to use REG64? I can see the need to split 128 bit registers but not 64 bit ones.
include/hw/cxl/cxl_device.h line 157
Quoted 4 lines
include/hw/cxl/cxl_device.h new file +4
-0+ +REG32(CXL_DEV_CMD_PAYLOAD, 0x20) + +#endif
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v4 05/42] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Jonathan CameronQuoted 69 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This implements all device MMIO up to the first capability. That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 105 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_device.h | 28 +++++++++- 3 files changed, 133 insertions(+), 1 deletion(-)hw/cxl/cxl-device-utils.c new file +46
-0@@ -0,0 +1,105 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +/* +* Device registers have no restrictions per the spec, and so fall back to the +* default memory mapped register rules in 8.2: +* Software shall use CXL.io Memory Read and Write to access memory mapped +* register defined in this section. Unless otherwise specified, software +* shall restrict the accesses width based on the following: +* • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes +* quantity. +* • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 +* Bytes +* • The address shall be a multiple of the access width, e.g. when +* accessing a register as a 4 Byte quantity, the address shall be +* multiple of 4. +* • The accesses shall map to contiguous bytes.If these rules are not +* followed, the behavior is undefined +*/ + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +return cxl_dstate->caps_reg_state32[offset /4 ]; +} + +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL ,/* status register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN ,
hw/cxl/cxl-device-utils.c line 56
Quoted 10 lines
hw/cxl/cxl-device-utils.c new file +10
-0+ .valid = { + .min_access_size = 1, + .max_access_size = 8, + .unaligned = false, + }, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + }, +};
I think for >64 bit registers you need to use the read_with_attrs
Quoted 105 lines
hw/cxl/cxl-device-utils.c new file +49
-0+ +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL ,/* caps registers are read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = { + .min_access_size =4 , + .max_access_size =4 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ +memory_region_init (&cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MMIO_SIZE )); + +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_CAPS_SIZE ); +memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate, +"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->device); +} + +static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =1 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); + +cxl_device_cap_init (cxl_dstate,DEVICE ,1 ); +device_reg_init_common (cxl_dstate); +}hw/cxl/meson.build +1
-0@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))include/hw/cxl/cxl_device.h +20
-1@@ -63,6 +63,8 @@#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10/* Figure 138 */ #define CXL_DEVICE_CAP_REG_SIZE 0x10/* 8.2.8.2 */ #define CXL_DEVICE_CAPS_MAX 4/* 8.2.8.2.1 + 8.2.8.5 */ +#define CXL_CAPS_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE * (CXL_DEVICE_CAPS_MAX + 1))/* +1 for header */ #define CXL_DEVICE_REGISTERS_OFFSET 0x80/* Read comment above */ #define CXL_DEVICE_REGISTERS_LENGTH 0x8/* 8.2.8.3.1 */ @@ -75,11 +77,18 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) +#define CXL_MMIO_SIZE \ + (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \ + CXL_MAILBOX_REGISTERS_LENGTH) + typedef struct cxl_device_state { MemoryRegion device_registers; /* mmio for device capabilities array - 8.2.8.2 */ - MemoryRegion caps; + struct { + MemoryRegion caps; + uint32_t caps_reg_state32[CXL_CAPS_SIZE / 4]; + }; /* mmio for the device status registers 8.2.8.3 */ MemoryRegion device;@@ -128,6 +137,23 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +#define cxl_device_cap_init (dstate, reg, cap_id) \ + do { \ + uint32_t *cap_hdrs = dstate->caps_reg_state32; \ + int which = R_CXL_DEV_##reg##_CAP_HDR0; \ + cap_hdrs[which] = \ + FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, \ + CAP_ID, cap_id); \ + cap_hdrs[which] = FIELD_DP32( \ + cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1); \ + cap_hdrs[which + 1] = \
include/hw/cxl/cxl_device.h CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET) · line 159
Quoted 10 lines
include/hw/cxl/cxl_device.h +7
-0+ FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1, \ + CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET); \ + cap_hdrs[which + 2] = \ + FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2, \ + CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH); \ + } while (0) + REG32(CXL_DEV_MAILBOX_CAP, 0) FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)FIELD (CXL_DEV_MAILBOX_CAP ,INT_CAP ,5 ,1 )
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v4 06/42] hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Jonathan CameronQuoted 54 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.No commands are implemented yet.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 129 ++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 201 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 29 +++++- 5 files changed, 358 insertions(+), 5 deletions(-)hw/cxl/cxl-device-utils.c +22
-0@@ -40,6 +40,115 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return 0 ; } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +switch (size) { +case 1 : +return cxl_dstate->mbox_reg_state[offset]; +case 2 : +return cxl_dstate->mbox_reg_state16[offset /2 ]; +case 4 : +return cxl_dstate->mbox_reg_state32[offset /4 ]; +case 8 : +return cxl_dstate->mbox_reg_state64[offset /8 ]; +default : +g_assert_not_reached (); + } +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) {
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 74
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ;
The log message implies WI but it isn't. Did you want an early return?
Quoted 12 lines
hw/cxl/cxl-device-utils.c +12
-0+ } + + reg_state[offset /4 ] = value; +} + +static void mailbox_mem_writeq (uint64_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS :
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 96
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ;
which is the behaviour here.
Quoted 13 lines
hw/cxl/cxl-device-utils.c +13
-0+ } + + + reg_state[offset /8 ] = value; +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) { +memcpy (cxl_dstate->mbox_reg_state + offset, &value, size);
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 119
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+return ; + } + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD ();
RCU_READ_LOCK doesn't prevent concurrent writes, although the BQL should.
Quoted 20 lines
hw/cxl/cxl-device-utils.c +20
-0+ +switch (size) { +case 4 : +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +mailbox_mem_writeq (cxl_dstate->mbox_reg_state64, offset, value); +break ; +default : +g_assert_not_reached (); + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write,
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 149
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+ .endianness = DEVICE_LITTLE_ENDIAN, + .valid = { + .min_access_size = 1, + .max_access_size = 8, + .unaligned = false, + }, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + },
with_attrs?
Quoted 161 lines
hw/cxl/cxl-device-utils.c +21
-1+}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,/* status register is read only */ @@ -80,20 +189,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_CAPS_SIZE);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -102,4 +224,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +105
-0@@ -0,0 +1,201 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define define_mailbox_handler_zeroed (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_const (name, data) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = sizeof(data); \ + memcpy(cmd->payload, data, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_nop (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \
hw/cxl/cxl-mailbox-utils.c line 115
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c new file +10
-0+ } + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid;
static?
but really this seems more like something that should be in a property of the device. Will it always be fixed or is it something the user might want to tweak?
Quoted 21 lines
hw/cxl/cxl-mailbox-utils.c new file +21
-0+ +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; + +/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { + qemu_log_mask(LOG_UNIMP, "Corrupt internal state for
Quoted 1 line
firmware\n");
is this actually UNIMP or has something gone wrong internally or is the guest doing something wrong?
Quoted 38 lines
hw/cxl/cxl-mailbox-utils.c new file +38
-0+return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { +qemu_log_mask (LOG_UNIMP ,"Command %04xh not implemented\n" , + set <<8 | cmd); + goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + + cxl_dstate->mbox_reg_state64[A_CXL_DEV_MAILBOX_CMD /8 ] = command_reg;
hw/cxl/cxl-mailbox-utils.c line 184
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c new file +10
-0+ cxl_dstate->mbox_reg_state64[A_CXL_DEV_MAILBOX_STS /8 ] = status_reg; + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ;
I'm curious as to where this magic number came from.
Quoted 46 lines
hw/cxl/cxl-mailbox-utils.c new file +17
-0+ +for (int set =0 ; set <256 ; set++) { +for (int cmd =0 ; cmd <256 ; cmd++) { +if (cxl_cmd_set[set][cmd].handler) { +struct cxl_cmd *c = &cxl_cmd_set[set][cmd]; +struct cel_log *log = + &cxl_dstate->cel_log[cxl_dstate->cel_size]; + + log->opcode = (set <<8 ) | cmd; + log->effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,4 +14,7 @@#include "cxl_component.h" #include "cxl_device.h" +#define CXL_COMPONENT_REG_BAR_IDX 0 +#define CXL_DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +0
-0@@ -85,16 +85,28 @@ typedef struct cxl_device_state {MemoryRegion device_registers;
include/hw/cxl/cxl_device.h typedef struct cxl_device_state { · line 95
Quoted 10 lines
include/hw/cxl/cxl_device.h +1
-3/* mmio for device capabilities array - 8.2.8.2 */ +MemoryRegion device;struct {MemoryRegion caps;uint32_t caps_reg_state32[CXL_CAPS_SIZE /4 ]; }; -/* mmio for the device status registers 8.2.8.3 */ -MemoryRegion device; -
The above code motion should probably be squashed into the original patch
Quoted 43 lines
include/hw/cxl/cxl_device.h +24
-1/* mmio for the mailbox registers 8.2.8.4 */ -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH /2 ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH /4 ]; +uint64_t mbox_reg_state64[CXL_MAILBOX_REGISTERS_LENGTH /8 ]; + }; +struct cel_log { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };/* memory region for persistent memory, HDM */ uint64_t pmem_size;@@ -137,6 +149,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \@@ -165,6 +180,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )
Signature · 1 line
Alex Bennée
Re: [PATCH v4 07/42] hw/cxl/device: Add memory device utilities
In reply to Jonathan CameronQuoted 12 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Memory devices implement extra capabilities on top of CXL devices. This adds support for that.
A large part of memory devices is the mailbox/command interface. All of the mailbox handling is done in the mailbox-utils library. Longer term, new CXL devices that are being emulated may want to handle commands differently, and therefore would need a mechanism to opt in/out of the specific generic handlers. As such, this is considered sufficient for now, but may need more depth in the future.
Quoted 2 lines
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v4 06/42] hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Jonathan CameronQuoted 12 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.
<snip>
Quoted 19 lines
hw/cxl/cxl-mailbox-utils.c new file +19
-0+ +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define define_mailbox_handler_zeroed (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_const (name, data) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \
hw/cxl/cxl-mailbox-utils.c line 106
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c new file +10
-0+ *len =sizeof (data); \ +memcpy (cmd->payload, data, *len); \ +return CXL_MBOX_SUCCESS ; \ + } +#define define_mailbox_handler_nop (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \ + }
I missed these #define's - can you upcase them to make it clear they are macros when used.
Signature · 1 line
Alex Bennée
Re: [PATCH v4 08/42] hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
In reply to Jonathan CameronQuoted 7 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Using the previously implemented stubbed helpers, it is now possible to easily add the missing, required commands to the implementation.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v4 09/42] hw/cxl/device: Timestamp implementation (8.2.9.3)
In reply to Jonathan CameronQuoted 56 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 53 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 59 insertions(+)hw/cxl/cxl-mailbox-utils.c +13
-0@@ -43,6 +43,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (TIMESTAMP_GET ); +declare_mailbox_handler (TIMESTAMP_SET );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)#define CXL_CMD (s, c, in, cel_effect) \@@ -129,10 +135,57 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (TIMESTAMP ,GET ,0 ,0 ), + CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE), };#undef CXL_CMD +/* +* 8.2.9.3.1 +*/ +define_mailbox_handler (TIMESTAMP_GET ) +{
hw/cxl/cxl-mailbox-utils.c static struct cxl_cmd cxl_cmd_set[256][256] = { · line 158
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+struct timespec ts; +uint64_t delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done; + } + +/* First find the delta from the last time the host set the time. */ +clock_gettime (CLOCK_REALTIME , &ts);
Could you consider using qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)? Otherwise by introducing a dependency on real time you'll loose the ability to get deterministic execution via icount.
Quoted 50 lines
hw/cxl/cxl-mailbox-utils.c +30
-0+ delta = (ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec) - + cxl_dstate->timestamp.last_set; + +/* Then adjust the actual time */ +stq_le_p (cmd->payload, cxl_dstate->timestamp.host_set + delta); + +done: + *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* +* 8.2.9.3.2 +*/ +define_mailbox_handler (TIMESTAMP_SET ) +{ +struct timespec ts; + +clock_gettime (CLOCK_REALTIME , &ts); + + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set = + ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec; + + cxl_dstate->timestamp.host_set =le64_to_cpu (*(uint64_t *)cmd->payload); + + *len =0 ; +return CXL_MBOX_SUCCESS ; +} +QemuUUID cel_uuid;void cxl_process_mailbox (CXLDeviceState *cxl_dstate)include/hw/cxl/cxl_device.h +6
-0@@ -114,6 +114,12 @@ typedef struct cxl_device_state {size_t cel_size; }; +struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +/* memory region for persistent memory, HDM */ uint64_t pmem_size; } CXLDeviceState;
Signature · 1 line
Alex Bennée
Re: [PATCH v4 10/42] hw/cxl/device: Add log commands (8.2.9.4) + CEL
In reply to Jonathan CameronQuoted 52 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 67 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)hw/cxl/cxl-mailbox-utils.c +7
-0@@ -46,6 +46,9 @@ enum { TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1 + LOGS = 0x04, +#define GET_SUPPORTED 0x0 +#define GET_LOG 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -122,6 +125,8 @@ define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY );declare_mailbox_handler (TIMESTAMP_GET );declare_mailbox_handler (TIMESTAMP_SET ); +declare_mailbox_handler (LOGS_GET_SUPPORTED ); +declare_mailbox_handler (LOGS_GET_LOG );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -137,6 +142,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ),CXL_CMD (TIMESTAMP ,GET ,0 ,0 ),CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ), +CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ), + CXL_CMD(LOGS, GET_LOG, 0x18, 0), };#undef CXL_CMD
hw/cxl/cxl-mailbox-utils.c define_mailbox_handler(TIMESTAMP_SET) · line 200
Quoted 6 lines
hw/cxl/cxl-mailbox-utils.c +3
-0@@ -188,6 +195,66 @@ define_mailbox_handler(TIMESTAMP_SET)QemuUUID cel_uuid; +/* 8.2.9.4.1 */ +define_mailbox_handler (LOGS_GET_SUPPORTED ) +{
Here is where I get a bit wary of the define_mailbox_handler define which from what I can tell just hides the declarations. This makes the handling of things like *cmd rather opaque. There is an argument for the boilerplate definitions (_nop and _zeroed) but perhaps not these.
Quoted 60 lines
hw/cxl/cxl-mailbox-utils.c +57
-0+struct { +uint16_t entries; +uint8_t rsvd[6 ]; +struct { +QemuUUID uuid; +uint32_t size; + } log_entries[1 ]; + } __attribute__((packed)) *supported_logs = (void *)cmd->payload; +_Static_assert (sizeof (*supported_logs) ==0x1c ,"Bad supported log size" ); + + supported_logs->entries =1 ; + supported_logs->log_entries[0 ].uuid = cel_uuid; + supported_logs->log_entries[0 ].size =4 * cxl_dstate->cel_size; + + *len =sizeof (*supported_logs); +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.4.2 */ +define_mailbox_handler (LOGS_GET_LOG ) +{ +struct { +QemuUUID uuid; +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (16 ))) *get_log = (void *)cmd->payload; + +/* +* 8.2.9.4.2 +* The device shall return Invalid Parameter if the Offset or Length +* fields attempt to access beyond the size of the log as reported by Get +* Supported Logs. +* +* XXX: Spec is wrong, "Invalid Parameter" isn't a thing. +* XXX: Spec doesn't address incorrect UUID incorrectness. +* +* The CEL buffer is large enough to fit all commands in the emulation, so +* the only possible failure would be if the mailbox itself isn't big +* enough. +*/ +if (get_log->offset + get_log->length > cxl_dstate->payload_size) { +return CXL_MBOX_INVALID_INPUT ; + } + +if (!qemu_uuid_is_equal (&get_log->uuid, &cel_uuid)) { +return CXL_MBOX_UNSUPPORTED ; + } + +/* Store off everything to local variables so we can wipe out the payload */ + *len = get_log->length; + +memmove (cmd->payload, cxl_dstate->cel_log + get_log->offset, + get_log->length); + +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
Alex Bennée
Re: [PATCH v4 11/42] hw/pxb: Use a type for realizing expanders
In reply to Jonathan CameronQuoted 7 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This opens up the possibility for more types of expanders (other than PCI and PCIe). We'll need this to create a CXL expander.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v4 12/42] hw/pci/cxl: Create a CXL bus type
In reply to Jonathan CameronQuoted 7 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
The easiest way to differentiate a CXL bus, and a PCIE bus is using a flag. A CXL bus, in hardware, is backward compatible with PCIE, and therefore the code tries pretty hard to keep them in sync as much as possible.
Quoted 6 lines
The other way to implement this would be to try to cast the bus to the correct type. This is less code and useful for debugging via simply looking at the flags.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v4 13/42] hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Jonathan CameronQuoted 122 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in
_SBnamed ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan.Cameron <[email protected]>--- hw/pci-bridge/pci_expander_bridge.c | 98 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 ++ 3 files changed, 109 insertions(+), 2 deletions(-)hw/pci-bridge/pci_expander_bridge.c +40
-2@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h" @@ -56,6 +57,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +struct PXBDev {/*< private >*/ PCIDevice parent_obj;@@ -66,8 +71,19 @@ struct PXBDev {bool bypass_iommu; }; +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }@@ -76,6 +92,9 @@ static GList *pxb_dev_list;#define TYPE_PXB_HOST "pxb-host" +#define TYPE_PXB_CXL_HOST "pxb-cxl-host" +#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);@@ -112,11 +131,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;@@ -218,6 +246,16 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); } +static void pxb_dev_reset (DeviceState *dev) +{ +CXLHost *cxl =PXB_CXL_HOST (dev); +CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT ,8 ); +} +static gint pxb_compare (gconstpointer a,gconstpointer b) {const PXBDev *pxb_a = a, *pxb_b = b;
hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, · line 335
Quoted 8 lines
hw/pci-bridge/pci_expander_bridge.c +5
-0@@ -290,6 +328,11 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,pci_config_set_class (dev->config,PCI_CLASS_BRIDGE_HOST ); pxb_dev_list =g_list_insert_sorted (pxb_dev_list, pxb, pxb_compare); + +if (type ==CXL ) { +pxb_dev_reset (ds); + } +
Couldn't this just be done in the cxl realize function after it calls the common code?
hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_class_init(ObjectClass *klass, void *data) · line 389
Quoted 13 lines
hw/pci-bridge/pci_expander_bridge.c +6
-0return ; err_register_bus:@@ -338,6 +381,12 @@ static void pxb_dev_class_init(ObjectClass *klass, void *data)device_class_set_props (dc, pxb_dev_properties); dc->hotpluggable = false;set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* +* Reset doesn't seem to actually be called, but maybe it will in the +* future? +*/ + dc->reset = pxb_dev_reset;
Surely because this should be in pxb_cxl_dev_class_init?
Quoted 111 lines
hw/pci-bridge/pci_expander_bridge.c +45
-0}static const TypeInfo pxb_dev_info = {@@ -389,13 +438,58 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)hw/pci/pci.c +7
-0@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);@@ -2892,6 +2898,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);include/hw/pci/pci.h +6
-0@@ -409,6 +409,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"typedef void (*pci_bus_dev_fn )(PCIBus *b,PCIDevice *d,void *opaque);typedef void (*pci_bus_fn )(PCIBus *b,void *opaque);@@ -768,6 +769,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
Signature · 1 line
Alex Bennée
Re: [PATCH v4 14/42] tests/acpi: allow DSDT.viot table changes.
In reply to Jonathan CameronQuoted 4 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The next patch unifies some of the PCI host bridge DSDT generation code and results in some minor changes to this file.
I'd just squash this in with the patch that makes the change... otherwise you risk breaking bisectablility.
Quoted 13 lines
Signed-off-by Jonathan Cameron <[email protected]>--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)tests/qtest/bios-tables-test-allowed-diff.h +1
-0@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.viot",
Signature · 1 line
Alex Bennée
Re: [PATCH v4 15/42] acpi/pci: Consolidate host bridge setup
In reply to Jonathan CameronQuoted 15 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This cleanup will make it easier to add support for CXL to the mix.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/i386/acpi-build.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)hw/i386/acpi-build.c +0
-0
hw/i386/acpi-build.c static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func) · line 1401
Quoted 4 lines
hw/i386/acpi-build.c +1
-0@@ -1398,6 +1398,23 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } +enum {PCI ,PCIE };
Make this a typed enum and update the signatures to use it instead of int to prevent confusion.
Quoted 53 lines
hw/i386/acpi-build.c +18
-12+static void init_pci_acpi (Aml *dev,int uid,int type,bool native_pcie_hp) +{ +if (type ==PCI ) { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +/* Expander bridges do not have ACPI PCI Hot-plug enabled */ +aml_append (dev,build_q35_osc_method (native_pcie_hp)); + } +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,@@ -1429,9 +1446,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,if (misc->is_piix4) { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); -aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (pcmc->pci_root_uid))); +init_pci_acpi (dev, pcmc->pci_root_uid,PCI , false);aml_append (sb_scope, dev);aml_append (dsdt, sb_scope);@@ -1447,11 +1462,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, }else { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); -aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (pcmc->pci_root_uid))); -aml_append (dev,build_q35_osc_method (!pm->pcihp_bridge_en)); +init_pci_acpi (dev, pcmc->pci_root_uid,PCIE , !pm->pcihp_bridge_en);aml_append (sb_scope, dev);if (mcfg_valid) {aml_append (sb_scope,build_q35_dram_controller (&mcfg));@@ -1562,17 +1573,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" ); dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -if (pci_bus_is_express (bus)) { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));
hw/i386/acpi-build.c build_dsdt(GArray *table_data, BIOSLinker *linker, · line 1582
Quoted 10 lines
hw/i386/acpi-build.c +2
-5-/* Expander bridges do not have ACPI PCI Hot-plug enabled */ -aml_append (dev,build_q35_osc_method (true)); - }else { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); - } +init_pci_acpi (dev, bus_num, +pci_bus_is_express (bus) ?PCIE :PCI , true);if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH v4 16/42] tests/acpi: Add update DSDT.viot
In reply to Jonathan CameronQuoted 44 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The consolidation of DSDT AML generation for PCI host bridges lead to some minor ordering changes and the addition of
_ADRwith a default of 0 for those case that didn't already have it. Only DSDT.viot test is affected.Changes all similar to:
Scope (\_SB)
{ Device (PC30) {(quoted diff) +2
-1- Name (_UID, 0x30) // _UID: Unique ID Name (_BBN, 0x30) // _BBN: BIOS Bus Number Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID + Name (_ADR, Zero) // _ADR: Address + Name (_UID, 0x30) // _UID: Unique ID Method (_OSC, 4, NotSerialized) // _OSC: Operating System CapabilitiesSigned-off-by Jonathan Cameron <[email protected]>--- tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes tests/qtest/bios-tables-test-allowed-diff.h | 1 - 2 files changed, 1 deletion(-)tests/data/acpi/q35/DSDT.viot binary +0
-0tests/qtest/bios-tables-test-allowed-diff.h +0
-0
tests/qtest/bios-tables-test-allowed-diff.h line 2
Quoted 2 lines
tests/qtest/bios-tables-test-allowed-diff.h +0
-1@@ -1,2 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/q35/DSDT.viot",
squash with the change please.
Signature · 1 line
Alex Bennée
Re: [PATCH v4 17/42] cxl: Machine level control on whether CXL support is enabled
In reply to Jonathan CameronQuoted 22 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
There are going to be some potential overheads to CXL enablement, for example the host bridge region reserved in memory maps. Add a machine level control so that CXL is disabled by default.
Signed-off-by Jonathan Cameron <[email protected]>--- hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++++++++++++++++++++++++++ hw/i386/microvm.c | 1 + hw/i386/pc.c | 1 + hw/ppc/spapr.c | 1 + include/hw/boards.h | 2 ++ include/hw/cxl/cxl.h | 4 ++++ 7 files changed, 36 insertions(+)hw/arm/virt.c +0
-0
hw/arm/virt.c static void virt_machine_class_init(ObjectClass *oc, void *data) · line 2859
Quoted 4 lines
hw/arm/virt.c +1
-0@@ -2856,6 +2856,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->unplug = virt_machine_device_unplug_cb; mc->nvdimm_supported = true; mc->smp_props.clusters_supported = true; + mc->cxl_supported = false;
You should be able to do this in machine_class_init and then...
<snip>
just turn it on for pc
hw/i386/pc.c static void pc_machine_class_init(ObjectClass *oc, void *data) · line 1744
Quoted 11 lines
hw/i386/pc.c +1
-0@@ -1739,6 +1739,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = true; mc->smp_props.dies_supported = true; + mc->cxl_supported = true; mc->default_ram_id ="pc.ram" ;
<snip>
Otherwise:
Signature · 1 line
Alex Bennée
Quoted 4 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.
I've been through and added comments through the first half of the patches. I'll see if I can get to the second half next week however if you beat me to it with a re-rev I expect some ripples from the requested changes.
Aside from ensuring the rest of the builds work:
https://gitlab.com/stsquad/qemu/-/pipelines/456700583/failures
it looks pretty good to me. I await the next version ;-)
Signature · 1 line
Alex Bennée
Quoted 12 lines · On Thu, 27 Jan 2022 14:22:52 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
Previous version was RFC v3: CXL 2.0 Support.
No longer an RFC as I would consider the vast majority of this to be ready for detailed review. There are still questions called out in some patches however.I've been through and added comments through the first half of the patches. I'll see if I can get to the second half next week however if you beat me to it with a re-rev I expect some ripples from the requested changes.
Quoted 6 lines
Aside from ensuring the rest of the builds work:
https://gitlab.com/stsquad/qemu/-/pipelines/456700583/failures
it looks pretty good to me. I await the next version ;-)
Thanks for ploughing through them - it's a great help. Hopefully I'll get a new version out before you get back to them.
The CI certainly threw up some unexpected issues alongside the bugs and wrong assumptions and build issues you pointed out.
* can't have a field called ERROR in a register on some archs
* doesn't work if you don't push the tags on the tree... (win builds)
but should be clean in next version.
Thanks,
Jonathan
Re: [PATCH v4 02/42] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
In reply to Alex BennéeQuoted 114 lines · On Wed, 26 Jan 2022 12:32:01 +0000 Alex Bennée <[email protected]> wrote:
Quoted 75 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The
cxl_pciheader manages bits associated with PCI, specifically the DVSEC and related fields. Thecxl_component.hvariant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] registers. Per spec, the registers are little endian.None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 212 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 3 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 16 +++ include/hw/cxl/cxl_component.h | 196 ++++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++ 8 files changed, 570 insertions(+)hw/Kconfig +1
-0@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfighw/cxl/Kconfig new file +3
-0@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESShw/cxl/cxl-component-utils.c new file +11
-0@@ -0,0 +1,212 @@ +/* +* CXL Utility library for components +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" hw/cxl/cxl-component-utils.c line 21
Quoted 10 lines
hw/cxl/cxl-component-utils.c new file +10
-0+#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" + +static uint64_t cxl_cache_mem_read_reg (void *opaque,hwaddr offset, +unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; + +assert (size ==4 );You assert here but bellow:
Quoted 24 lines
hw/cxl/cxl-component-utils.c new file +24
-0+ +/* +* 8.2.3 +* The access restrictions specified in Section 8.2.2 also apply to CXL 2.0 +* Component Registers. +* +* 8.2.2 +* • A 32 bit register shall be accessed as a 4 Bytes quantity. Partial +* reads are not permitted. +* • A 64 bit register shall be accessed as a 8 Bytes quantity. Partial +* reads are not permitted. +* +* As of the spec defined today, only 4 byte registers exist. +*/ +static const MemoryRegionOps cache_mem_ops = { + .read = cxl_cache_mem_read_reg, + .write = cxl_cache_mem_write_reg, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = {
Quoted 9 lines
hw/cxl/cxl-component-utils.c line 71
Quoted 4 lines
hw/cxl/cxl-component-utils.c new file +4
-0+ .min_access_size = 4, + .max_access_size = 4, + }, +};You have constrained the access to 4 so you will only see 4 bytes accesses. If it is valid for the guest to access 64bit words then it would be better to no-op that case and maybe
LOG_UNIMPthe fact.
Ugh. This looks suspiciously like a work around for a kernel bug (possibly an accidental one).
If the comment above is correct (I've checked the spec and agree with it)..
https://elixir.bootlin.com/linux/v5.17-rc1/source/drivers/cxl/core/regs.c#L48
cap_array = readq(base + CXL_CM_CAP_HDR_OFFSET);
is undefined behavior as CAP_HDR is a 32bit register.
I guess the undefined choice on all the hardware people actually have is to service the 8 byte read, but there is no reason to believe future hardware will.
... and this is why we like having emulation in Qemu :)
+ eagle eyed reviewers!
Guess I'd better send a kernel fix sometime soon.
Jonathan
Quoted 4 lines
Otherwise the rest looks ok to me:
Reviewed-by Alex Bennée <[email protected]>
Re: [PATCH v4 02/42] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
Quoted 138 lines · On Fri, 28 Jan 2022 14:22:51 +0000 Jonathan Cameron…
Quoted 114 lines · On Wed, 26 Jan 2022 12:32:01 +0000 Alex Bennée <[email protected]> wrote:
Quoted 75 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The
cxl_pciheader manages bits associated with PCI, specifically the DVSEC and related fields. Thecxl_component.hvariant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] registers. Per spec, the registers are little endian.None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 212 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 3 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 16 +++ include/hw/cxl/cxl_component.h | 196 ++++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++ 8 files changed, 570 insertions(+)hw/Kconfig +1
-0@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfighw/cxl/Kconfig new file +3
-0@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESShw/cxl/cxl-component-utils.c new file +11
-0@@ -0,0 +1,212 @@ +/* +* CXL Utility library for components +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" hw/cxl/cxl-component-utils.c line 21
Quoted 10 lines
hw/cxl/cxl-component-utils.c new file +10
-0+#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" + +static uint64_t cxl_cache_mem_read_reg (void *opaque,hwaddr offset, +unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; + +assert (size ==4 );You assert here but bellow:
Quoted 24 lines
hw/cxl/cxl-component-utils.c new file +24
-0+ +/* +* 8.2.3 +* The access restrictions specified in Section 8.2.2 also apply to CXL 2.0 +* Component Registers. +* +* 8.2.2 +* • A 32 bit register shall be accessed as a 4 Bytes quantity. Partial +* reads are not permitted. +* • A 64 bit register shall be accessed as a 8 Bytes quantity. Partial +* reads are not permitted. +* +* As of the spec defined today, only 4 byte registers exist. +*/ +static const MemoryRegionOps cache_mem_ops = { + .read = cxl_cache_mem_read_reg, + .write = cxl_cache_mem_write_reg, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + .unaligned = false, + }, + .impl = {Quoted 9 lines
hw/cxl/cxl-component-utils.c line 71
Quoted 4 lines
hw/cxl/cxl-component-utils.c new file +4
-0+ .min_access_size = 4, + .max_access_size = 4, + }, +};You have constrained the access to 4 so you will only see 4 bytes accesses. If it is valid for the guest to access 64bit words then it would be better to no-op that case and maybe
LOG_UNIMPthe fact.Ugh. This looks suspiciously like a work around for a kernel bug (possibly an accidental one).
If the comment above is correct (I've checked the spec and agree with it)..
https://elixir.bootlin.com/linux/v5.17-rc1/source/drivers/cxl/core/regs.c#L48
cap_array=readq(base + CXL_CM_CAP_HDR_OFFSET);is undefined behavior as
CAP_HDRis a 32bit register.
Quoted 8 lines
I guess the undefined choice on all the hardware people actually have is to service the 8 byte read, but there is no reason to believe future hardware will.
... and this is why we like having emulation in Qemu :)
+ eagle eyed reviewers!Guess I'd better send a kernel fix sometime soon.
However, after another read of the spec there are link related registers which we currently don't implement emulation for but do advertise as being present (as they are required) are 8 bytes long.
Linux doesn't use them yet, but it may come so I'll need to leave max_access_size = 8 enabled and as you suggested LOG_UNIMP.
Probably makes sense to implement the link registers in a follow up patch set along with (possibly) some kernel support to expose the information available in those registers.
Jonathan
Quoted 8 lines
Jonathan
Quoted 4 lines
Otherwise the rest looks ok to me:
Reviewed-by Alex Bennée <[email protected]>
Re: [PATCH v4 04/42] hw/cxl/device: Introduce a CXL device (8.2.8)
In reply to Alex BennéeQuoted 136 lines · On Wed, 26 Jan 2022 18:07:40 +0000 Alex Bennée <[email protected]> wrote:
Quoted 86 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
A CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.
This new device type is specifically catered towards the eventual implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_device.h | 157 ++++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+)include/hw/cxl/cxl.h +1
-0@@ -12,5 +12,6 @@#include "cxl_pci.h" #include "cxl_component.h" +#include "cxl_device.h" #endif include/hw/cxl/cxl_device.h new file +51
-0@@ -0,0 +1,157 @@ +/* +* QEMU CXL Devices +* +* Copyright (c) 2020 Intel +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#ifndef CXL_DEVICE_H +#define CXL_DEVICE_H + +#include "hw/register.h" + +/* +* The following is how a CXL device's MMIO space is laid out. The only +* requirement from the spec is that the capabilities array and the capability +* headers start at offset 0 and are contiguously packed. The headers themselves +* provide offsets to the register fields. For this emulation, registers will +* start at offset 0x80 (m == 0x80). No secondary mailbox is implemented which +* means that n = m + sizeof(mailbox registers) + sizeof(device registers). +* +* This is roughly described in 8.2.8 Figure 138 of the CXL 2.0 spec. +* +* +---------------------------------+ +* | | +* | Memory Device Registers | +* | | +* n + PAYLOAD_SIZE_MAX ----------------------------------- +* ^ | | +* | | | +* | | | +* | | | +* | | | +* | | Mailbox Payload | +* | | | +* | | | +* | | | +* | ----------------------------------- +* | | Mailbox Registers | +* | | | +* n ----------------------------------- +* ^ | | +* | | Device Registers | +* | | | +* m ----------------------------------> +* ^ | Memory Device Capability Header| +* | ----------------------------------- +* | | Mailbox Capability Header | +* | -------------- -------------------- +* | | Device Capability Header | include/hw/cxl/cxl_device.h line 61
Quoted 10 lines
include/hw/cxl/cxl_device.h new file +10
-0+* | ----------------------------------- +* | | | +* | | | +* | | Device Cap Array[0..n] | +* | | | +* | | | +* | | +* 0 +---------------------------------+ +* +*/ Excellent diagram ;-)
Quoted 35 lines
include/hw/cxl/cxl_device.h new file +35
-0+ +#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10/* Figure 138 */ +#define CXL_DEVICE_CAP_REG_SIZE 0x10/* 8.2.8.2 */ +#define CXL_DEVICE_CAPS_MAX 4/* 8.2.8.2.1 + 8.2.8.5 */ + +#define CXL_DEVICE_REGISTERS_OFFSET 0x80/* Read comment above */ +#define CXL_DEVICE_REGISTERS_LENGTH 0x8/* 8.2.8.3.1 */ + +#define CXL_MAILBOX_REGISTERS_OFFSET \ + (CXL_DEVICE_REGISTERS_OFFSET + CXL_DEVICE_REGISTERS_LENGTH) +#define CXL_MAILBOX_REGISTERS_SIZE 0x20/* 8.2.8.4, Figure 139 */ +#define CXL_MAILBOX_PAYLOAD_SHIFT 11 +#define CXL_MAILBOX_MAX_PAYLOAD_SIZE (1 << CXL_MAILBOX_PAYLOAD_SHIFT) +#define CXL_MAILBOX_REGISTERS_LENGTH \ + (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) + +typedef struct cxl_device_state { +MemoryRegion device_registers; + +/* mmio for device capabilities array - 8.2.8.2 */ +MemoryRegion caps; + +/* mmio for the device status registers 8.2.8.3 */ +MemoryRegion device; + +/* mmio for the mailbox registers 8.2.8.4 */ +MemoryRegion mailbox; + +/* memory region for persistent memory, HDM */ +uint64_t pmem_size; +}CXLDeviceState ; + +/* Initialize the register block for a device */ +void cxl_device_register_block_init (Object *obj,CXLDeviceState *dev); +
Quoted 8 lines
include/hw/cxl/cxl_device.h line 101
Quoted 5 lines
include/hw/cxl/cxl_device.h new file +5
-0+/* Set up default values for the register block */ +void cxl_device_register_init_common (CXLDeviceState *dev); + +/* CXL 2.0 - 8.2.8.1 */ +REG32 (CXL_DEV_CAP_ARRAY ,0 )/* 48b!?!?! */ What does this comment mean? A 48 bit register stuffed in a 32 bit one? Doesn't seem right.
ah. I'd forgotten about this.. !?!?! is code word for spec bug without actually saying it in a public email...
It's fixed now in the published errata which makes this a 128bit register but then says you can access it as a 64 bit register and reserves everything above bit 48. I'll update the comment and make it a REG64.
Quoted 8 lines
include/hw/cxl/cxl_device.h line 105
Quoted 5 lines
include/hw/cxl/cxl_device.h new file +4
-0+ FIELD(CXL_DEV_CAP_ARRAY, CAP_ID, 0, 16) + FIELD(CXL_DEV_CAP_ARRAY, CAP_VERSION, 16, 8) +REG32(CXL_DEV_CAP_ARRAY2, 4) /* We're going to pretend it's 64b */ +FIELD (CXL_DEV_CAP_ARRAY2 ,CAP_COUNT ,0 ,16 )I'm confused why you don't treat it as a single register it is in the spec.
Code predates introduction of REG64() and I'd missed that turning up.
Quoted 12 lines
include/hw/cxl/cxl_device.h line 116
Quoted 12 lines
include/hw/cxl/cxl_device.h new file +11
-0+ +/* +* Helper macro to initialize capability headers for CXL devices. +* +* In the 8.2.8.2, this is listed as a 128b register, but in 8.2.8, it says: +* > No registers defined in Section 8.2.8 are larger than 64-bits wide so that +* > is the maximum access size allowed for these registers. If this rule is not +* > followed, the behavior is undefined +* +* Here we've chosen to make it 4 dwords. The spec allows any pow2 multiple +* access to be used for a register (2 qwords, 8 words, 128 bytes).
The F4 errata covers this as well, making it clear that 128bit registers can be accessed as 1,2,4,8 byte aligned reads. No read advantage of REG64 over REG32 as neither matches the spec, so I'll leave this one alone but update the comment..
Quoted 32 lines
Quoted 32 lines
include/hw/cxl/cxl_device.h new file +32
-0+*/ +#define CXL_DEVICE_CAPABILITY_HEADER_REGISTER (n, offset) \ + REG32(CXL_DEV_##n##_CAP_HDR0, offset) \ + FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_ID, 0, 16) \ + FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_VERSION, 16, 8) \ + REG32(CXL_DEV_##n##_CAP_HDR1, offset + 4) \ + FIELD(CXL_DEV_##n##_CAP_HDR1, CAP_OFFSET, 0, 32) \ + REG32(CXL_DEV_##n##_CAP_HDR2, offset + 8) \ + FIELD(CXL_DEV_##n##_CAP_HDR2, CAP_LENGTH, 0, 32) + +CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET) +CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ + CXL_DEVICE_CAP_REG_SIZE) + +REG32(CXL_DEV_MAILBOX_CAP, 0) + FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5) + FIELD(CXL_DEV_MAILBOX_CAP, INT_CAP, 5, 1) + FIELD(CXL_DEV_MAILBOX_CAP, BG_INT_CAP, 6, 1) + FIELD(CXL_DEV_MAILBOX_CAP, MSI_N, 7, 4) + +REG32(CXL_DEV_MAILBOX_CTRL, 4) + FIELD(CXL_DEV_MAILBOX_CTRL, DOORBELL, 0, 1) + FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1) +FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) + +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_STS, 0x10) + FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1) + FIELD(CXL_DEV_MAILBOX_STS, ERRNO, 32, 16) +FIELD (CXL_DEV_MAILBOX_STS ,VENDOR_ERRNO ,48 ,16 ) + +/* XXX: actually a 64b register */
Quoted 8 lines
include/hw/cxl/cxl_device.h line 153
Quoted 5 lines
include/hw/cxl/cxl_device.h new file +5
-0+REG32(CXL_DEV_BG_CMD_STS, 0x18) + FIELD(CXL_DEV_BG_CMD_STS, BG, 0, 16) + FIELD(CXL_DEV_BG_CMD_STS, DONE, 16, 7) + FIELD(CXL_DEV_BG_CMD_STS, ERRNO, 32, 16) +FIELD (CXL_DEV_BG_CMD_STS ,VENDOR_ERRNO ,48 ,16 )Again is there a reason not to use REG64? I can see the need to split 128 bit registers but not 64 bit ones.
Will do.
Quoted 10 lines
include/hw/cxl/cxl_device.h line 157
Quoted 5 lines
include/hw/cxl/cxl_device.h new file +4
-0+ +REG32(CXL_DEV_CMD_PAYLOAD, 0x20) + +#endif Otherwise:
Reviewed-by Alex Bennée <[email protected]>
Re: [PATCH v4 05/42] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Alex BennéeQuoted 75 lines · On Wed, 26 Jan 2022 18:17:12 +0000 Alex Bennée <[email protected]> wrote:
Quoted 73 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This implements all device MMIO up to the first capability. That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 105 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_device.h | 28 +++++++++- 3 files changed, 133 insertions(+), 1 deletion(-)hw/cxl/cxl-device-utils.c new file +50
-0@@ -0,0 +1,105 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +/* +* Device registers have no restrictions per the spec, and so fall back to the +* default memory mapped register rules in 8.2: +* Software shall use CXL.io Memory Read and Write to access memory mapped +* register defined in this section. Unless otherwise specified, software +* shall restrict the accesses width based on the following: +* • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes +* quantity. +* • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 +* Bytes +* • The address shall be a multiple of the access width, e.g. when +* accessing a register as a 4 Byte quantity, the address shall be +* multiple of 4. +* • The accesses shall map to contiguous bytes.If these rules are not +* followed, the behavior is undefined +*/ + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +return cxl_dstate->caps_reg_state32[offset /4 ]; +} + +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL ,/* status register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false,
Quoted 8 lines
hw/cxl/cxl-device-utils.c line 56
Quoted 6 lines
hw/cxl/cxl-device-utils.c new file +6
-0+ }, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + }, +};I think for >64 bit registers you need to use the
read_with_attrs
I don't follow this comment. Max access to registers is 64 bits. A few are documented as 128 bit or indeed larger in the spec, but the access is as if they were multiple 64 bit registers accesses. It's not permissible to do a single 128bit access for example.
The F4 errata clarified that - previously it was rather unclear what the restrictions on access to the larger registers were.
I've updated a few comments on this to reflect the errata.
Thanks,
Jonathan
Re: [PATCH v4 06/42] hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Alex BennéeQuoted 19 lines · On Wed, 26 Jan 2022 18:22:18 +0000 Alex Bennée <[email protected]> wrote:
Quoted 17 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This is the beginning of implementing mailbox support for CXL 2.0 devices. The implementation recognizes when the doorbell is rung, handles the command/payload, clears the doorbell while returning error codes and data.
Generally the mailbox mechanism is designed to permit communication between the host OS and the firmware running on the device. For our purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.No commands are implemented yet.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>---
...
Quoted 11 lines
Quoted 11 lines
hw/cxl/cxl-device-utils.c +11
-0+static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) { +memcpy (cxl_dstate->mbox_reg_state + offset, &value, size); +return ; + } + +/*
Quoted 8 lines
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 119
Quoted 6 lines
hw/cxl/cxl-device-utils.c +6
-0+* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD ();
RCU_READ_LOCKdoesn't prevent concurrent writes, although the BQL should.
Indeed, odd choice that I'd missed completely. I'm don't think we need anything at all until we consider implementing background commands / secondary mailbox. So for now I'll adjust the comment a little to say a lock 'would be needed'. and take no lock at all.
Quoted 25 lines
Quoted 25 lines
hw/cxl/cxl-device-utils.c +24
-0+ +switch (size) { +case 4 : +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +mailbox_mem_writeq (cxl_dstate->mbox_reg_state64, offset, value); +break ; +default : +g_assert_not_reached (); + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 ,
Quoted 8 lines
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 149
Quoted 6 lines
hw/cxl/cxl-device-utils.c +6
-0+ .unaligned = false, + }, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + },
with_attrs?
As before I don't follow. 8 byte is maximum size so should be fine.
Quoted 118 lines
Quoted 118 lines
(quoted diff) +2
-0+}; +hw/cxl/cxl-mailbox-utils.c new file +109
-0@@ -0,0 +1,201 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define define_mailbox_handler_zeroed (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_const (name, data) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = sizeof(data); \ + memcpy(cmd->payload, data, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_nop (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \ + } + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c line 115
Quoted 6 lines
hw/cxl/cxl-mailbox-utils.c new file +6
-0+ +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid;static?
done
Quoted 4 lines
but really this seems more like something that should be in a property of the device. Will it always be fixed or is it something the user might want to tweak?
It's fixed as it comes straight from the spec. I'll add a comment to make that clear where the UUID is below.
Quoted 18 lines
Quoted 18 lines
hw/cxl/cxl-mailbox-utils.c new file +17
-0+ +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; +
Quoted 8 lines
Quoted 5 lines
hw/cxl/cxl-mailbox-utils.c new file +4
-0+/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { + qemu_log_mask(LOG_UNIMP, "Corrupt internal state forfirmware\n");
is this actually UNIMP or has something gone wrong internally or is the guest doing something wrong?
It can't happen. I'd guess this is probably a left over from some earlier more complex implementation.
As things currently stand we only call this function if that bit is set.
I'll drop the check.
Quoted 43 lines
Quoted 43 lines
hw/cxl/cxl-mailbox-utils.c new file +42
-0+return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { +qemu_log_mask (LOG_UNIMP ,"Command %04xh not implemented\n" , + set <<8 | cmd); + goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + + cxl_dstate->mbox_reg_state64[A_CXL_DEV_MAILBOX_CMD /8 ] = command_reg; + cxl_dstate->mbox_reg_state64[A_CXL_DEV_MAILBOX_STS /8 ] = status_reg; + +/* Tell the host we're done */ + ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c line 184
Quoted 6 lines
hw/cxl/cxl-mailbox-utils.c new file +6
-0+ DOORBELL, 0); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ;I'm curious as to where this magic number came from.
CXL 2.0: Table 169 There are a couple defined so far but this is the only we've yet implemented in Qemu.
It has to be the same for all devices because this is a spec defined query.
Quoted 18 lines
Quoted 18 lines
hw/cxl/cxl-mailbox-utils.c new file +17
-0+ +for (int set =0 ; set <256 ; set++) { +for (int cmd =0 ; cmd <256 ; cmd++) { +if (cxl_cmd_set[set][cmd].handler) { +struct cxl_cmd *c = &cxl_cmd_set[set][cmd]; +struct cel_log *log = + &cxl_dstate->cel_log[cxl_dstate->cel_size]; + + log->opcode = (set <<8 ) | cmd; + log->effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}
...
Re: [PATCH v4 05/42] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Jonathan CameronQuoted 87 lines · Jonathan Cameron
Quoted 75 lines · On Wed, 26 Jan 2022 18:17:12 +0000 Alex Bennée <[email protected]> wrote:
Quoted 73 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This implements all device MMIO up to the first capability. That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-device-utils.c | 105 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_device.h | 28 +++++++++- 3 files changed, 133 insertions(+), 1 deletion(-)hw/cxl/cxl-device-utils.c new file +50
-0@@ -0,0 +1,105 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +/* +* Device registers have no restrictions per the spec, and so fall back to the +* default memory mapped register rules in 8.2: +* Software shall use CXL.io Memory Read and Write to access memory mapped +* register defined in this section. Unless otherwise specified, software +* shall restrict the accesses width based on the following: +* • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes +* quantity. +* • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 +* Bytes +* • The address shall be a multiple of the access width, e.g. when +* accessing a register as a 4 Byte quantity, the address shall be +* multiple of 4. +* • The accesses shall map to contiguous bytes.If these rules are not +* followed, the behavior is undefined +*/ + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +return cxl_dstate->caps_reg_state32[offset /4 ]; +} + +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +return 0 ; +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL ,/* status register is read only */ + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =1 , + .max_access_size =8 , + .unaligned = false,Quoted 8 lines
hw/cxl/cxl-device-utils.c line 56
Quoted 6 lines
hw/cxl/cxl-device-utils.c new file +6
-0+ }, + .impl = { + .min_access_size = 1, + .max_access_size = 8, + }, +};I think for >64 bit registers you need to use the
read_with_attrs
Quoted 4 lines
I don't follow this comment. Max access to registers is 64 bits. A few are documented as 128 bit or indeed larger in the spec, but the access is as if they were multiple 64 bit registers accesses. It's not permissible to do a single 128bit access for example.
No that was my brain fart - of course 8 bytes = 64 bit which is fine for the current accesses functions (unless you want bus faults).
Quoted 9 lines
The F4 errata clarified that - previously it was rather unclear what the restrictions on access to the larger registers were.
I've updated a few comments on this to reflect the errata.
Thanks,
Jonathan
Signature · 1 line
Alex Bennée
Re: [PATCH v4 10/42] hw/cxl/device: Add log commands (8.2.9.4) + CEL
In reply to Alex BennéeQuoted 57 lines · On Thu, 27 Jan 2022 11:55:47 +0000 Alex Bennée <[email protected]> wrote:
Quoted 55 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 67 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)hw/cxl/cxl-mailbox-utils.c +7
-0@@ -46,6 +46,9 @@ enum { TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1 + LOGS = 0x04, +#define GET_SUPPORTED 0x0 +#define GET_LOG 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -122,6 +125,8 @@ define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY );declare_mailbox_handler (TIMESTAMP_GET );declare_mailbox_handler (TIMESTAMP_SET ); +declare_mailbox_handler (LOGS_GET_SUPPORTED ); +declare_mailbox_handler (LOGS_GET_LOG );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -137,6 +142,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ),CXL_CMD (TIMESTAMP ,GET ,0 ,0 ),CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ), +CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ), + CXL_CMD(LOGS, GET_LOG, 0x18, 0), };#undef CXL_CMD@@ -188,6 +195,66 @@ define_mailbox_handler(TIMESTAMP_SET)QemuUUID cel_uuid;
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c define_mailbox_handler(TIMESTAMP_SET) · line 200
Quoted 3 lines
hw/cxl/cxl-mailbox-utils.c +3
-0+/* 8.2.9.4.1 */ +define_mailbox_handler (LOGS_GET_SUPPORTED ) +{Here is where I get a bit wary of the
define_mailbox_handlerdefine which from what I can tell just hides the declarations. This makes the handling of things like *cmd rather opaque. There is an argument for the boilerplate definitions (_nopand_zeroed) but perhaps not these.
Agreed. I think these macros got a bit too clever.
I debated keeping the CXL_CMD one but that then forces us to have ugly mixed lower case and upper case function names, so I've dropped that as well.
I'm debating whether to go with
[EVENTS][GET] = ...
[EVENTS][SET] = ...
vs
[EVENTS] = {
[GET] = { ..
[SET] = { ..
},For now I'll go with the [][] variant. The other one may make more sense as we add more commands.
Reorganizing the code a little gets rid of the need for the forward declarations as well so end result is less code than with the macros even if a few corners are a little repetitive.
Thanks,
Jonathan
Quoted 63 lines
Quoted 61 lines
hw/cxl/cxl-mailbox-utils.c +57
-0+struct { +uint16_t entries; +uint8_t rsvd[6 ]; +struct { +QemuUUID uuid; +uint32_t size; + } log_entries[1 ]; + } __attribute__((packed)) *supported_logs = (void *)cmd->payload; +_Static_assert (sizeof (*supported_logs) ==0x1c ,"Bad supported log size" ); + + supported_logs->entries =1 ; + supported_logs->log_entries[0 ].uuid = cel_uuid; + supported_logs->log_entries[0 ].size =4 * cxl_dstate->cel_size; + + *len =sizeof (*supported_logs); +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.4.2 */ +define_mailbox_handler (LOGS_GET_LOG ) +{ +struct { +QemuUUID uuid; +uint32_t offset; +uint32_t length; + } __attribute__((packed,__aligned__ (16 ))) *get_log = (void *)cmd->payload; + +/* +* 8.2.9.4.2 +* The device shall return Invalid Parameter if the Offset or Length +* fields attempt to access beyond the size of the log as reported by Get +* Supported Logs. +* +* XXX: Spec is wrong, "Invalid Parameter" isn't a thing. +* XXX: Spec doesn't address incorrect UUID incorrectness. +* +* The CEL buffer is large enough to fit all commands in the emulation, so +* the only possible failure would be if the mailbox itself isn't big +* enough. +*/ +if (get_log->offset + get_log->length > cxl_dstate->payload_size) { +return CXL_MBOX_INVALID_INPUT ; + } + +if (!qemu_uuid_is_equal (&get_log->uuid, &cel_uuid)) { +return CXL_MBOX_UNSUPPORTED ; + } + +/* Store off everything to local variables so we can wipe out the payload */ + *len = get_log->length; + +memmove (cmd->payload, cxl_dstate->cel_log + get_log->offset, + get_log->length); + +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Re: [PATCH v4 27/42] hw/cxl/device: Implement get/set Label Storage Area (LSA)
In reply to Jonathan CameronQuoted 29 lines · On Mon, 24 Jan 2022 17:16:50 +0000 Jonathan Cameron…
From: Ben Widawsky <[email protected]>
Implement get and set handlers for the Label Storage Area used to hold data describing persistent memory configuration so that it can be ensured it is seen in the same configuration after reboot.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 54 +++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 56 ++++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 5 ++++ 3 files changed, 114 insertions(+), 1 deletion(-)hw/cxl/cxl-mailbox-utils.c +2
-0@@ -55,6 +55,8 @@ enum {#define MEMORY_DEVICE 0x0 CCLS = 0x41,#define GET_PARTITION_INFO 0x0 +#define GET_LSA 0x2 +#define SET_LSA 0x3 };/* 8.2.8.4.5.1 Command Return Codes */
hw/cxl/cxl-mailbox-utils.c declare_mailbox_handler(LOGS_GET_SUPPORTED); · line 145
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c +3
-0@@ -136,8 +138,11 @@ declare_mailbox_handler(LOGS_GET_SUPPORTED);declare_mailbox_handler (LOGS_GET_LOG );declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );declare_mailbox_handler (CCLS_GET_PARTITION_INFO ); +declare_mailbox_handler (CCLS_GET_LSA ); +declare_mailbox_handler (CCLS_SET_LSA );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_DATA_CHANGE (1 << 1)
Notice whilst reworking this code to drop the macro compexity. This should be (1 << 2) I think as it is bit 2 in the CEL.
Quoted 3 lines
hw/cxl/cxl-mailbox-utils.c +0
-0#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)
Re: [PATCH v4 09/42] hw/cxl/device: Timestamp implementation (8.2.9.3)
In reply to Alex BennéeQuoted 64 lines · On Thu, 27 Jan 2022 11:50:08 +0000 Alex Bennée <[email protected]> wrote:
Quoted 62 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.
Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 53 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 59 insertions(+)hw/cxl/cxl-mailbox-utils.c +19
-0@@ -43,6 +43,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (TIMESTAMP_GET ); +declare_mailbox_handler (TIMESTAMP_SET );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)#define CXL_CMD (s, c, in, cel_effect) \@@ -129,10 +135,57 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (TIMESTAMP ,GET ,0 ,0 ), + CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE), };#undef CXL_CMD +/* +* 8.2.9.3.1 +*/ +define_mailbox_handler (TIMESTAMP_GET ) +{ +struct timespec ts; +uint64_t delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done;
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c static struct cxl_cmd cxl_cmd_set[256][256] = { · line 158
Quoted 4 lines
hw/cxl/cxl-mailbox-utils.c +4
-0+ } + +/* First find the delta from the last time the host set the time. */ +clock_gettime (CLOCK_REALTIME , &ts);Could you consider using
qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)? Otherwise by introducing a dependency on real time you'll loose the ability to get deterministic execution via icount.
I don't have a strong opinion either way, so Ill go with qemu_clock_get_ns() and see if any comments on it in v5.
I've also updated the commit message as the catch all errata which was F4 included a clarification that the timestamp was as implemented here.
Thanks,
Jonathan
Re: [PATCH v4 13/42] hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Alex BennéeQuoted 110 lines · On Thu, 27 Jan 2022 13:59:56 +0000 Alex Bennée <[email protected]> wrote:
Quoted 108 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in
_SBnamed ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan.Cameron <[email protected]>--- hw/pci-bridge/pci_expander_bridge.c | 98 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 ++ 3 files changed, 109 insertions(+), 2 deletions(-)hw/pci-bridge/pci_expander_bridge.c +30
-2@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h" @@ -56,6 +57,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +struct PXBDev {/*< private >*/ PCIDevice parent_obj;@@ -66,8 +71,19 @@ struct PXBDev {bool bypass_iommu; }; +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }@@ -76,6 +92,9 @@ static GList *pxb_dev_list;#define TYPE_PXB_HOST "pxb-host" +#define TYPE_PXB_CXL_HOST "pxb-cxl-host" +#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);@@ -112,11 +131,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;@@ -218,6 +246,16 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); }
Quoted 3 lines
hw/pci-bridge/pci_expander_bridge.c static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin) · line 251
Quoted 3 lines
hw/pci-bridge/pci_expander_bridge.c +3
-0+static void pxb_dev_reset (DeviceState *dev) +{ +CXLHost *cxl =PXB_CXL_HOST (dev);
Having fixed the stuff you observed below...
It's not that device by rather the PXB_CXL_DEV and we can access the CXLHost via PXB_CXL_DEV(dv)->cxl.cxl_host_bridge()
at which point this seems to be working.
Quoted 13 lines
Quoted 13 lines
hw/pci-bridge/pci_expander_bridge.c +7
-0+CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT ,8 ); +} +static gint pxb_compare (gconstpointer a,gconstpointer b) {const PXBDev *pxb_a = a, *pxb_b = b;@@ -290,6 +328,11 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,pci_config_set_class (dev->config,PCI_CLASS_BRIDGE_HOST );
Quoted 9 lines
hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, · line 335
Quoted 6 lines
hw/pci-bridge/pci_expander_bridge.c +5
-0pxb_dev_list =g_list_insert_sorted (pxb_dev_list, pxb, pxb_compare); + +if (type ==CXL ) { +pxb_dev_reset (ds); + } +Couldn't this just be done in the cxl realize function after it calls the common code?
yup.
Quoted 16 lines
Quoted 5 lines
hw/pci-bridge/pci_expander_bridge.c +0
-0return ; err_register_bus:hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_class_init(ObjectClass *klass, void *data) · line 389
Quoted 9 lines
hw/pci-bridge/pci_expander_bridge.c +6
-0@@ -338,6 +381,12 @@ static void pxb_dev_class_init(ObjectClass *klass, void *data)device_class_set_props (dc, pxb_dev_properties); dc->hotpluggable = false;set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* +* Reset doesn't seem to actually be called, but maybe it will in the +* future? +*/ + dc->reset = pxb_dev_reset;Surely because this should be in
pxb_cxl_dev_class_init?
That would indeed do it.
Thanks.
Jonathan
Quoted 114 lines
Quoted 112 lines
hw/pci-bridge/pci_expander_bridge.c +45
-0}static const TypeInfo pxb_dev_info = {@@ -389,13 +438,58 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)hw/pci/pci.c +7
-0@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);@@ -2892,6 +2898,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);include/hw/pci/pci.h +6
-0@@ -409,6 +409,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"typedef void (*pci_bus_dev_fn )(PCIBus *b,PCIDevice *d,void *opaque);typedef void (*pci_bus_fn )(PCIBus *b,void *opaque);@@ -768,6 +769,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
Re: [PATCH v4 14/42] tests/acpi: allow DSDT.viot table changes.
In reply to Alex BennéeQuoted 9 lines · On Thu, 27 Jan 2022 14:06:42 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The next patch unifies some of the PCI host bridge DSDT generation code and results in some minor changes to this file.
I'd just squash this in with the patch that makes the change... otherwise you risk breaking bisectablility.
Hi Alex,
This sequence of 3 patches is as described intests/qtest/bios-tables-test.c
According to description there the point is to allow whoever picks these patches up to just change the 3rd patch if the tables need updating to reflect some other change.
It would be a lot easier obviously to just squash it, but given the process is laid out in that file, I'd rather leave it like this.
Or am I missing a reason this particular one is better squashed?
Thanks,
Jonathan
Quoted 16 lines
Quoted 14 lines
Signed-off-by Jonathan Cameron <[email protected]>--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)tests/qtest/bios-tables-test-allowed-diff.h +1
-0@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.viot",
Re: [PATCH v4 14/42] tests/acpi: allow DSDT.viot table changes.
Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Thu, 27 Jan 2022 14:06:42 +0000 Alex Bennée <[email protected]> wrote:
Quoted 4 lines · Jonathan Cameron
From: Jonathan Cameron <[email protected]>
The next patch unifies some of the PCI host bridge DSDT generation code and results in some minor changes to this file.
I'd just squash this in with the patch that makes the change... otherwise you risk breaking bisectablility.
Hi Alex,
This sequence of 3 patches is as described in
tests/qtest/bios-tables-test.c
Quoted 7 lines
According to description there the point is to allow whoever picks these patches up to just change the 3rd patch if the tables need updating to reflect some other change.
It would be a lot easier obviously to just squash it, but given the process is laid out in that file, I'd rather leave it like this.
Hmm I shall defer to the maintainer here. I assume this process is to deal with complex changes over several commits. So ignore that comment for now.
Quoted 25 lines
Or am I missing a reason this particular one is better squashed?
Thanks,
Jonathan
Quoted 16 lines
Quoted 14 lines
Signed-off-by Jonathan Cameron <[email protected]>--- tests/qtest/bios-tables-test-allowed-diff.h | 1 + 1 file changed, 1 insertion(+)tests/qtest/bios-tables-test-allowed-diff.h +1
-0@@ -1 +1,2 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.viot",
Signature · 1 line
Alex Bennée
Re: [PATCH v4 13/42] hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Jonathan CameronQuoted 118 lines · On Fri, 28 Jan 2022 18:20:01 +0000 Jonathan Cameron…
Quoted 110 lines · On Thu, 27 Jan 2022 13:59:56 +0000 Alex Bennée <[email protected]> wrote:
Quoted 108 lines · Jonathan Cameron
From: Ben Widawsky <[email protected]>
This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in
_SBnamed ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan.Cameron <[email protected]>--- hw/pci-bridge/pci_expander_bridge.c | 98 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 ++ 3 files changed, 109 insertions(+), 2 deletions(-)hw/pci-bridge/pci_expander_bridge.c +30
-2@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h" @@ -56,6 +57,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +struct PXBDev {/*< private >*/ PCIDevice parent_obj;@@ -66,8 +71,19 @@ struct PXBDev {bool bypass_iommu; }; +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }@@ -76,6 +92,9 @@ static GList *pxb_dev_list;#define TYPE_PXB_HOST "pxb-host" +#define TYPE_PXB_CXL_HOST "pxb-cxl-host" +#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);@@ -112,11 +131,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;@@ -218,6 +246,16 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)return pin -PCI_SLOT (pxb->devfn); }Quoted 3 lines
hw/pci-bridge/pci_expander_bridge.c static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin) · line 251
Quoted 3 lines
hw/pci-bridge/pci_expander_bridge.c +3
-0+static void pxb_dev_reset (DeviceState *dev) +{ +CXLHost *cxl =PXB_CXL_HOST (dev);Having fixed the stuff you observed below...
Quoted 4 lines
It's not that device by rather the
PXB_CXL_DEVand we can access the CXLHost via PXB_CXL_DEV(dv)->cxl.cxl_host_bridge()
at which point this seems to be working.
The infrastructure to do this isn't in place until patch 24 so I'll have to pull some of it into this patch to allow this reset to work.
Jonathan
Quoted 163 lines
Quoted 14 lines
Quoted 14 lines
hw/pci-bridge/pci_expander_bridge.c +7
-0+CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_ROOT_PORT ); +ARRAY_FIELD_DP32 (reg_state,CXL_HDM_DECODER_CAPABILITY ,TARGET_COUNT ,8 ); +} +static gint pxb_compare (gconstpointer a,gconstpointer b) {const PXBDev *pxb_a = a, *pxb_b = b;@@ -290,6 +328,11 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,pci_config_set_class (dev->config,PCI_CLASS_BRIDGE_HOST );Quoted 9 lines
hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, · line 335
Quoted 6 lines
hw/pci-bridge/pci_expander_bridge.c +5
-0pxb_dev_list =g_list_insert_sorted (pxb_dev_list, pxb, pxb_compare); + +if (type ==CXL ) { +pxb_dev_reset (ds); + } +Couldn't this just be done in the cxl realize function after it calls the common code?
yup.
Quoted 16 lines
Quoted 5 lines
hw/pci-bridge/pci_expander_bridge.c +0
-0return ; err_register_bus:hw/pci-bridge/pci_expander_bridge.c static void pxb_dev_class_init(ObjectClass *klass, void *data) · line 389
Quoted 9 lines
hw/pci-bridge/pci_expander_bridge.c +6
-0@@ -338,6 +381,12 @@ static void pxb_dev_class_init(ObjectClass *klass, void *data)device_class_set_props (dc, pxb_dev_properties); dc->hotpluggable = false;set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* +* Reset doesn't seem to actually be called, but maybe it will in the +* future? +*/ + dc->reset = pxb_dev_reset;Surely because this should be in
pxb_cxl_dev_class_init?That would indeed do it.
Thanks.
Jonathan
Quoted 114 lines
Quoted 112 lines
hw/pci-bridge/pci_expander_bridge.c +45
-0}static const TypeInfo pxb_dev_info = {@@ -389,13 +438,58 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)hw/pci/pci.c +7
-0@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);@@ -2892,6 +2898,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);include/hw/pci/pci.h +6
-0@@ -409,6 +409,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"typedef void (*pci_bus_dev_fn )(PCIBus *b,PCIDevice *d,void *opaque);typedef void (*pci_bus_fn )(PCIBus *b,void *opaque);@@ -768,6 +769,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
Re: [PATCH v4 23/42] tests/acpi: allow CEDT table addition
In reply to Jonathan CameronQuoted 24 lines · On Mon, 24 Jan 2022 17:16:46 +0000 Jonathan Cameron…
From: Ben Widawsky <[email protected]>
Following patches will add a new ACPI table, the
CXL Early Discovery Table (CEDT).Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- tests/data/acpi/pc/CEDT | 0 tests/data/acpi/q35/CEDT | 0 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++ 3 files changed, 2 insertions(+)tests/data/acpi/pc/CEDT new file +0
-0tests/data/acpi/q35/CEDT new file +0
-0tests/qtest/bios-tables-test-allowed-diff.h +0
-0
tests/qtest/bios-tables-test-allowed-diff.h line 3
Quoted 3 lines
tests/qtest/bios-tables-test-allowed-diff.h +2
-0@@ -1 +1,3 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/pc/CEDT" , +"tests/data/acpi/q35/CEDT",
Whilst reordering the series to allow partial sets to be picked up it occurred to me that there is reason why we should now generate a CEDT table unless cxl=on is set for the relevant machine.
Adding the relevant conditions means we can avoid this dance to add effectively empty CEDT tables. I have added a proper test for to bios-test-tables but that can be introduced once everything is in place, rather than as we go along (whereas for qtests/cxl-test.c I am doing a series of updates as the functionality is built up.).
Makes for a simpler patch series to update.
Jonathan
Re: [PATCH v4 23/42] tests/acpi: allow CEDT table addition
Quoted 26 lines · Jonathan Cameron
Quoted 23 lines · On Mon, 24 Jan 2022 17:16:46 +0000 Jonathan Cameron…
From: Ben Widawsky <[email protected]>
Following patches will add a new ACPI table, the
CXL Early Discovery Table (CEDT).Signed-off-by Ben Widawsky <[email protected]>Signed-off-by Jonathan Cameron <[email protected]>--- tests/data/acpi/pc/CEDT | 0 tests/data/acpi/q35/CEDT | 0 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++ 3 files changed, 2 insertions(+)tests/data/acpi/pc/CEDT new file +0
-0tests/data/acpi/q35/CEDT new file +0
-0tests/qtest/bios-tables-test-allowed-diff.h +0
-0
Quoted 8 lines
tests/qtest/bios-tables-test-allowed-diff.h line 3
Quoted 4 lines
tests/qtest/bios-tables-test-allowed-diff.h +2
-0@@ -1 +1,3 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/pc/CEDT" , +"tests/data/acpi/q35/CEDT",Whilst reordering the series to allow partial sets to be picked up it occurred to me that there is reason why we should now generate a CEDT table unless cxl=on is set for the relevant machine.
Absolutely. But this is just an example, it's an instance of a bigger issue: skipping new features when they are off reduces the chance of legacy guest breakage.
Quoted 10 lines
Adding the relevant conditions means we can avoid this dance to add effectively empty CEDT tables. I have added a proper test for to bios-test-tables but that can be introduced once everything is in place, rather than as we go along (whereas for
qtests/cxl-test.cI am doing a series of updates as the functionality is built up.).Makes for a simpler patch series to update.
Jonathan