CXL 2.0 Support
v2 cover CXL 2.0 Support
Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbed device_reg_init_common() (Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)
New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support
(There are some new patches that I just named 'v2' for simplicity's sake)
Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).
The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.
Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).
CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).
We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs
As mentioned above there are three benefits to carrying this enabling in upstream QEMU:
1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.
2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (nfit_test). In retrospect a QEMU emulation of nfit_test capabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.
3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?
There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity
The flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.
The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case
[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/
Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb _UID through host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests
Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
MAINTAINERS | 6 + hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl.c | 173 ++++++++++ hw/acpi/meson.build | 1 + hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++ hw/core/numa.c | 3 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 193 +++++++++++ hw/cxl/cxl-device-utils.c | 278 ++++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 386 ++++++++++++++++++++++ hw/cxl/meson.build | 5 + hw/i386/acpi-build.c | 87 ++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 2 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 334 +++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 32 +- hw/pci/pcie.c | 30 ++ hw/ppc/spapr.c | 2 + include/hw/acpi/cxl.h | 27 ++ include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 30 ++ include/hw/cxl/cxl_component.h | 187 +++++++++++ include/hw/cxl/cxl_device.h | 210 ++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 15 + include/hw/pci/pci_bridge.h | 25 ++ include/hw/pci/pci_bus.h | 8 + include/hw/pci/pci_ids.h | 1 + include/standard-headers/linux/pci_regs.h | 1 + monitor/hmp-cmds.c | 15 + qapi/machine.json | 1 + tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/pc/DSDT | Bin 5065 -> 5065 bytes tests/data/acpi/pc/DSDT.acpihmat | Bin 6390 -> 6390 bytes tests/data/acpi/pc/DSDT.bridge | Bin 6924 -> 6924 bytes tests/data/acpi/pc/DSDT.cphp | Bin 5529 -> 5529 bytes tests/data/acpi/pc/DSDT.dimmpxm | Bin 6719 -> 6719 bytes tests/data/acpi/pc/DSDT.hpbridge | Bin 5026 -> 5026 bytes tests/data/acpi/pc/DSDT.hpbrroot | Bin 3084 -> 3084 bytes tests/data/acpi/pc/DSDT.ipmikcs | Bin 5137 -> 5137 bytes tests/data/acpi/pc/DSDT.memhp | Bin 6424 -> 6424 bytes tests/data/acpi/pc/DSDT.numamem | Bin 5071 -> 5071 bytes tests/data/acpi/pc/DSDT.roothp | Bin 5261 -> 5261 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/DSDT | Bin 7801 -> 7801 bytes tests/data/acpi/q35/DSDT.acpihmat | Bin 9126 -> 9126 bytes tests/data/acpi/q35/DSDT.bridge | Bin 7819 -> 7819 bytes tests/data/acpi/q35/DSDT.cphp | Bin 8265 -> 8265 bytes tests/data/acpi/q35/DSDT.dimmpxm | Bin 9455 -> 9455 bytes tests/data/acpi/q35/DSDT.ipmibt | Bin 7876 -> 7876 bytes tests/data/acpi/q35/DSDT.memhp | Bin 9160 -> 9160 bytes tests/data/acpi/q35/DSDT.mmio64 | Bin 8932 -> 8932 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7807 -> 7807 bytes tests/qtest/cxl-test.c | 93 ++++++ tests/qtest/meson.build | 4 + 65 files changed, 2777 insertions(+), 30 deletions(-) create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT create mode 100644 tests/qtest/cxl-test.c
Signature · 1 line
2.30.0
RFC PATCH v2 1/32 Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
From: Jonathan Cameron <[email protected]>
This hasn't yet been added to the linux kernel tree, so for purposes of this RFC just add it locally.
--- include/standard-headers/linux/pci_regs.h | 1 + 1 file changed, 1 insertion(+)
include/standard-headers/linux/pci_regs.h +1 -0
@@ -723,6 +723,7 @@#define PCI_EXT_CAP_ID_DPC 0x1D/* Downstream Port Containment */ #define PCI_EXT_CAP_ID_L1SS 0x1E/* L1 PM Substates */ #define PCI_EXT_CAP_ID_PTM 0x1F/* Precision Time Measurement */ +#define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */ #define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */ #define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PL_16GT
Signature · 1 line
2.30.0
RFC PATCH v2 2/32 hw/pci/cxl: Add a CXL component type (interface)
In reply to Ben WidawskyA CXL component is a hardware entity that implements CXL component registers from the CXL 2.0 spec (8.2.3). Currently these represent 3 general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)
A CXL component can be conceptually thought of as a PCIe device with extra functionality when enumerated and enabled. For this reason, CXL does here, and will continue to add on to existing PCI code paths.
Host bridges will typically need to be handled specially and so they can implement this newly introduced interface or not. All other components should implement this interface. Implementing this interface allows the core pci code to treat these devices as special where appropriate.
--- hw/pci/pci.c | 10 ++++++++++ include/hw/pci/pci.h | 8 ++++++++ 2 files changed, 18 insertions(+)
hw/pci/pci.c +10 -0
@@ -194,6 +194,11 @@ static const TypeInfo pci_bus_info = { .class_init = pci_bus_class_init, }; +static const TypeInfo cxl_interface_info = { + .name =INTERFACE_CXL_DEVICE , + .parent =TYPE_INTERFACE , +}; +static const TypeInfo pcie_interface_info = { .name =INTERFACE_PCIE_DEVICE , .parent =TYPE_INTERFACE ,
@@ -2115,6 +2120,10 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp) pci_dev->cap_present |=QEMU_PCI_CAP_EXPRESS ; } +if (object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE )) { + pci_dev->cap_present |=QEMU_PCIE_CAP_CXL ; + } + pci_dev =do_pci_register_device (pci_dev,object_get_typename (OBJECT (qdev)), pci_dev->devfn, errp);
@@ -2841,6 +2850,7 @@ static void pci_register_types(void)type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info);type_register_static (&conventional_pci_interface_info); +type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);type_register_static (&pci_device_type_info); }
include/hw/pci/pci.h +8 -0
@@ -194,6 +194,8 @@ enum { QEMU_PCIE_LNKSTA_DLLLA = (1 << QEMU_PCIE_LNKSTA_DLLLA_BITNR),#define QEMU_PCIE_EXTCAP_INIT_BITNR 9 QEMU_PCIE_EXTCAP_INIT = (1 << QEMU_PCIE_EXTCAP_INIT_BITNR), +#define QEMU_PCIE_CXL_BITNR 10 + QEMU_PCIE_CAP_CXL = (1 << QEMU_PCIE_CXL_BITNR), };#define TYPE_PCI_DEVICE "pci-device"
@@ -201,6 +203,12 @@ typedef struct PCIDeviceClass PCIDeviceClass;DECLARE_OBJ_CHECKERS (PCIDevice, PCIDeviceClass,PCI_DEVICE ,TYPE_PCI_DEVICE ) +/* +* Implemented by devices that can be plugged on CXL buses. In the spec, this is +* actually a "CXL Component, but we name it device to match the PCI naming. + */ +#define INTERFACE_CXL_DEVICE "cxl-device" +/* Implemented by devices that can be plugged on PCI Express buses */ #define INTERFACE_PCIE_DEVICE "pci-express-device"
Signature · 1 line
2.30.0
RFC PATCH v2 3/32 hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
In reply to Ben WidawskyA CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.
The cxl_pci header manages bits associated with PCI, specifically the DVSEC and related fields. The cxl_component.h variant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] regisetrs.
None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.
v2: Improve assertions in DVSEC creation (Jonathan)
Use 'n' for HDM register offsets (Jonathan) Correct revision ID for extensions (Jonathan) Minor cleanups and clarifications (Jonathan)
--- MAINTAINERS | 6 + hw/Kconfig | 1 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 193 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 3 + hw/meson.build | 1 + include/hw/cxl/cxl.h | 17 +++ include/hw/cxl/cxl_component.h | 187 ++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++++ 9 files changed, 549 insertions(+) create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_pci.h
MAINTAINERS +6 -0
@@ -2225,6 +2225,12 @@ F: qapi/block*.json F: qapi/transaction.json T: git https://repo.or.cz/qemu/armbru.git block-next +Compute Express Link +M: Ben Widawsky <[email protected]> +S: Supported +F: hw/cxl/ +F: include/hw/cxl/ + Dirty Bitmaps M: Eric Blake <[email protected]> M: Vladimir Sementsov-Ogievskiy <[email protected]>
hw/Kconfig +1 -0
@@ -6,6 +6,7 @@ source audio/Kconfig source block/Kconfig source char/Kconfig source core/Kconfig +source cxl/Kconfig source display/Kconfig source dma/Kconfig source gpio/Kconfig
hw/cxl/Kconfig new file +3 -0
@@ -0,0 +1,3 @@ +config CXL + bool + default y if PCI_EXPRESS
hw/cxl/cxl-component-utils.c new file +193 -0
193 lines not on this page · raw message
hw/cxl/meson.build new file +3 -0
@@ -0,0 +1,3 @@ +softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', +))
hw/meson.build +1 -0
@@ -6,6 +6,7 @@ subdir('block') subdir('char') subdir('core') subdir('cpu') +subdir('cxl') subdir('display') subdir('dma') subdir('gpio')
include/hw/cxl/cxl.h new file +17 -0
@@ -0,0 +1,17 @@ +/* +* QEMU CXL Support +* +* Copyright (c) 2020 Intel +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#ifndef CXL_H +#define CXL_H + +#include "cxl_pci.h" +#include "cxl_component.h" + +#endif +
include/hw/cxl/cxl_component.h new file +187 -0
187 lines not on this page · raw message
include/hw/cxl/cxl_pci.h new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
2.30.0
RFC PATCH v2 4/32 hw/cxl/device: Introduce a CXL device (8.2.8)
In reply to Ben WidawskyA CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.
This new device type is specifically catered towards the eventually implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.
v2: Remove error codes not yet used (Jonathan)
Fix interrupt enable bit width (Jonathan) Add comment for weird register size (Jonathan) Remove commands/registers not yet used (Ben)
--- include/hw/cxl/cxl.h | 1 + include/hw/cxl/cxl_device.h | 151 ++++++++++++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 include/hw/cxl/cxl_device.h
include/hw/cxl/cxl.h +1 -0
@@ -12,6 +12,7 @@#include "cxl_pci.h" #include "cxl_component.h" +#include "cxl_device.h" #endif
include/hw/cxl/cxl_device.h new file +151 -0
151 lines not on this page · raw message
Signature · 1 line
2.30.0
RFC PATCH v2 5/32 hw/cxl/device: Implement the CAP array (8.2.8.1-2)
In reply to Ben WidawskyThis implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
v2: Break out register alignment checks (Jonathan)
--- hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + 2 files changed, 73 insertions(+) create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +72 -0
@@ -0,0 +1,72 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +static int cxl_device_check_register_alignment (hwaddr offset,unsigned size) +{ +if (unlikely (offset & (size -1 ))) { +return 1 ; + } + +return 0 ; +} + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); +} + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ +memory_region_init ( + &cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MAILBOX_REGISTERS_LENGTH +CXL_MAILBOX_REGISTERS_OFFSET )); + +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +} + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =0 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); +}
hw/cxl/meson.build +1 -0
@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
Signature · 1 line
2.30.0
Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:
This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
v2: Break out register alignment checks (Jonathan)
Signed-off-by Ben Widawsky <[email protected]>
Hi Ben,
One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.
Will get to a full review after messing around ('testing') this a bit more ;)
Jonathan
Quoted 56 lines
--- hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + 2 files changed, 73 insertions(+) create mode 100644 hw/cxl/cxl-device-utils.chw/cxl/cxl-device-utils.c new file +44
-0@@ -0,0 +1,72 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +static int cxl_device_check_register_alignment (hwaddr offset,unsigned size) +{ +if (unlikely (offset & (size -1 ))) { +return 1 ; + } + +return 0 ; +} + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); +} + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 ,
hw/cxl/cxl-device-utils.c line 54
Quoted 10 lines
hw/cxl/cxl-device-utils.c new file +10
-0+ .max_access_size = 8, + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ +memory_region_init ( + &cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MAILBOX_REGISTERS_LENGTH +CXL_MAILBOX_REGISTERS_OFFSET ));
I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).
Quoted 27 lines
hw/cxl/cxl-device-utils.c new file +18
-0+ +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +} + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =0 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); +}hw/cxl/meson.build +1
-0@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:
This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
v2: Break out register alignment checks (Jonathan)
Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.
Will get to a full review after messing around ('testing') this a bit more ;)
Jonathan
Quoted 63 lines
--- hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + 2 files changed, 73 insertions(+) create mode 100644 hw/cxl/cxl-device-utils.chw/cxl/cxl-device-utils.c new file +51
-0@@ -0,0 +1,72 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +static int cxl_device_check_register_alignment (hwaddr offset,unsigned size) +{ +if (unlikely (offset & (size -1 ))) { +return 1 ; + } + +return 0 ; +} + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); +} + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
Quoted 10 lines
hw/cxl/cxl-device-utils.c line 54
Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3
-0+memory_region_init ( + &cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MAILBOX_REGISTERS_LENGTH +CXL_MAILBOX_REGISTERS_OFFSET ));I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).
Maybe I misunderstand, but this is the intended behavior. cxl_dstate->device_registers is the MemoryRegion container for all the subregions that are the actual device MMIO.
+------------------+
^ | |
| | unused |
| --------------------
| | memdev regs |
| --------------------
| | |
| | +--------------+ |
cxl_dstate-> | | | | |
device_registers | | | | |
| | |payload | |
| | |(2k currently)| |
| | | | |
| | | | |
| | +--------------+ |
| | mailbox regs |
| --------------------
| | device regs |
v --------------------
| caps regs |
BAR ------> +------------------+Perhaps I should add this as a comment in the code?
Quoted 28 lines
Quoted 27 lines
hw/cxl/cxl-device-utils.c new file +18
-0+ +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +} + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =0 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); +}hw/cxl/meson.build +1
-0@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
Quoted 122 lines · On Wed, 6 Jan 2021 08:49:48 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:
This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
v2: Break out register alignment checks (Jonathan)
Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.
Will get to a full review after messing around ('testing') this a bit more ;)
Jonathan
Quoted 63 lines
--- hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + 2 files changed, 73 insertions(+) create mode 100644 hw/cxl/cxl-device-utils.chw/cxl/cxl-device-utils.c new file +51
-0@@ -0,0 +1,72 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +static int cxl_device_check_register_alignment (hwaddr offset,unsigned size) +{ +if (unlikely (offset & (size -1 ))) { +return 1 ; + } + +return 0 ; +} + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); +} + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ Quoted 10 lines
hw/cxl/cxl-device-utils.c line 54
Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3
-0+memory_region_init ( + &cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MAILBOX_REGISTERS_LENGTH +CXL_MAILBOX_REGISTERS_OFFSET ));I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).Maybe I misunderstand, but this is the intended behavior.
cxl_dstate->device_registersis the MemoryRegion container for all the subregions that are the actual device MMIO.+------------------+ ^ | | | | unused | | -------------------- | | memdev regs | | -------------------- | | | | | +--------------+ | cxl_dstate-> | | | | | device_registers | | | | | | | |payload | | | | |(2k currently)| | | | | | | | | | | | | | +--------------+ | | | mailbox regs | | -------------------- | | device regs | v -------------------- | caps regs | BAR ------> +------------------+
Quoted 1 line
Perhaps I should add this as a comment in the code?
I agree with intent but above it is setting limit to the top of the mailbox regs, not the memdev regs which I'd expect to see.
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
hw/cxl/cxl-device-utils.c new file +18
-0+ +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +} + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =0 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); +}hw/cxl/meson.build +1
-0@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
Quoted 125 lines · Jonathan Cameron
Quoted 122 lines · On Wed, 6 Jan 2021 08:49:48 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:
This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.
v2: Break out register alignment checks (Jonathan)
Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.
Will get to a full review after messing around ('testing') this a bit more ;)
Jonathan
Quoted 63 lines
--- hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + 2 files changed, 73 insertions(+) create mode 100644 hw/cxl/cxl-device-utils.chw/cxl/cxl-device-utils.c new file +51
-0@@ -0,0 +1,72 @@ +/* +* CXL Utility library for devices +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "hw/cxl/cxl.h" + +static int cxl_device_check_register_alignment (hwaddr offset,unsigned size) +{ +if (unlikely (offset & (size -1 ))) { +return 1 ; + } + +return 0 ; +} + +static uint64_t caps_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); +} + +static const MemoryRegionOps caps_ops = { + .read = caps_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; + +void cxl_device_register_block_init (Object *obj,CXLDeviceState *cxl_dstate) +{ +/* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */ Quoted 10 lines
hw/cxl/cxl-device-utils.c line 54
Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3
-0+memory_region_init ( + &cxl_dstate->device_registers, obj,"device-registers" , +pow2ceil (CXL_MAILBOX_REGISTERS_LENGTH +CXL_MAILBOX_REGISTERS_OFFSET ));I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).Maybe I misunderstand, but this is the intended behavior.
cxl_dstate->device_registersis the MemoryRegion container for all the subregions that are the actual device MMIO.+------------------+ ^ | | | | unused | | -------------------- | | memdev regs | | -------------------- | | | | | +--------------+ | cxl_dstate-> | | | | | device_registers | | | | | | | |payload | | | | |(2k currently)| | | | | | | | | | | | | | +--------------+ | | | mailbox regs | | -------------------- | | device regs | v -------------------- | caps regs | BAR ------> +------------------+
Quoted 5 lines
Quoted 1 line
Perhaps I should add this as a comment in the code?
I agree with intent but above it is setting limit to the top of the mailbox regs, not the memdev regs which I'd expect to see.
oops :D - you're correct
Quoted 30 lines
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
hw/cxl/cxl-device-utils.c new file +18
-0+ +memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate, +"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); + +memory_region_add_subregion (&cxl_dstate->device_registers,0 , + &cxl_dstate->caps); +} + +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) +{ +uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; +const int cap_count =0 ; + +/* CXL Device Capabilities Array Register */ +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 ); +ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); +}hw/cxl/meson.build +1
-0@@ -1,3 +1,4 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', + 'cxl-device-utils.c', ))
RFC PATCH v2 6/32 hw/cxl/device: Add device status (8.2.8.3)
In reply to Ben WidawskyThis implements the CXL device status registers from 8.2.8.3.1 in the CXL 2.0 specification. It is capability ID 0001h.
v2: Use the reg alignment helper (Jonathan)
Add stubbed device_reg_init_common() (Ben)
--- hw/cxl/cxl-device-utils.c | 38 ++++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_device.h | 16 ++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c +37 -1
@@ -32,6 +32,32 @@ static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (cxl_dstate->caps_reg_state + offset, size); } +static uint64_t dev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +uint64_t retval =0 ; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (&retval, size); +} + +static const MemoryRegionOps dev_ops = { + .read = dev_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps caps_ops = { .read = caps_reg_read, .write =NULL ,
@@ -55,18 +81,28 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)memory_region_init_io (&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,"cap-array" ,CXL_DEVICE_REGISTERS_OFFSET -0 ); +memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate, +"device-status" ,CXL_DEVICE_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->device); } +static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =0 ; +const int cap_count =1 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_VERSION ,1 );ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY2 ,CAP_COUNT , cap_count); + +cxl_device_cap_init (cxl_dstate,DEVICE ,1 ); +device_reg_init_common (cxl_dstate); }
include/hw/cxl/cxl_device.h +16 -0
@@ -122,6 +122,22 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +#define cxl_device_cap_init (dstate, reg, cap_id) \ + do { \ + uint32_t *cap_hdrs = dstate->caps_reg_state32; \ + int which = R_CXL_DEV_##reg##_CAP_HDR0; \ + cap_hdrs[which] = \ + FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_ID, cap_id); \ + cap_hdrs[which] = FIELD_DP32( \ + cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1); \ + cap_hdrs[which + 1] = \ + FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1, \ + CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET); \ + cap_hdrs[which + 2] = \ + FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2, \ + CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH); \ + } while (0) + REG32(CXL_DEV_MAILBOX_CAP, 0) FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)FIELD (CXL_DEV_MAILBOX_CAP ,INT_CAP ,5 ,1 )
Signature · 1 line
2.30.0
RFC PATCH v2 7/32 hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Ben WidawskyThis is the beginning of implementing mailbox support for CXL 2.0 devices.
v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)
--- hw/cxl/cxl-device-utils.c | 122 ++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 173 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 27 +++++- 5 files changed, 322 insertions(+), 4 deletions(-) create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +121 -1
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (&retval, size); } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->mbox_reg_state + offset, size); +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + +stl_le_p ((uint8_t *)reg_state + offset, value); +} + +static void mailbox_mem_writeq (uint8_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + +stq_le_p (reg_state + offset, value); +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writeq (cxl_dstate->mbox_reg_state, offset, value); +break ; + } + +if (ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) +cxl_process_mailbox (cxl_dstate); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write =NULL ,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
hw/cxl/cxl-mailbox-utils.c new file +173 -0
173 lines not on this page · raw message
hw/cxl/meson.build +1 -0
@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))
include/hw/cxl/cxl.h +3 -0
@@ -14,5 +14,8 @@#include "cxl_component.h" #include "cxl_device.h" +#define COMPONENT_REG_BAR_IDX 0 +#define DEVICE_REG_BAR_IDX 2 +#endif
include/hw/cxl/cxl_device.h +24 -3
@@ -70,9 +70,10 @@ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) typedef struct cxl_device_state { - /* Boss container and caps registers */ + /* Main register container */ MemoryRegion device_registers; + MemoryRegion device; struct { MemoryRegion caps; union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >>2 ]; }; }; -MemoryRegion device; -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >>2 ]; + }; +struct { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };MemoryRegion *pmem;MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )
Signature · 1 line
2.30.0
Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:
This is the beginning of implementing mailbox support for CXL 2.0 devices.
v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)Signed-off-by Ben Widawsky <[email protected]>
Hi Ben,
I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.
The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain withmemcpy_from_io()
If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.
I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)
Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).
Quoted 101 lines
--- hw/cxl/cxl-device-utils.c | 122 ++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 173 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 27 +++++- 5 files changed, 322 insertions(+), 4 deletions(-) create mode 100644 hw/cxl/cxl-mailbox-utils.chw/cxl/cxl-device-utils.c +84
-0@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (&retval, size); } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->mbox_reg_state + offset, size); +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + +stl_le_p ((uint8_t *)reg_state + offset, value); +} + +static void mailbox_mem_writeq (uint8_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + +stq_le_p (reg_state + offset, value); +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writeq (cxl_dstate->mbox_reg_state, offset, value); +break ; + } + + if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+ DOORBELL)) +cxl_process_mailbox (cxl_dstate ); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 ,
Hack was to set this to 1 (and the impl below - no idea if both are needed).
Quoted 319 lines
hw/cxl/cxl-device-utils.c +27
-1+ .max_access_size = 8, + }, + .impl = { + .min_access_size = 4, + .max_access_size = 8, + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write = NULL,@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +173
-0@@ -0,0 +1,173 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; + +/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { +qemu_log_mask (LOG_UNIMP ,"Corrupt internal state for firmware\n" ); +return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { + goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_CMD , command_reg); +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_STS , status_reg); + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ; + +for (int i =0 ; i <256 ; i++) { +for (int j =0 ; j <256 ; j++) { +if (cxl_cmd_set[i][j].handler) { +struct cxl_cmd *c = &cxl_cmd_set[i][j]; + + cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i <<8 ) | j; + cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,5 +14,8 @@#include "cxl_component.h" #include "cxl_device.h" +#define COMPONENT_REG_BAR_IDX 0 +#define DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +24
-3@@ -70,9 +70,10 @@ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) typedef struct cxl_device_state { - /* Boss container and caps registers */ + /* Main register container */ MemoryRegion device_registers; + MemoryRegion device; struct { MemoryRegion caps; union {@@ -80,8 +81,19 @@ typedef struct cxl_device_state {uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >>2 ]; }; }; -MemoryRegion device; -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >>2 ]; + }; +struct { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };MemoryRegion *pmem;MemoryRegion *vmem;@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )
Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 13 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:
This is the beginning of implementing mailbox support for CXL 2.0 devices.
v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)
Quoted 6 lines
Quoted 1 line
Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.
Cool!
Quoted 5 lines
The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()
https://gitlab.com/bwidawsk/linux/-/tree/cxl-2.0v3
Vishal in the Cc is finishing up some work on adding support to ndctl, after which I'll be sending that (or something closely related) to the mailing list.
Quoted 4 lines
If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.
Could you add the backtrace here? You caused me to go look at what x86 does. I actually assumed on modern platforms it did movsb (byte moves), which are highly optimized in microcode. In fact, it does some amount of word moves too, which was new info to me.
So I'm not really sure yet why ARM64 is special here.
Quoted 5 lines
I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)
8.2
• A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes quantity.
• A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 Bytes quantity
Interesting though, I really thought QEMU should do the right thing here and you wouldn't need to allow 1byte accesses.
Quoted 5 lines
Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).
I think the right solution is to support/allow byte and word access. I just thought it should work in the existing code. I'd like to try to figure out what's going on.
How much effort would it be for me to reproduce what you've done?
Quoted 435 lines
Quoted 102 lines
--- hw/cxl/cxl-device-utils.c | 122 ++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 173 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 27 +++++- 5 files changed, 322 insertions(+), 4 deletions(-) create mode 100644 hw/cxl/cxl-mailbox-utils.chw/cxl/cxl-device-utils.c +84
-0@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (&retval, size); } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->mbox_reg_state + offset, size); +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + +stl_le_p ((uint8_t *)reg_state + offset, value); +} + +static void mailbox_mem_writeq (uint8_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +return ; + } + +stq_le_p (reg_state + offset, value); +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writeq (cxl_dstate->mbox_reg_state, offset, value); +break ; + } + + if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+ DOORBELL)) +cxl_process_mailbox (cxl_dstate ); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 ,Hack was to set this to 1 (and the impl below - no idea if both are needed).
Quoted 319 lines
hw/cxl/cxl-device-utils.c +27
-1+ .max_access_size = 8, + }, + .impl = { + .min_access_size = 4, + .max_access_size = 8, + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write = NULL,@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +173
-0@@ -0,0 +1,173 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; + +/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { +qemu_log_mask (LOG_UNIMP ,"Corrupt internal state for firmware\n" ); +return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) { + goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ; + ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_CMD , command_reg); +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_STS , status_reg); + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ; + +for (int i =0 ; i <256 ; i++) { +for (int j =0 ; j <256 ; j++) { +if (cxl_cmd_set[i][j].handler) { +struct cxl_cmd *c = &cxl_cmd_set[i][j]; + + cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i <<8 ) | j; + cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,5 +14,8 @@#include "cxl_component.h" #include "cxl_device.h" +#define COMPONENT_REG_BAR_IDX 0 +#define DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +24
-3@@ -70,9 +70,10 @@ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) typedef struct cxl_device_state { - /* Boss container and caps registers */ + /* Main register container */ MemoryRegion device_registers; + MemoryRegion device; struct { MemoryRegion caps; union {@@ -80,8 +81,19 @@ typedef struct cxl_device_state {uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >>2 ]; }; }; -MemoryRegion device; -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >>2 ]; + }; +struct { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };MemoryRegion *pmem;MemoryRegion *vmem;@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 ) +/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )
Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
In reply to Jonathan CameronQuoted 34 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:
This is the beginning of implementing mailbox support for CXL 2.0 devices.
v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.
The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.
I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)
Quoted 4 lines
Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).
Bit more testing an another little thing below.
J
Quoted 57 lines
Quoted 57 lines
--- hw/cxl/cxl-device-utils.c | 122 ++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 173 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 27 +++++- 5 files changed, 322 insertions(+), 4 deletions(-) create mode 100644 hw/cxl/cxl-mailbox-utils.chw/cxl/cxl-device-utils.c +39
-0@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (&retval, size); } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->mbox_reg_state + offset, size); +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + +stl_le_p ((uint8_t *)reg_state + offset, value); +} + +static void mailbox_mem_writeq (uint8_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */
Quoted 8 lines
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 93
Quoted 8 lines
hw/cxl/cxl-device-utils.c +8
-0+/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset);
I've been debugging mail box issues and it seems we can hit this path if a payload is written by the OS. Result is it never gets written into the memory and hence isn't available when we try to read it below.
Quoted 233 lines
Quoted 37 lines
hw/cxl/cxl-device-utils.c +37
-0+return ; + } + +stq_le_p (reg_state + offset, value); +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writeq (cxl_dstate->mbox_reg_state, offset, value); +break ; + } + + if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+ DOORBELL)) +cxl_process_mailbox (cxl_dstate ); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 ,Hack was to set this to 1 (and the impl below - no idea if both are needed).
Quoted 183 lines
hw/cxl/cxl-device-utils.c +27
-1+ .max_access_size = 8, + }, + .impl = { + .min_access_size = 4, + .max_access_size = 8, + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write = NULL,@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +121
-0@@ -0,0 +1,173 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; + +/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { +qemu_log_mask (LOG_UNIMP ,"Corrupt internal state for firmware\n" ); +return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) {
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c line 129
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c new file +8
-0+ goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ;
The return highlighted above on which registers are writable meant the payload never has anything useful in it. Remove that and this seems to work and I can query implemented commands + enable them.
Quoted 131 lines
Quoted 118 lines
hw/cxl/cxl-mailbox-utils.c new file +44
-0+ ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_CMD , command_reg); +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_STS , status_reg); + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ; + +for (int i =0 ; i <256 ; i++) { +for (int j =0 ; j <256 ; j++) { +if (cxl_cmd_set[i][j].handler) { +struct cxl_cmd *c = &cxl_cmd_set[i][j]; + + cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i <<8 ) | j; + cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,5 +14,8 @@#include "cxl_component.h" #include "cxl_device.h" +#define COMPONENT_REG_BAR_IDX 0 +#define DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +18
-3@@ -70,9 +70,10 @@ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) typedef struct cxl_device_state { - /* Boss container and caps registers */ + /* Main register container */ MemoryRegion device_registers; + MemoryRegion device; struct { MemoryRegion caps; union {@@ -80,8 +81,19 @@ typedef struct cxl_device_state {uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >>2 ]; }; }; -MemoryRegion device; -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >>2 ]; + }; +struct { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };MemoryRegion *pmem;MemoryRegion *vmem;@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 )include/hw/cxl/cxl_device.h REG32(CXL_DEV_MAILBOX_CTRL, 4) · line 175
Quoted 10 lines
include/hw/cxl/cxl_device.h +6
-0+/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )_______________________________________________
Linuxarm mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 107 lines · Jonathan Cameron
Quoted 34 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:
This is the beginning of implementing mailbox support for CXL 2.0 devices.
v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)Signed-off-by Ben Widawsky <[email protected]>Hi Ben,
I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.
The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.
I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)
Quoted 4 lines
Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).Bit more testing an another little thing below.
J
Quoted 61 lines
Quoted 61 lines
--- hw/cxl/cxl-device-utils.c | 122 ++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 173 ++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl.h | 3 + include/hw/cxl/cxl_device.h | 27 +++++- 5 files changed, 322 insertions(+), 4 deletions(-) create mode 100644 hw/cxl/cxl-mailbox-utils.chw/cxl/cxl-device-utils.c +43
-0@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)return ldn_le_p (&retval, size); } +static uint64_t mailbox_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +if (cxl_device_check_register_alignment (offset, size)) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } + +return ldn_le_p (cxl_dstate->mbox_reg_state + offset, size); +} + +static void mailbox_mem_writel (uint32_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CTRL : +/* fallthrough */ +case A_CXL_DEV_MAILBOX_CAP : +/* RO register */ +break ; +default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 32-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset); +break ; + } + +stl_le_p ((uint8_t *)reg_state + offset, value); +} + +static void mailbox_mem_writeq (uint8_t *reg_state,hwaddr offset, +uint64_t value) +{ +switch (offset) { +case A_CXL_DEV_MAILBOX_CMD : +break ; +case A_CXL_DEV_BG_CMD_STS : +/* BG not supported */ +/* fallthrough */ +case A_CXL_DEV_MAILBOX_STS : +/* Read only register, will get updated by the state machine */ +return ;
Quoted 9 lines
Quoted 4 lines
hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 93
Quoted 4 lines
hw/cxl/cxl-device-utils.c +4
-0+default : +qemu_log_mask (LOG_UNIMP , +"%s Unexpected 64-bit access to 0x%" PRIx64" (WI)\n" , + __func__, offset);I've been debugging mail box issues and it seems we can hit this path if a payload is written by the OS. Result is it never gets written into the memory and hence isn't available when we try to read it below.
I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.
I think the fix should be something like this, but I can't easily test at the moment:
hw/cxl/cxl-device-utils.c +3 -0
@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value, {CXLDeviceState *cxl_dstate = opaque; +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) +stn_le_p (cxl_dstate->mbox_reg_state, size, value); +/* * Lock is needed to prevent concurrent writes as well as to prevent writes * coming in while the firmware is processing. Without background commands
Quoted 378 lines
Quoted 233 lines
Quoted 37 lines
hw/cxl/cxl-device-utils.c +37
-0+return ; + } + +stq_le_p (reg_state + offset, value); +} + +static void mailbox_reg_write (void *opaque,hwaddr offset,uint64_t value, +unsigned size) +{ +CXLDeviceState *cxl_dstate = opaque; + +/* +* Lock is needed to prevent concurrent writes as well as to prevent writes +* coming in while the firmware is processing. Without background commands +* or the second mailbox implemented, this serves no purpose since the +* memory access is synchronized at a higher level (per memory region). +*/ +RCU_READ_LOCK_GUARD (); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writel (cxl_dstate->mbox_reg_state32, offset, value); +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return ; + } +mailbox_mem_writeq (cxl_dstate->mbox_reg_state, offset, value); +break ; + } + + if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140
Quoted 10 lines
hw/cxl/cxl-device-utils.c +10
-0+ DOORBELL)) +cxl_process_mailbox (cxl_dstate ); +} + +static const MemoryRegionOps mailbox_ops = { + .read = mailbox_reg_read, + .write = mailbox_reg_write, + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 ,Hack was to set this to 1 (and the impl below - no idea if both are needed).
Quoted 183 lines
hw/cxl/cxl-device-utils.c +27
-1+ .max_access_size = 8, + }, + .impl = { + .min_access_size = 4, + .max_access_size = 8, + }, +}; +static const MemoryRegionOps dev_ops = { .read = dev_reg_read, .write = NULL,@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);memory_region_init_io (&cxl_dstate->device, obj, &dev_ops, cxl_dstate,"device-status" ,CXL_DEVICE_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate, +"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);memory_region_add_subregion (&cxl_dstate->device_registers,CXL_DEVICE_REGISTERS_OFFSET , &cxl_dstate->device); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MAILBOX_REGISTERS_OFFSET , + &cxl_dstate->mailbox); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { } +static void mailbox_reg_init_common (CXLDeviceState *cxl_dstate) +{ +/* 2048 payload size, with no interrupt or background support */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CAP , +PAYLOAD_SIZE ,CXL_MAILBOX_PAYLOAD_SHIFT ); + cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; +} +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =1 ; +const int cap_count =2 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,DEVICE ,1 );device_reg_init_common (cxl_dstate); + +cxl_device_cap_init (cxl_dstate,MAILBOX ,2 ); +mailbox_reg_init_common (cxl_dstate); + +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }hw/cxl/cxl-mailbox-utils.c new file +121
-0@@ -0,0 +1,173 @@ +/* +* CXL Utility library for mailbox interface +* +* Copyright(C) 2020 Intel Corporation. +* +* This work is licensed under the terms of the GNU GPL, version 2. See the +* COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci.h" +#include "qemu/log.h" +#include "qemu/uuid.h" + +/* +* How to add a new command, example. The command set FOO, with cmd BAR. +* 1. Add the command set and cmd to the enum. +* FOO = 0x7f, +* #define BAR 0 +* 2. Forward declare the handler. +* declare_mailbox_handler(FOO_BAR); +* 3. Add the command to the cxl_cmd_set[][] +* CXL_CMD(FOO, BAR, 0, 0), +* 4. Implement your handler +* define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; } +* +* +* Writing the handler: +* The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the +* in/out length of the payload. The handler is responsible for consuming the +* payload from cmd->payload and operating upon it as necessary. It must then +* fill the output data into cmd->payload (overwriting what was there), +* setting the length, and returning a valid return code. +* +* XXX: The handler need not worry about endianess. The payload is read out of +* a register interface that already deals with it. +*/ + +/* 8.2.8.4.5.1 Command Return Codes */ +typedef enum { +CXL_MBOX_SUCCESS =0x0 , +CXL_MBOX_BG_STARTED =0x1 , +CXL_MBOX_INVALID_INPUT =0x2 , +CXL_MBOX_UNSUPPORTED =0x3 , +CXL_MBOX_INTERNAL_ERROR =0x4 , +CXL_MBOX_RETRY_REQUIRED =0x5 , +CXL_MBOX_BUSY =0x6 , +CXL_MBOX_MEDIA_DISABLED =0x7 , +CXL_MBOX_FW_XFER_IN_PROGRESS =0x8 , +CXL_MBOX_FW_XFER_OUT_OF_ORDER =0x9 , +CXL_MBOX_FW_AUTH_FAILED =0xa , +CXL_MBOX_FW_INVALID_SLOT =0xb , +CXL_MBOX_FW_ROLLEDBACK =0xc , +CXL_MBOX_FW_REST_REQD =0xd , +CXL_MBOX_INVALID_HANDLE =0xe , +CXL_MBOX_INVALID_PA =0xf , +CXL_MBOX_INJECT_POISON_LIMIT =0x10 , +CXL_MBOX_PERMANENT_MEDIA_FAILURE =0x11 , +CXL_MBOX_ABORTED =0x12 , +CXL_MBOX_INVALID_SECURITY_STATE =0x13 , +CXL_MBOX_INCORRECT_PASSPHRASE =0x14 , +CXL_MBOX_UNSUPPORTED_MAILBOX =0x15 , +CXL_MBOX_INVALID_PAYLOAD_LENGTH =0x16 , +CXL_MBOX_MAX =0x17 +}ret_code ; + +struct cxl_cmd ; +typedef ret_code (*opcode_handler )(struct cxl_cmd *cmd, +CXLDeviceState *cxl_dstate,uint16_t *len); +struct cxl_cmd { +const char *name; +opcode_handler handler; +ssize_t in; +uint16_t effect;/* Reported in CEL */ +uint8_t *payload; +}; + +#define define_mailbox_handler (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) +#define declare_mailbox_handler (name) define_mailbox_handler(name) + +#define CXL_CMD (s, c, in, cel_effect) \ + [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } + +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; + +#undef CXL_CMD + +QemuUUID cel_uuid; + +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) +{ +uint16_t ret =CXL_MBOX_SUCCESS ; +struct cxl_cmd *cxl_cmd; +uint64_t status_reg; +opcode_handler h; + +/* +* current state of mailbox interface +* mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP]; +* mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL]; +* status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS]; +*/ +uint64_t command_reg = + *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD ]; + +/* Check if we have to do anything */ +if (!ARRAY_FIELD_EX32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL )) { +qemu_log_mask (LOG_UNIMP ,"Corrupt internal state for firmware\n" ); +return ; + } + +uint8_t set =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ); +uint8_t cmd =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ); +uint16_t len =FIELD_EX64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH ); + cxl_cmd = &cxl_cmd_set[set][cmd]; + h = cxl_cmd->handler; +if (!h) {Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c line 129
Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c new file +8
-0+ goto handled; + } + +if (len != cxl_cmd->in) { + ret =CXL_MBOX_INVALID_PAYLOAD_LENGTH ; + } + + cxl_cmd->payload = cxl_dstate->mbox_reg_state +A_CXL_DEV_CMD_PAYLOAD ;The return highlighted above on which registers are writable meant the payload never has anything useful in it. Remove that and this seems to work and I can query implemented commands + enable them.
Quoted 131 lines
Quoted 118 lines
hw/cxl/cxl-mailbox-utils.c new file +44
-0+ ret = (*h)(cxl_cmd, cxl_dstate, &len); +assert (len <= cxl_dstate->payload_size); + +handled: +/* +* Set the return code +* XXX: it's a 64b register, but we're not setting the vendor, so we can get +* away with this +*/ + status_reg =FIELD_DP64 (0 ,CXL_DEV_MAILBOX_STS ,ERRNO , ret); + +/* +* Set the return length +*/ + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND_SET ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,COMMAND ,0 ); + command_reg =FIELD_DP64 (command_reg,CXL_DEV_MAILBOX_CMD ,LENGTH , len); + +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_CMD , command_reg); +stq_le_p (cxl_dstate->mbox_reg_state +A_CXL_DEV_MAILBOX_STS , status_reg); + +/* Tell the host we're done */ +ARRAY_FIELD_DP32 (cxl_dstate->mbox_reg_state32,CXL_DEV_MAILBOX_CTRL , +DOORBELL ,0 ); +} + +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate) +{ +const char *cel_uuidstr ="0da9c0b5-bf41-4b78-8f79-96b1623b3f17" ; + +for (int i =0 ; i <256 ; i++) { +for (int j =0 ; j <256 ; j++) { +if (cxl_cmd_set[i][j].handler) { +struct cxl_cmd *c = &cxl_cmd_set[i][j]; + + cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i <<8 ) | j; + cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect; + cxl_dstate->cel_size++; + } + } + } + +return qemu_uuid_parse (cel_uuidstr, &cel_uuid); +}hw/cxl/meson.build +1
-0@@ -1,4 +1,5 @@ softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( 'cxl-component-utils.c', 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', ))include/hw/cxl/cxl.h +3
-0@@ -14,5 +14,8 @@#include "cxl_component.h" #include "cxl_device.h" +#define COMPONENT_REG_BAR_IDX 0 +#define DEVICE_REG_BAR_IDX 2 +#endif include/hw/cxl/cxl_device.h +18
-3@@ -70,9 +70,10 @@ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) typedef struct cxl_device_state { - /* Boss container and caps registers */ + /* Main register container */ MemoryRegion device_registers; + MemoryRegion device; struct { MemoryRegion caps; union {@@ -80,8 +81,19 @@ typedef struct cxl_device_state {uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >>2 ]; }; }; -MemoryRegion device; -MemoryRegion mailbox; +struct { +MemoryRegion mailbox; +uint16_t payload_size; +union { +uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH ]; +uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >>2 ]; + }; +struct { +uint16_t opcode; +uint16_t effect; + } cel_log[1 <<16 ]; +size_t cel_size; + };MemoryRegion *pmem;MemoryRegion *vmem;@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate); +void cxl_process_mailbox (CXLDeviceState *cxl_dstate); +#define cxl_device_cap_init (dstate, reg, cap_id) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4) FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)FIELD (CXL_DEV_MAILBOX_CTRL ,BG_INT_EN ,2 ,1 )include/hw/cxl/cxl_device.h REG32(CXL_DEV_MAILBOX_CTRL, 4) · line 175
Quoted 10 lines
include/hw/cxl/cxl_device.h +6
-0+/* XXX: actually a 64b register */ +REG32(CXL_DEV_MAILBOX_CMD, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8) + FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8) +FIELD (CXL_DEV_MAILBOX_CMD ,LENGTH ,16 ,20 ) + /* XXX: actually a 64b register */ REG32(CXL_DEV_MAILBOX_STS, 0x10)FIELD (CXL_DEV_MAILBOX_STS ,BG_OP ,0 ,1 )_______________________________________________
Linuxarm mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 7 lines · Ben Widawsky
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
On Tue, 5 Jan 2021 08:52:58 -0800
Ben Widawsky <[email protected]> wrote:
[snip]
Quoted 15 lines
I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.
I think the fix should be something like this, but I can't easily test at the moment:
hw/cxl/cxl-device-utils.c +0
-0
hw/cxl/cxl-device-utils.c static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value, · line 110
Quoted 9 lines
hw/cxl/cxl-device-utils.c +3
-0@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value, {CXLDeviceState *cxl_dstate = opaque; +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) +stn_le_p (cxl_dstate->mbox_reg_state, size, value); +/* * Lock is needed to prevent concurrent writes as well as to prevent writes * coming in while the firmware is processing. Without background commands
+ if (offset >= A_CXL_DEV_CMD_PAYLOAD) {
+ stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+ return;
+ }
+
[snip]
Quoted 1 line
Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)
Quoted 36 lines · Ben Widawsky
Quoted 7 lines · Ben Widawsky
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
On Tue, 5 Jan 2021 08:52:58 -0800
Ben Widawsky <[email protected]> wrote:[snip]
Quoted 24 lines
I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.
I think the fix should be something like this, but I can't easily test at the moment:
hw/cxl/cxl-device-utils.c +3
-0@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value, {CXLDeviceState *cxl_dstate = opaque; +if (offset >=A_CXL_DEV_CMD_PAYLOAD ) +stn_le_p (cxl_dstate->mbox_reg_state, size, value); +/* * Lock is needed to prevent concurrent writes as well as to prevent writes * coming in while the firmware is processing. Without background commands
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (offset >= A_CXL_DEV_CMD_PAYLOAD) { + stn_le_p(cxl_dstate->mbox_reg_state, size, value); + return; + } +
Last time's a charm
stn_le_p(cxl_dstate->mbox_reg_state + offset, size, value);
Quoted 4 lines
[snip]
Quoted 1 line
RFC PATCH v2 8/32 hw/cxl/device: Add memory devices (8.2.8.5)
In reply to Ben WidawskyMemory devices implement extra capabilities on top of CXL devices. This adds support for that.
A large part of memory devices is the mailbox/command interface. All of the mailbox handling is done in the mailbox-utils library. Longer term, new CXL devices that are being emulated may want to handle commands differently, and therefore would need a mechanism to opt in/out of the specific generic handlers. As such, this is considered sufficient for now, but may need more depth in the future.
--- hw/cxl/cxl-device-utils.c | 52 ++++++++++++++++++++++++++++++++++++- hw/cxl/cxl-mailbox-utils.c | 24 +++++++++++++++++ include/hw/cxl/cxl_device.h | 16 ++++++++++++ 3 files changed, 91 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c +51 -1
@@ -132,6 +132,45 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,cxl_process_mailbox (cxl_dstate); } +static uint64_t mdev_reg_read (void *opaque,hwaddr offset,unsigned size) +{ +uint64_t retval =0 ; + + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MEDIA_STATUS ,1 ); + retval =FIELD_DP64 (retval,CXL_MEM_DEV_STS ,MBOX_READY ,1 ); + +switch (size) { +case 4 : +if (unlikely (offset & (sizeof (uint32_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } +break ; +case 8 : +if (unlikely (offset & (sizeof (uint64_t ) -1 ))) { +qemu_log_mask (LOG_UNIMP ,"Unaligned register read\n" ); +return 0 ; + } +break ; + } + +return ldn_le_p (&retval, size); +} + +static const MemoryRegionOps mdev_ops = { + .read = mdev_reg_read, + .write =NULL , + .endianness =DEVICE_LITTLE_ENDIAN , + .valid = { + .min_access_size =4 , + .max_access_size =8 , + }, + .impl = { + .min_access_size =4 , + .max_access_size =8 , + }, +}; +static const MemoryRegionOps mailbox_ops = { .read = mailbox_reg_read, .write = mailbox_reg_write,
@@ -187,6 +226,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate) "device-status", CXL_DEVICE_REGISTERS_LENGTH);memory_region_init_io (&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,"mailbox" ,CXL_MAILBOX_REGISTERS_LENGTH ); +memory_region_init_io (&cxl_dstate->memory_device, obj, &mdev_ops, + cxl_dstate,"memory device caps" , +CXL_MEMORY_DEVICE_REGISTERS_LENGTH );memory_region_add_subregion (&cxl_dstate->device_registers,0 , &cxl_dstate->caps);
@@ -196,6 +238,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)memory_region_add_subregion (&cxl_dstate->device_registers,CXL_MAILBOX_REGISTERS_OFFSET , &cxl_dstate->mailbox); +memory_region_add_subregion (&cxl_dstate->device_registers, +CXL_MEMORY_DEVICE_REGISTERS_OFFSET , + &cxl_dstate->memory_device); }static void device_reg_init_common (CXLDeviceState *cxl_dstate) { }
@@ -208,10 +253,12 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) cxl_dstate->payload_size =CXL_MAILBOX_MAX_PAYLOAD_SIZE ; } +static void memdev_reg_init_common (CXLDeviceState *cxl_dstate) { } +void cxl_device_register_init_common (CXLDeviceState *cxl_dstate) {uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32; -const int cap_count =2 ; +const int cap_count =3 ;/* CXL Device Capabilities Array Register */ ARRAY_FIELD_DP32 (cap_hdrs,CXL_DEV_CAP_ARRAY ,CAP_ID ,0 );
@@ -224,5 +271,8 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)cxl_device_cap_init (cxl_dstate,MAILBOX ,2 );mailbox_reg_init_common (cxl_dstate); +cxl_device_cap_init (cxl_dstate,MEMORY_DEVICE ,0x4000 ); +memdev_reg_init_common (cxl_dstate); +assert (cxl_initialize_mailbox (cxl_dstate) ==0 ); }
hw/cxl/cxl-mailbox-utils.c +24 -0
@@ -81,6 +81,30 @@ struct cxl_cmd { CXLDeviceState *cxl_dstate, uint16_t *len)#define declare_mailbox_handler (name) define_mailbox_handler(name) +#define define_mailbox_handler_zeroed (name, size) \ + uint16_t __zero##name = size; \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = __zero##name; \ + memset(cmd->payload, 0, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_const (name, data) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + *len = sizeof(data); \ + memcpy(cmd->payload, data, *len); \ + return CXL_MBOX_SUCCESS; \ + } +#define define_mailbox_handler_nop (name) \ + static ret_code cmd_##name(struct cxl_cmd *cmd, \ + CXLDeviceState *cxl_dstate, uint16_t *len) \ + { \ + return CXL_MBOX_SUCCESS; \ + } +#define CXL_CMD (s, c, in, cel_effect) \ [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
include/hw/cxl/cxl_device.h +16 -0
@@ -69,11 +69,16 @@#define CXL_MAILBOX_REGISTERS_LENGTH \ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE) +#define CXL_MEMORY_DEVICE_REGISTERS_OFFSET \ + (CXL_MAILBOX_REGISTERS_OFFSET + CXL_MAILBOX_REGISTERS_LENGTH) +#define CXL_MEMORY_DEVICE_REGISTERS_LENGTH 0x8 + typedef struct cxl_device_state { /* Main register container */ MemoryRegion device_registers; MemoryRegion device; + MemoryRegion memory_device; struct { MemoryRegion caps; union {
@@ -133,6 +138,9 @@ REG32(CXL_DEV_CAP_ARRAY2, 4) /* We're going to pretend it's 64b */ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MAILBOX ,CXL_DEVICE_CAP_HDR1_OFFSET + \ CXL_DEVICE_CAP_REG_SIZE) +CXL_DEVICE_CAPABILITY_HEADER_REGISTER (MEMORY_DEVICE , +CXL_DEVICE_CAP_HDR1_OFFSET + + CXL_DEVICE_CAP_REG_SIZE * 2)int cxl_initialize_mailbox (CXLDeviceState *cxl_dstate);void cxl_process_mailbox (CXLDeviceState *cxl_dstate);
@@ -185,4 +193,12 @@ REG32(CXL_DEV_BG_CMD_STS, 0x18)REG32 (CXL_DEV_CMD_PAYLOAD ,0x20 ) +/* XXX: actually a 64b registers */ +REG32(CXL_MEM_DEV_STS, 0) + FIELD(CXL_MEM_DEV_STS, FATAL, 0, 1) + FIELD(CXL_MEM_DEV_STS, FW_HALT, 1, 1) + FIELD(CXL_MEM_DEV_STS, MEDIA_STATUS, 2, 2) + FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) + FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +#endif
Signature · 1 line
2.30.0
RFC PATCH v2 9/32 hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
In reply to Ben WidawskyUsing the previously implemented stubbed helpers, it is now possible to easily add the missing, required commands to the implementation.
--- hw/cxl/cxl-mailbox-utils.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +22 -1
@@ -37,6 +37,14 @@* a register interface that already deals with it. */ +enum { + EVENTS = 0x01, + #define GET_RECORDS 0x0 +#define CLEAR_RECORDS 0x1 +#define GET_INTERRUPT_POLICY 0x2 +#define SET_INTERRUPT_POLICY 0x3 +}; +/* 8.2.8.4.5.1 Command Return Codes */ typedef enum { CXL_MBOX_SUCCESS = 0x0,
@@ -105,10 +113,23 @@ struct cxl_cmd {return CXL_MBOX_SUCCESS ; \ } +define_mailbox_handler_zeroed (EVENTS_GET_RECORDS ,0x20 ); +define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS ); +define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 ); +define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); + +#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_LOG_CHANGE (1 << 4) +#define CXL_CMD (s, c, in, cel_effect) \ [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect } -static struct cxl_cmd cxl_cmd_set[256 ][256 ] = {}; +static struct cxl_cmd cxl_cmd_set[256 ][256 ] = { +CXL_CMD (EVENTS ,GET_RECORDS ,1 ,0 ), +CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ), +CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ), +CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +};#undef CXL_CMD
Signature · 1 line
2.30.0
RFC PATCH v2 10/32 hw/cxl/device: Placeholder for firmware commands
In reply to Ben WidawskyRFC PATCH v2 11/32 hw/cxl/device: Timestamp implementation (8.2.9.3)
In reply to Ben WidawskyPer spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.
--- hw/cxl/cxl-mailbox-utils.c | 52 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 58 insertions(+)
hw/cxl/cxl-mailbox-utils.c +52 -0
@@ -43,6 +43,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (TIMESTAMP_GET ); +declare_mailbox_handler (TIMESTAMP_SET );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)#define CXL_CMD (s, c, in, cel_effect) \
@@ -129,10 +135,56 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (TIMESTAMP ,GET ,0 ,0 ), + CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE), };#undef CXL_CMD +/* +* 8.2.9.3.1 +*/ +define_mailbox_handler (TIMESTAMP_GET ) +{ +struct timespec ts; + +clock_gettime (CLOCK_REALTIME , &ts); + + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set = + ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec; + cxl_dstate->timestamp.host_set = *(uint64_t *)cmd->payload; + + *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* +* 8.2.9.3.2 +*/ +define_mailbox_handler (TIMESTAMP_SET ) +{ +struct timespec ts; +uint64_t delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done; + } + +/* First find the delta from the last time the host set the time. */ +clock_gettime (CLOCK_REALTIME , &ts); + delta = (ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec) - + cxl_dstate->timestamp.host_set; + +/* Then adjust the actual time */ + *(uint64_t *)cmd->payload = cxl_dstate->timestamp.host_set + delta; + +done: + *len =8 ; +return CXL_MBOX_SUCCESS ; +} +QemuUUID cel_uuid;void cxl_process_mailbox (CXLDeviceState *cxl_dstate)
include/hw/cxl/cxl_device.h +6 -0
@@ -100,6 +100,12 @@ typedef struct cxl_device_state {size_t cel_size; }; +struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +MemoryRegion *pmem;MemoryRegion *vmem; } CXLDeviceState;
Signature · 1 line
2.30.0
Re: [RFC PATCH v2 11/32] hw/cxl/device: Timestamp implementation (8.2.9.3)
Quoted 82 lines · Ben Widawsky
Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.
Signed-off-by Ben Widawsky <[email protected]>--- hw/cxl/cxl-mailbox-utils.c | 52 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 6 +++++ 2 files changed, 58 insertions(+)hw/cxl/cxl-mailbox-utils.c +42
-0@@ -43,6 +43,9 @@ enum {#define CLEAR_RECORDS 0x1#define GET_INTERRUPT_POLICY 0x2#define SET_INTERRUPT_POLICY 0x3 + TIMESTAMP = 0x03, +#define GET 0x0 +#define SET 0x1 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);define_mailbox_handler_nop (EVENTS_CLEAR_RECORDS );define_mailbox_handler_zeroed (EVENTS_GET_INTERRUPT_POLICY ,4 );define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY ); +declare_mailbox_handler (TIMESTAMP_GET ); +declare_mailbox_handler (TIMESTAMP_SET );#define IMMEDIATE_CONFIG_CHANGE (1 << 1) +#define IMMEDIATE_POLICY_CHANGE (1 << 3)#define IMMEDIATE_LOG_CHANGE (1 << 4)#define CXL_CMD (s, c, in, cel_effect) \@@ -129,10 +135,56 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,CLEAR_RECORDS , ~0 ,IMMEDIATE_LOG_CHANGE ),CXL_CMD (EVENTS ,GET_INTERRUPT_POLICY ,0 ,0 ),CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ), +CXL_CMD (TIMESTAMP ,GET ,0 ,0 ), + CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE), };#undef CXL_CMD +/* +* 8.2.9.3.1 +*/ +define_mailbox_handler (TIMESTAMP_GET ) +{ +struct timespec ts; + +clock_gettime (CLOCK_REALTIME , &ts); + + cxl_dstate->timestamp.set = true; + cxl_dstate->timestamp.last_set = + ts.tv_sec *NANOSECONDS_PER_SECOND + ts.tv_nsec; + cxl_dstate->timestamp.host_set = *(uint64_t *)cmd->payload; + + *len =8 ; +return CXL_MBOX_SUCCESS ; +} + +/* +* 8.2.9.3.2 +*/ +define_mailbox_handler (TIMESTAMP_SET ) +{ +struct timespec ts; +uint64_t delta; + +if (!cxl_dstate->timestamp.set) { + *(uint64_t *)cmd->payload =0 ; + goto done; + } + +/* First find the delta from the last time the host set the time. */ +clock_gettime (CLOCK_REALTIME , &ts); + delta = (ts.tv_sec * NANOSECONDS_PER_SECOND + ts.tv_nsec) -
hw/cxl/cxl-mailbox-utils.c static struct cxl_cmd cxl_cmd_set[256][256] = { · line 187
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+ cxl_dstate->timestamp.host_set; + +/* Then adjust the actual time */ + *(uint64_t *)cmd->payload = cxl_dstate->timestamp.host_set + delta; + +done: + *len =8 ; +return CXL_MBOX_SUCCESS ; +} +
Set/get are reversed here. I missed the fixup before sending.
Quoted 23 lines
hw/cxl/cxl-mailbox-utils.c +0
-0QemuUUID cel_uuid;void cxl_process_mailbox (CXLDeviceState *cxl_dstate)include/hw/cxl/cxl_device.h +6
-0@@ -100,6 +100,12 @@ typedef struct cxl_device_state {size_t cel_size; }; +struct { +bool set; +uint64_t last_set; +uint64_t host_set; + } timestamp; +MemoryRegion *pmem;MemoryRegion *vmem; } CXLDeviceState;Signature · 1 line
2.30.0
RFC PATCH v2 12/32 hw/cxl/device: Add log commands (8.2.9.4) + CEL
In reply to Ben WidawskyCXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.
--- hw/cxl/cxl-mailbox-utils.c | 75 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
hw/cxl/cxl-mailbox-utils.c +75 -0
@@ -46,6 +46,9 @@ enum { TIMESTAMP = 0x03,#define GET 0x0#define SET 0x1 + LOGS = 0x04, +#define GET_SUPPORTED 0x0 +#define GET_LOG 0x1 };/* 8.2.8.4.5.1 Command Return Codes */
@@ -122,6 +125,8 @@ define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);define_mailbox_handler_nop (EVENTS_SET_INTERRUPT_POLICY );declare_mailbox_handler (TIMESTAMP_GET );declare_mailbox_handler (TIMESTAMP_SET ); +declare_mailbox_handler (LOGS_GET_SUPPORTED ); +declare_mailbox_handler (LOGS_GET_LOG );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -137,6 +142,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (EVENTS ,SET_INTERRUPT_POLICY ,4 ,IMMEDIATE_CONFIG_CHANGE ),CXL_CMD (TIMESTAMP ,GET ,0 ,0 ),CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ), +CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ), + CXL_CMD(LOGS, GET_LOG, 0x18, 0), };#undef CXL_CMD
@@ -187,6 +194,74 @@ done:QemuUUID cel_uuid; +/* 8.2.9.4.1 */ +define_mailbox_handler (LOGS_GET_SUPPORTED ) +{ +struct { +uint16_t entries; +uint8_t rsvd[6 ]; +struct { +QemuUUID uuid; +uint32_t size; + } log_entries[1 ]; + } __attribute__((packed)) *supported_logs; +_Static_assert (sizeof (*supported_logs) ==0x1c ,"Bad supported log size" ); + + supported_logs = (void *)cmd->payload; + + supported_logs->entries =1 ; + supported_logs->log_entries[0 ].uuid = cel_uuid; + supported_logs->log_entries[0 ].size =4 * cxl_dstate->cel_size; + + *len =sizeof (*supported_logs); +return CXL_MBOX_SUCCESS ; +} + +/* 8.2.9.4.2 */ +define_mailbox_handler (LOGS_GET_LOG ) +{ +struct { +union { +struct { +QemuUUID uuid; +uint32_t offset; +uint32_t length; + }; + }; + } __attribute__((packed,__aligned__ (16 ))) *get_log; + + get_log = (void *)cmd->payload; + +/* +* 8.2.9.4.2 +* The device shall return Invalid Parameter if the Offset or Length +* fields attempt to access beyond the size of the log as reported by Get +* Supported Logs. +* +* XXX: Spec is wrong, "Invalid Parameter" isn't a thing. +* XXX: Spec doesn't address incorrect UUID incorrectness. +* +* The CEL buffer is large enough to fit all commands in the emulation, so +* the only possible failure would be if the mailbox itself isn't big +* enough. +*/ +if (get_log->offset + get_log->length > cxl_dstate->payload_size) { +return CXL_MBOX_INVALID_INPUT ; + } + +if (!qemu_uuid_is_equal (&get_log->uuid, &cel_uuid)) { +return CXL_MBOX_UNSUPPORTED ; + } + +/* Store off everything to local variables so we can wipe out the payload */ + *len = get_log->length; + +memmove (cmd->payload, cxl_dstate->cel_log + get_log->offset, + get_log->length); + +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;
Signature · 1 line
2.30.0
RFC PATCH v2 13/32 hw/pxb: Use a type for realizing expanders
In reply to Ben WidawskyThis opens up the possibility for more types of expanders (other than PCI and PCIe). We'll need this to create a CXL expander.
--- hw/pci-bridge/pci_expander_bridge.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +7 -4
@@ -24,6 +24,8 @@#include "hw/boards.h" #include "qom/object.h" +enum BusType {PCI ,PCIE }; +#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ; DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
@@ -214,7 +216,8 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)0 ; } -static void pxb_dev_realize_common (PCIDevice *dev,bool pcie,Error **errp) +static void pxb_dev_realize_common (PCIDevice *dev,enum BusType type, +Error **errp) {PXBDev *pxb =convert_to_pxb (dev);DeviceState *ds, *bds =NULL ;
@@ -239,7 +242,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp) } ds =qdev_new (TYPE_PXB_HOST ); -if (pcie) { +if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS );
@@ -287,7 +290,7 @@ static void pxb_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, false, errp); +pxb_dev_realize_common (dev,PCI , errp); }static void pxb_dev_exitfn (PCIDevice *pci_dev)
@@ -339,7 +342,7 @@ static void pxb_pcie_dev_realize(PCIDevice *dev, Error **errp)return ; } -pxb_dev_realize_common (dev, true, errp); +pxb_dev_realize_common (dev,PCIE , errp); }static void pxb_pcie_dev_class_init (ObjectClass *klass,void *data)
Signature · 1 line
2.30.0
RFC PATCH v2 14/32 hw/pci/cxl: Create a CXL bus type
In reply to Ben WidawskyThe easiest way to differentiate a CXL bus, and a PCIE bus is using a flag. A CXL bus, in hardware, is backward compatible with PCIE, and therefore the code tries pretty hard to keep them in sync as much as possible.
The other way to implement this would be to try to cast the bus to the correct type. This is less code and useful for debugging via simply looking at the flags.
--- hw/pci-bridge/pci_expander_bridge.c | 9 ++++++++- include/hw/pci/pci_bus.h | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +8 -1
@@ -24,7 +24,7 @@#include "hw/boards.h" #include "qom/object.h" -enum BusType {PCI ,PCIE }; +enum BusType {PCI ,PCIE ,CXL };#define TYPE_PXB_BUS "pxb-bus"typedef struct PXBBus PXBBus ;
@@ -35,6 +35,10 @@ DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,DECLARE_INSTANCE_CHECKER (PXBBus,PXB_PCIE_BUS , TYPE_PXB_PCIE_BUS) +#define TYPE_PXB_CXL_BUS "pxb-cxl-bus" +DECLARE_INSTANCE_CHECKER (PXBBus,PXB_CXL_BUS , + TYPE_PXB_CXL_BUS) +struct PXBBus {/*< private >*/ PCIBus parent_obj;
@@ -244,6 +248,9 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, ds =qdev_new (TYPE_PXB_HOST );if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); + }else if (type ==CXL ) { + bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); + bus->flags |=PCI_BUS_CXL ; }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
include/hw/pci/pci_bus.h +7 -0
@@ -24,6 +24,8 @@ enum PCIBusFlags {PCI_BUS_IS_ROOT =0x0001 ,/* PCIe extended configuration space is accessible on this bus */ PCI_BUS_EXTENDED_CONFIG_SPACE =0x0002 , +/* This is a CXL Type BUS */ + PCI_BUS_CXL = 0x0004, }; struct PCIBus {
@@ -53,6 +55,11 @@ struct PCIBus {Notifier machine_done; }; +static inline bool pci_bus_is_cxl (PCIBus *bus) +{ +return !!(bus->flags &PCI_BUS_CXL ); +} +static inline bool pci_bus_is_root (PCIBus *bus) {return !!(bus->flags &PCI_BUS_IS_ROOT );
Signature · 1 line
2.30.0
RFC PATCH v2 15/32 hw/pxb: Allow creation of a CXL PXB (host bridge)
In reply to Ben WidawskyThis works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1
A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in _SB named ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.
Upcoming patches will allow creating multiple host bridges.
v2: Remove vendor and device ID (Ben)
--- hw/pci-bridge/pci_expander_bridge.c | 67 ++++++++++++++++++++++++++++- hw/pci/pci.c | 7 +++ include/hw/pci/pci.h | 6 +++ 3 files changed, 78 insertions(+), 2 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +65 -2
@@ -56,6 +56,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +struct PXBDev {/*< private >*/ PCIDevice parent_obj;
@@ -67,6 +71,11 @@ struct PXBDev {static PXBDev *convert_to_pxb (PCIDevice *dev) { +/* A CXL PXB's parent bus is PCIe, so the normal check won't work */ +if (object_dynamic_cast (OBJECT (dev),TYPE_PXB_CXL_DEVICE )) { +return PXB_CXL_DEV (dev); + } +return pci_bus_is_express (pci_get_bus (dev)) ?PXB_PCIE_DEV (dev) :PXB_DEV (dev); }
@@ -111,11 +120,20 @@ static const TypeInfo pxb_pcie_bus_info = { .class_init = pxb_bus_class_init, }; +static const TypeInfo pxb_cxl_bus_info = { + .name =TYPE_PXB_CXL_BUS , + .parent =TYPE_CXL_BUS , + .instance_size =sizeof (PXBBus), + .class_init = pxb_bus_class_init, +}; +static const char *pxb_host_root_bus_path (PCIHostState *host_bridge,PCIBus *rootbus) { -PXBBus *bus =pci_bus_is_express (rootbus) ? -PXB_PCIE_BUS (rootbus) :PXB_BUS (rootbus); +PXBBus *bus =pci_bus_is_cxl (rootbus) ? +PXB_CXL_BUS (rootbus) : +pci_bus_is_express (rootbus) ?PXB_PCIE_BUS (rootbus) : +PXB_BUS (rootbus);snprintf (bus->bus_path,8 ,"0000:%02x" ,pxb_bus_num (rootbus));return bus->bus_path;
@@ -380,13 +398,58 @@ static const TypeInfo pxb_pcie_dev_info = { }, }; +static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) +{ +/* A CXL PXB's parent bus is still PCIe */ +if (!pci_bus_is_express (pci_get_bus (dev))) { +error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" ); +return ; + } + +pxb_dev_realize_common (dev,CXL , errp); +} + +static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (klass); +PCIDeviceClass *k =PCI_DEVICE_CLASS (klass); + + k->realize = pxb_cxl_dev_realize; + k->exit = pxb_dev_exitfn; +/* +* XXX: These types of bridges don't actually show up in the hierarchy so +* vendor, device, class, etc. ids are intentionally left out. +*/ + + dc->desc ="CXL Host Bridge" ; +device_class_set_props (dc, pxb_dev_properties); +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + +/* Host bridges aren't hotpluggable. FIXME: spec reference */ + dc->hotpluggable = false; +} + +static const TypeInfo pxb_cxl_dev_info = { + .name =TYPE_PXB_CXL_DEVICE , + .parent =TYPE_PCI_DEVICE , + .instance_size =sizeof (PXBDev), + .class_init = pxb_cxl_dev_class_init, + .interfaces = + (InterfaceInfo []){ + {INTERFACE_CONVENTIONAL_PCI_DEVICE }, + {}, + }, +}; +static void pxb_register_types (void ) {type_register_static (&pxb_bus_info);type_register_static (&pxb_pcie_bus_info); +type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info); +type_register_static (&pxb_cxl_dev_info); }type_init (pxb_register_types)
hw/pci/pci.c +7 -0
@@ -222,6 +222,12 @@ static const TypeInfo pcie_bus_info = { .class_init = pcie_bus_class_init, }; +static const TypeInfo cxl_bus_info = { + .name =TYPE_CXL_BUS , + .parent =TYPE_PCIE_BUS , + .class_init = pcie_bus_class_init, +}; +static PCIBus *pci_find_bus_nr (PCIBus *bus,int bus_num);static void pci_update_mappings (PCIDevice *d);static void pci_irq_handler (void *opaque,int irq_num,int level);
@@ -2849,6 +2855,7 @@ static void pci_register_types(void) {type_register_static (&pci_bus_info);type_register_static (&pcie_bus_info); +type_register_static (&cxl_bus_info);type_register_static (&conventional_pci_interface_info);type_register_static (&cxl_interface_info);type_register_static (&pcie_interface_info);
include/hw/pci/pci.h +6 -0
@@ -405,6 +405,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);#define TYPE_PCI_BUS "PCI" OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)#define TYPE_PCIE_BUS "PCIE" +#define TYPE_CXL_BUS "CXL"bool pci_bus_is_express (PCIBus *bus);
@@ -753,6 +754,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)pci_irq_deassert (pci_dev); } +static inline int pci_is_cxl (const PCIDevice *d) +{ +return d->cap_present &QEMU_PCIE_CAP_CXL ; +} +static inline int pci_is_express (const PCIDevice *d) {return d->cap_present &QEMU_PCI_CAP_EXPRESS ;
Signature · 1 line
2.30.0
RFC PATCH v2 16/32 qtest: allow DSDT acpi table changes
In reply to Ben Widawsky--- tests/qtest/bios-tables-test-allowed-diff.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
tests/qtest/bios-tables-test-allowed-diff.h +21 -0
@@ -1 +1,22 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/pc/DSDT" , +"tests/data/acpi/pc/DSDT.acpihmat" , +"tests/data/acpi/pc/DSDT.bridge" , +"tests/data/acpi/pc/DSDT.cphp" , +"tests/data/acpi/pc/DSDT.dimmpxm" , +"tests/data/acpi/pc/DSDT.hpbridge" , +"tests/data/acpi/pc/DSDT.hpbrroot" , +"tests/data/acpi/pc/DSDT.ipmikcs" , +"tests/data/acpi/pc/DSDT.memhp" , +"tests/data/acpi/pc/DSDT.numamem" , +"tests/data/acpi/pc/DSDT.roothp" , +"tests/data/acpi/q35/DSDT" , +"tests/data/acpi/q35/DSDT.acpihmat" , +"tests/data/acpi/q35/DSDT.bridge" , +"tests/data/acpi/q35/DSDT.cphp" , +"tests/data/acpi/q35/DSDT.dimmpxm" , +"tests/data/acpi/q35/DSDT.ipmibt" , +"tests/data/acpi/q35/DSDT.memhp" , +"tests/data/acpi/q35/DSDT.mmio64" , +"tests/data/acpi/q35/DSDT.numamem" , +"tests/data/acpi/q35/DSDT.tis",
Signature · 1 line
2.30.0
RFC PATCH v2 17/32 acpi/pci: Consolidate host bridge setup
In reply to Ben WidawskyThis cleanup will make it easier to add support for CXL to the mix.
--- hw/i386/acpi-build.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)
hw/i386/acpi-build.c +17 -14
@@ -1194,6 +1194,20 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } +enum {PCI ,PCIE }; +static void init_pci_acpi (Aml *dev,int uid,int type) +{ +if (type ==PCI ) { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); + }else { +aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); +aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +aml_append (dev,build_q35_osc_method ()); + } +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1222,9 +1236,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,if (misc->is_piix4) { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); +init_pci_acpi (dev,0 ,PCI );aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (0 )));aml_append (sb_scope, dev);aml_append (dsdt, sb_scope);
@@ -1238,11 +1251,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, }else { sb_scope =aml_scope ("_SB" ); dev =aml_device ("PCI0" ); -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); +init_pci_acpi (dev,0 ,PCIE );aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 ))); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (0 ))); -aml_append (dev,build_q35_osc_method ());aml_append (sb_scope, dev);if (pm->smi_on_cpuhp) {
@@ -1345,15 +1355,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" ); dev =aml_device ("PC%.02X" , bus_num); -aml_append (dev,aml_name_decl ("_UID" ,aml_int (bus_num)));aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -if (pci_bus_is_express (bus)) { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" ))); -aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" ))); -aml_append (dev,build_q35_osc_method ()); - }else { -aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" ))); - } +init_pci_acpi (dev, bus_num,pci_bus_is_express (bus) ?PCIE :PCI );if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
Signature · 1 line
2.30.0
RFC PATCH v2 18/32 tests/acpi: remove stale allowed tables
In reply to Ben Widawsky Scope (_SB)
{
Device (PCI0)
{
Name (_HID, EisaId ("PNP0A03") /* PCI Bus */) // _HID: Hardware ID
- Name (_ADR, Zero) // _ADR: Address
Name (_UID, Zero) // _UID: Unique ID
+ Name (_ADR, Zero) // _ADR: Address--- tests/data/acpi/pc/DSDT | Bin 5065 -> 5065 bytes tests/data/acpi/pc/DSDT.acpihmat | Bin 6390 -> 6390 bytes tests/data/acpi/pc/DSDT.bridge | Bin 6924 -> 6924 bytes tests/data/acpi/pc/DSDT.cphp | Bin 5529 -> 5529 bytes tests/data/acpi/pc/DSDT.dimmpxm | Bin 6719 -> 6719 bytes tests/data/acpi/pc/DSDT.hpbridge | Bin 5026 -> 5026 bytes tests/data/acpi/pc/DSDT.hpbrroot | Bin 3084 -> 3084 bytes tests/data/acpi/pc/DSDT.ipmikcs | Bin 5137 -> 5137 bytes tests/data/acpi/pc/DSDT.memhp | Bin 6424 -> 6424 bytes tests/data/acpi/pc/DSDT.numamem | Bin 5071 -> 5071 bytes tests/data/acpi/pc/DSDT.roothp | Bin 5261 -> 5261 bytes tests/data/acpi/q35/DSDT | Bin 7801 -> 7801 bytes tests/data/acpi/q35/DSDT.acpihmat | Bin 9126 -> 9126 bytes tests/data/acpi/q35/DSDT.bridge | Bin 7819 -> 7819 bytes tests/data/acpi/q35/DSDT.cphp | Bin 8265 -> 8265 bytes tests/data/acpi/q35/DSDT.dimmpxm | Bin 9455 -> 9455 bytes tests/data/acpi/q35/DSDT.ipmibt | Bin 7876 -> 7876 bytes tests/data/acpi/q35/DSDT.memhp | Bin 9160 -> 9160 bytes tests/data/acpi/q35/DSDT.mmio64 | Bin 8932 -> 8932 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7807 -> 7807 bytes tests/qtest/bios-tables-test-allowed-diff.h | 21 -------------------- 21 files changed, 21 deletions(-)
tests/data/acpi/pc/DSDT binary +0 -0
tests/data/acpi/pc/DSDT.acpihmat binary +0 -0
tests/data/acpi/pc/DSDT.bridge binary +0 -0
tests/data/acpi/pc/DSDT.cphp binary +0 -0
tests/data/acpi/pc/DSDT.dimmpxm binary +0 -0
tests/data/acpi/pc/DSDT.hpbridge binary +0 -0
tests/data/acpi/pc/DSDT.hpbrroot binary +0 -0
tests/data/acpi/pc/DSDT.ipmikcs binary +0 -0
tests/data/acpi/pc/DSDT.memhp binary +0 -0
tests/data/acpi/pc/DSDT.numamem binary +0 -0
tests/data/acpi/pc/DSDT.roothp binary +0 -0
tests/data/acpi/q35/DSDT binary +0 -0
tests/data/acpi/q35/DSDT.acpihmat binary +0 -0
tests/data/acpi/q35/DSDT.bridge binary +0 -0
tests/data/acpi/q35/DSDT.cphp binary +0 -0
tests/data/acpi/q35/DSDT.dimmpxm binary +0 -0
tests/data/acpi/q35/DSDT.ipmibt binary +0 -0
tests/data/acpi/q35/DSDT.memhp binary +0 -0
tests/data/acpi/q35/DSDT.mmio64 binary +0 -0
tests/data/acpi/q35/DSDT.numamem binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -21
@@ -1,22 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/pc/DSDT" , -"tests/data/acpi/pc/DSDT.acpihmat" , -"tests/data/acpi/pc/DSDT.bridge" , -"tests/data/acpi/pc/DSDT.cphp" , -"tests/data/acpi/pc/DSDT.dimmpxm" , -"tests/data/acpi/pc/DSDT.hpbridge" , -"tests/data/acpi/pc/DSDT.hpbrroot" , -"tests/data/acpi/pc/DSDT.ipmikcs" , -"tests/data/acpi/pc/DSDT.memhp" , -"tests/data/acpi/pc/DSDT.numamem" , -"tests/data/acpi/pc/DSDT.roothp" , -"tests/data/acpi/q35/DSDT" , -"tests/data/acpi/q35/DSDT.acpihmat" , -"tests/data/acpi/q35/DSDT.bridge" , -"tests/data/acpi/q35/DSDT.cphp" , -"tests/data/acpi/q35/DSDT.dimmpxm" , -"tests/data/acpi/q35/DSDT.ipmibt" , -"tests/data/acpi/q35/DSDT.memhp" , -"tests/data/acpi/q35/DSDT.mmio64" , -"tests/data/acpi/q35/DSDT.numamem" , -"tests/data/acpi/q35/DSDT.tis",
Signature · 1 line
2.30.0
RFC PATCH v2 19/32 hw/pci: Plumb _UID through host bridges
In reply to Ben WidawskyCurrently, QEMU makes _UID equivalent to the bus number (_BBN). While there is nothing wrong with doing it this way, CXL spec has a heavy reliance on _UID to identify host bridges and there is no link to the bus number. Having a distinct UID solves two problems. The first is it gets us around the limitation of 256 (current max bus number). The second is it allows us to replicate hardware configurations where bus number and uid aren't equivalent. The latter has benefits for our development and debugging using QEMU.
The other way to do this would be to implement the expanded bus numbering, but having an explicit uid makes more sense when trying to replicate real hardware configurations.
The QEMU commandline to utilize this would be:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1,uid=x
--
I'm guessing this patch will be somewhat controversial. For early CXL work, this can be dropped without too much heartache.
--- hw/i386/acpi-build.c | 3 ++- hw/pci-bridge/pci_expander_bridge.c | 19 +++++++++++++++++++ hw/pci/pci.c | 11 +++++++++++ include/hw/pci/pci.h | 1 + include/hw/pci/pci_bus.h | 1 + 5 files changed, 34 insertions(+), 1 deletion(-)
hw/i386/acpi-build.c +2 -1
@@ -1343,6 +1343,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, QLIST_FOREACH(bus, &bus->child, sibling) {uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus); +int32_t uid =pci_bus_uid (bus);/* look only for expander root buses */ if (!pci_bus_is_root (bus)) {
@@ -1356,7 +1357,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" ); dev =aml_device ("PC%.02X" , bus_num);aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -init_pci_acpi (dev, bus_num,pci_bus_is_express (bus) ?PCIE :PCI ); +init_pci_acpi (dev, uid,pci_bus_is_express (bus) ?PCIE :PCI );if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
hw/pci-bridge/pci_expander_bridge.c +19 -0
@@ -67,6 +67,7 @@ struct PXBDev {uint8_t bus_nr;uint16_t numa_node; +int32_t uid; };static PXBDev *convert_to_pxb (PCIDevice *dev)
@@ -98,12 +99,20 @@ static uint16_t pxb_bus_numa_node(PCIBus *bus)return pxb->numa_node; } +static int32_t pxb_bus_uid (PCIBus *bus) +{ +PXBDev *pxb =convert_to_pxb (bus->parent_dev); + +return pxb->uid; +} +static void pxb_bus_class_init (ObjectClass *class,void *data) {PCIBusClass *pbc =PCI_BUS_CLASS (class); pbc->bus_num = pxb_bus_num; pbc->numa_node = pxb_bus_numa_node; + pbc->uid = pxb_bus_uid; }static const TypeInfo pxb_bus_info = {
@@ -329,6 +338,7 @@ static Property pxb_dev_properties[] = {/* Note: 0 is not a legal PXB bus number. */ DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ),DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ), +DEFINE_PROP_INT32 ("uid" , PXBDev, uid,-1 ), DEFINE_PROP_END_OF_LIST(), };
@@ -400,12 +410,21 @@ static const TypeInfo pxb_pcie_dev_info = {static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) { +PXBDev *pxb =convert_to_pxb (dev); +/* A CXL PXB's parent bus is still PCIe */ if (!pci_bus_is_express (pci_get_bus (dev))) {error_setg (errp,"pxb-cxl devices cannot reside on a PCI bus" );return ; } +if (pxb->uid <0 ) { +error_setg (errp,"pxb-cxl devices must have a valid uid (0-2147483647)" ); +return ; + } + +/* FIXME: Check that uid doesn't collide with UIDs of other host bridges */ +pxb_dev_realize_common (dev,CXL , errp); }
hw/pci/pci.c +11 -0
@@ -170,6 +170,11 @@ static uint16_t pcibus_numa_node(PCIBus *bus)return NUMA_NODE_UNASSIGNED ; } +static int32_t pcibus_uid (PCIBus *bus) +{ +return -1 ; +} +static void pci_bus_class_init (ObjectClass *klass,void *data) {BusClass *k =BUS_CLASS (klass);
@@ -184,6 +189,7 @@ static void pci_bus_class_init(ObjectClass *klass, void *data) pbc->bus_num = pcibus_num; pbc->numa_node = pcibus_numa_node; + pbc->uid = pcibus_uid; }static const TypeInfo pci_bus_info = {
@@ -530,6 +536,11 @@ int pci_bus_numa_node(PCIBus *bus)return PCI_BUS_GET_CLASS (bus)->numa_node (bus); } +int pci_bus_uid (PCIBus *bus) +{ +return PCI_BUS_GET_CLASS (bus)->uid (bus); +} +static int get_pci_config_device (QEMUFile *f,void *pv,size_t size,const VMStateField *field) {
include/hw/pci/pci.h +1 -0
@@ -462,6 +462,7 @@ static inline int pci_dev_bus_num(const PCIDevice *dev) }int pci_bus_numa_node (PCIBus *bus); +int pci_bus_uid (PCIBus *bus);void pci_for_each_device (PCIBus *bus,int bus_num,void (*fn)(PCIBus *bus,PCIDevice *d,void *opaque),void *opaque);
include/hw/pci/pci_bus.h +1 -0
@@ -17,6 +17,7 @@ struct PCIBusClass {int (*bus_num)(PCIBus *bus);uint16_t (*numa_node)(PCIBus *bus); +int32_t (*uid)(PCIBus *bus); };enum PCIBusFlags {
Signature · 1 line
2.30.0
RFC PATCH v2 20/32 hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
In reply to Ben WidawskyCXL host bridges themselves may have MMIO. Since host bridges don't have a BAR they are treated as special for MMIO.
--
It's arbitrarily chosen here to pick 0xD0000000 as the base for the host bridge MMIO. I'm not sure what the right way to find free space for platform hardcoded things like this is.
--- hw/pci-bridge/pci_expander_bridge.c | 53 ++++++++++++++++++++++++++++- include/hw/cxl/cxl.h | 2 ++ 2 files changed, 54 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +52 -1
@@ -17,6 +17,7 @@#include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h" +#include "hw/cxl/cxl.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "qemu/module.h"
@@ -70,6 +71,12 @@ struct PXBDev {int32_t uid; }; +typedef struct CXLHost { +PCIHostState parent_obj; + +CXLComponentState cxl_cstate; +}CXLHost ; +static PXBDev *convert_to_pxb (PCIDevice *dev) {/* A CXL PXB's parent bus is PCIe, so the normal check won't work */
@@ -85,6 +92,9 @@ static GList *pxb_dev_list;#define TYPE_PXB_HOST "pxb-host" +#define TYPE_PXB_CXL_HOST "pxb-cxl-host" +#define PXB_CXL_HOST (obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST) +static int pxb_bus_num (PCIBus *bus) {PXBDev *pxb =convert_to_pxb (bus->parent_dev);
@@ -198,6 +208,46 @@ static const TypeInfo pxb_host_info = { .class_init = pxb_host_class_init, }; +static void pxb_cxl_realize (DeviceState *dev,Error **errp) +{ +SysBusDevice *sbd =SYS_BUS_DEVICE (dev); +PCIHostState *phb =PCI_HOST_BRIDGE (dev); +CXLHost *cxl =PXB_CXL_HOST (dev); +CXLComponentState *cxl_cstate = &cxl->cxl_cstate; +struct MemoryRegion *mr = &cxl_cstate->crb.component_registers; + +cxl_component_register_block_init (OBJECT (dev), cxl_cstate, +TYPE_PXB_CXL_HOST ); +sysbus_init_mmio (sbd, mr); + +/* FIXME: support multiple host bridges. */ +sysbus_mmio_map (sbd,0 ,CXL_HOST_BASE + +memory_region_size (mr) *pci_bus_uid (phb->bus)); +} + +static void pxb_cxl_host_class_init (ObjectClass *class,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (class); +PCIHostBridgeClass *hc =PCI_HOST_BRIDGE_CLASS (class); + + hc->root_bus_path = pxb_host_root_bus_path; + dc->fw_name ="cxl" ; + dc->realize = pxb_cxl_realize; +/* Reason: Internal part of the pxb/pxb-pcie device, not usable by itself */ + dc->user_creatable = false; +} + +/* +* This is a device to handle the MMIO for a CXL host bridge. It does nothing +* else. +*/ +static const TypeInfo cxl_host_info = { + .name =TYPE_PXB_CXL_HOST , + .parent =TYPE_PCI_HOST_BRIDGE , + .instance_size =sizeof (CXLHost), + .class_init = pxb_cxl_host_class_init, +}; +/* * Registers the PXB bus as a child of pci host root bus. */
@@ -272,7 +322,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, dev_name = dev->qdev.id; } - ds =qdev_new (TYPE_PXB_HOST ); + ds =qdev_new (type ==CXL ?TYPE_PXB_CXL_HOST :TYPE_PXB_HOST );if (type ==PCIE ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_PCIE_BUS ); }else if (type ==CXL ) {
@@ -466,6 +516,7 @@ static void pxb_register_types(void)type_register_static (&pxb_pcie_bus_info);type_register_static (&pxb_cxl_bus_info);type_register_static (&pxb_host_info); +type_register_static (&cxl_host_info);type_register_static (&pxb_dev_info);type_register_static (&pxb_pcie_dev_info);type_register_static (&pxb_cxl_dev_info);
include/hw/cxl/cxl.h +2 -0
@@ -17,5 +17,7 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define CXL_HOST_BASE 0xD0000000 +#endif
Signature · 1 line
2.30.0
RFC PATCH v2 21/32 acpi/pxb/cxl: Reserve host bridge MMIO
In reply to Ben WidawskyFor all host bridges, reserve MMIO space with _CRS. The MMIO for the host bridge lives in a magically hard coded space in the system's physical address space. The standard mechanism to tell the OS about regions which can't be used for host bridges is _CRS.
--- hw/i386/acpi-build.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
hw/i386/acpi-build.c +19 -3
@@ -28,6 +28,7 @@#include "qemu/bitmap.h" #include "qemu/error-report.h" #include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" #include "hw/core/cpu.h" #include "target/i386/cpu.h" #include "hw/misc/pvpanic.h"
@@ -1194,7 +1195,7 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)aml_append (table, scope); } -enum {PCI ,PCIE }; +enum {PCI ,PCIE ,CXL };static void init_pci_acpi (Aml *dev,int uid,int type) {if (type ==PCI ) {
@@ -1344,20 +1345,28 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,uint8_t bus_num =pci_bus_num (bus);uint8_t numa_node =pci_bus_numa_node (bus);int32_t uid =pci_bus_uid (bus); +int type;/* look only for expander root buses */ if (!pci_bus_is_root (bus)) {continue ; } + type =pci_bus_is_cxl (bus) ?CXL : +pci_bus_is_express (bus) ?PCIE :PCI ; +if (bus_num < root_bus_limit) { root_bus_limit = bus_num -1 ; } scope =aml_scope ("\\_SB" ); - dev =aml_device ("PC%.02X" , bus_num); +if (type ==CXL ) { + dev =aml_device ("CXL%.01X" ,pci_bus_uid (bus)); + }else { + dev =aml_device ("PC%.02X" , bus_num); + }aml_append (dev,aml_name_decl ("_BBN" ,aml_int (bus_num))); -init_pci_acpi (dev, uid,pci_bus_is_express (bus) ?PCIE :PCI ); +init_pci_acpi (dev, uid, type);if (numa_node !=NUMA_NODE_UNASSIGNED ) {aml_append (dev,aml_name_decl ("_PXM" ,aml_int (numa_node)));
@@ -1368,6 +1377,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (dev,aml_name_decl ("_CRS" , crs));aml_append (scope, dev);aml_append (dsdt, scope); + +/* Handle the ranges for the PXB expanders */ +if (type ==CXL ) { +uint64_t base =CXL_HOST_BASE + uid *0x10000 ; +crs_range_insert (crs_range_set.mem_ranges, base, + base +0x10000 -1 ); + } } }
Signature · 1 line
2.30.0
RFC PATCH v2 22/32 hw/pxb/cxl: Add "windows" for host bridges
In reply to Ben WidawskyIn a bare metal CXL capable system, system firmware will program physical address ranges on the host. This is done by programming internal registers that aren't typically known to OS. These address ranges might be contiguous or interleaved across host bridges.
For a QEMU guest a new construct is introduced allowing passing a memory backend to the host bridge for this same purpose. Each memory backend needs to be passed to the host bridge as well as any device that will be emulating that memory (not implemented here).
I'm hopeful the interleaving work in the link can be re-purposed here (see Link).
An example to create a host bridges with a 512M window at 0x4c0000000
-object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-memory-base=1,memory-base\[0\]=0x4c0000000,memory\[0\]=cxl-mem1
--- hw/pci-bridge/pci_expander_bridge.c | 65 +++++++++++++++++++++++++++-- include/hw/cxl/cxl.h | 1 + 2 files changed, 62 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +61 -4
@@ -69,12 +69,19 @@ struct PXBDev {uint8_t bus_nr;uint16_t numa_node;int32_t uid; +struct cxl_dev { +HostMemoryBackend *memory_window[CXL_WINDOW_MAX ]; + +uint32_t num_windows; +hwaddr *window_base[CXL_WINDOW_MAX ]; + } cxl; };typedef struct CXLHost {PCIHostState parent_obj;CXLComponentState cxl_cstate; +PXBDev *dev; }CXLHost ;static PXBDev *convert_to_pxb (PCIDevice *dev)
@@ -213,16 +220,31 @@ static void pxb_cxl_realize(DeviceState *dev, Error **errp)SysBusDevice *sbd =SYS_BUS_DEVICE (dev);PCIHostState *phb =PCI_HOST_BRIDGE (dev);CXLHost *cxl =PXB_CXL_HOST (dev); +struct cxl_dev *cxl_dev = &cxl->dev->cxl;CXLComponentState *cxl_cstate = &cxl->cxl_cstate;struct MemoryRegion *mr = &cxl_cstate->crb.component_registers; +int uid =pci_bus_uid (phb->bus);cxl_component_register_block_init (OBJECT (dev), cxl_cstate,TYPE_PXB_CXL_HOST );sysbus_init_mmio (sbd, mr); -/* FIXME: support multiple host bridges. */ -sysbus_mmio_map (sbd,0 ,CXL_HOST_BASE + -memory_region_size (mr) *pci_bus_uid (phb->bus)); +sysbus_mmio_map (sbd,0 ,CXL_HOST_BASE +memory_region_size (mr) * uid); + +/* +* A CXL host bridge can exist without a fixed memory window, but it would +* only operate in legacy PCIe mode. +*/ +if (!cxl_dev->memory_window[uid]) { +warn_report ( +"CXL expander bridge created without window. Consider using %s" , +"memdev[0]=<memory_backend>" ); +return ; + } + + mr =host_memory_backend_get_memory (cxl_dev->memory_window[uid]); +sysbus_init_mmio (sbd, mr); +sysbus_mmio_map (sbd,1 + uid, *cxl_dev->window_base[uid]); }static void pxb_cxl_host_class_init (ObjectClass *class,void *data)
@@ -328,6 +350,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, }else if (type ==CXL ) { bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); bus->flags |=PCI_BUS_CXL ; +PXB_CXL_HOST (ds)->dev =PXB_CXL_DEV (dev); }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
@@ -389,6 +412,8 @@ static Property pxb_dev_properties[] = {DEFINE_PROP_UINT8 ("bus_nr" , PXBDev, bus_nr,0 ),DEFINE_PROP_UINT16 ("numa_node" , PXBDev, numa_node,NUMA_NODE_UNASSIGNED ),DEFINE_PROP_INT32 ("uid" , PXBDev, uid,-1 ), +DEFINE_PROP_ARRAY ("window-base" , PXBDev, cxl.num_windows, cxl.window_base, + qdev_prop_uint64, hwaddr), DEFINE_PROP_END_OF_LIST(), };
@@ -460,7 +485,9 @@ static const TypeInfo pxb_pcie_dev_info = {static void pxb_cxl_dev_realize (PCIDevice *dev,Error **errp) { -PXBDev *pxb =convert_to_pxb (dev); +PXBDev *pxb =PXB_CXL_DEV (dev); +struct cxl_dev *cxl = &pxb->cxl; +int count =0 ;/* A CXL PXB's parent bus is still PCIe */ if (!pci_bus_is_express (pci_get_bus (dev))) {
@@ -476,6 +503,23 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)/* FIXME: Check that uid doesn't collide with UIDs of other host bridges */ pxb_dev_realize_common (dev,CXL , errp); + +for (unsigned i =0 ; i <CXL_WINDOW_MAX ; i++) { +if (!cxl->memory_window[i]) { +continue ; + } + + count++; + } + +if (!count) { +warn_report ("memory-windows should be set when creating CXL host bridges" ); + } + +if (count != cxl->num_windows) { +error_setg (errp,"window bases count (%d) must match window count (%d)" , + cxl->num_windows, count); + } }static void pxb_cxl_dev_class_init (ObjectClass *klass,void *data)
@@ -496,6 +540,19 @@ static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)/* Host bridges aren't hotpluggable. FIXME: spec reference */ dc->hotpluggable = false; + +/* +* Below is moral equivalent of: +* DEFINE_PROP_ARRAY("memdev", PXBDev, window_count, windows, +* qdev_prop_memory_device, HostMemoryBackend) +*/ +for (unsigned i =0 ; i <CXL_WINDOW_MAX ; i++) { + g_autofree char *name = g_strdup_printf("memdev[%u]", i); +object_class_property_add_link (klass, name,TYPE_MEMORY_BACKEND , + offsetof(PXBDev, cxl.memory_window[i]), + qdev_prop_allow_set_link_before_realize, + OBJ_PROP_LINK_STRONG); + } }static const TypeInfo pxb_cxl_dev_info = {
include/hw/cxl/cxl.h +1 -0
@@ -18,6 +18,7 @@#define DEVICE_REG_BAR_IDX 2#define CXL_HOST_BASE 0xD0000000 +#define CXL_WINDOW_MAX 10#endif
Signature · 1 line
2.30.0
RFC PATCH v2 23/32 hw/cxl/rp: Add a root port
In reply to Ben WidawskyThis adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4
Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.
--- hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 4 +- 5 files changed, 245 insertions(+), 2 deletions(-) create mode 100644 hw/pci-bridge/cxl_root_port.c
hw/pci-bridge/Kconfig +5 -0
@@ -27,3 +27,8 @@ config DEC_PCI config SIMBA bool + +config CXL + bool + default y if PCI_EXPRESS && PXB + depends on PCI_EXPRESS && MSI_NONBROKEN && PXB
hw/pci-bridge/cxl_root_port.c new file +231 -0
231 lines not on this page · raw message
hw/pci-bridge/meson.build +1 -0
@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c')) pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c')) pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c')) pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c')) +pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c')) # NewWorld PowerMac pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))
hw/pci-bridge/pcie_root_port.c +5 -1
@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)int rc;pci_config_set_interrupt_pin (d->config,1 ); -pci_bridge_initfn (d,TYPE_PCIE_BUS ); +if (d->cap_present &QEMU_PCIE_CAP_CXL ) { +pci_bridge_initfn (d,TYPE_CXL_BUS ); + }else { +pci_bridge_initfn (d,TYPE_PCIE_BUS ); + }pcie_port_init_reg (d); rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,
hw/pci/pci.c +3 -1
@@ -2692,7 +2692,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)object_class_dynamic_cast (klass,INTERFACE_CONVENTIONAL_PCI_DEVICE );ObjectClass *pcie =object_class_dynamic_cast (klass,INTERFACE_PCIE_DEVICE ); -assert (conventional || pcie); +ObjectClass *cxl = +object_class_dynamic_cast (klass,INTERFACE_CXL_DEVICE ); +assert (conventional || pcie || cxl); } }
Signature · 1 line
2.30.0
RFC PATCH v2 24/32 hw/cxl/device: Add a memory device (8.2.8.5)
In reply to Ben WidawskyA CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"
--- hw/core/numa.c | 3 + hw/cxl/cxl-mailbox-utils.c | 41 ++++++ hw/i386/pc.c | 1 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 262 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + hw/pci/pcie.c | 30 +++++ include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 ++++ include/hw/pci/pci_ids.h | 1 + monitor/hmp-cmds.c | 15 +++ qapi/machine.json | 1 + 12 files changed, 384 insertions(+) create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[pcdimm_info->node].node_plugged_mem += pcdimm_info->size;break ; +case MEMORY_DEVICE_INFO_KIND_CXL : +/* FINISHME */ +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data;/* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0
65 lines not on this page · raw message
hw/i386/pc.c +1 -0
@@ -79,6 +79,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
@@ -10,3 +10,8 @@ config NVDIMM default y depends on (PC || PSERIES || ARM_VIRT) select MEM_DEVICE + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICE
hw/mem/cxl_type3.c new file +262 -0
262 lines not on this page · raw message
hw/mem/meson.build +1 -0
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +30 -0
@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); +* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } +* } +*/ + +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + } +#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10
include/hw/cxl/cxl_pci.h +22 -0
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06
monitor/hmp-cmds.c +15 -0
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Signature · 1 line
2.30.0
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Review
Quoted 9 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"
I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?
Quoted 235 lines
Signed-off-by Ben Widawsky <[email protected]>--- hw/core/numa.c | 3 + hw/cxl/cxl-mailbox-utils.c | 41 ++++++ hw/i386/pc.c | 1 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 262 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + hw/pci/pcie.c | 30 +++++ include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 ++++ include/hw/pci/pci_ids.h | 1 + monitor/hmp-cmds.c | 15 +++ qapi/machine.json | 1 + 12 files changed, 384 insertions(+) create mode 100644 hw/mem/cxl_type3.chw/core/numa.c +3
-0@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[pcdimm_info->node].node_plugged_mem += pcdimm_info->size;break ; +case MEMORY_DEVICE_INFO_KIND_CXL : +/* FINISHME */ +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data;/* TODO: once we support numa, assign to right node */ hw/cxl/cxl-mailbox-utils.c +41
-0@@ -49,6 +49,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG ); +declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ), + CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0), };#undef CXL_CMD@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +define_mailbox_handler (IDENTIFY_MEMORY_DEVICE ) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +if (memory_region_size (cxl_dstate->pmem) < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + id->total_capacity =memory_region_size (cxl_dstate->pmem); + id->persistent_capacity =memory_region_size (cxl_dstate->pmem); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;hw/i386/pc.c +1
-0@@ -79,6 +79,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/visitor.h" hw/mem/Kconfig +5
-0@@ -10,3 +10,8 @@ config NVDIMM default y depends on (PC || PSERIES || ARM_VIRT) select MEM_DEVICE + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICEhw/mem/cxl_type3.c new file +100
-0@@ -0,0 +1,262 @@ +#include "qemu/osdep.h" +#include "qemu/units.h" +#include "qemu/error-report.h" +#include "hw/mem/memory-device.h" +#include "hw/mem/pc-dimm.h" +#include "hw/pci/pci.h" +#include "hw/qdev-properties.h" +#include "qapi/error.h" +#include "qemu/log.h" +#include "qemu/module.h" +#include "qemu/range.h" +#include "qemu/rcu.h" +#include "sysemu/hostmem.h" +#include "hw/cxl/cxl.h" + +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) + +static void build_dvsecs (CXLType3Dev *ct3d) +{ +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct dvsec_device ){ + .cap =0x1e , + .ctrl =0x6 , + .status2 =0x2 , + .range1_size_hi =0 , + .range1_size_lo = (2 <<5 ) | (2 <<2 ) |0x3 | ct3d->size, + .range1_base_hi =0 , + .range1_base_lo =0 , + }; +cxl_component_create_dvsec (cxl_cstate,PCIE_CXL_DEVICE_DVSEC_LENGTH , +PCIE_CXL_DEVICE_DVSEC , +PCIE_CXL2_DEVICE_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + .reg1_base_lo =RBI_CXL_DEVICE_REG |DEVICE_REG_BAR_IDX , + .reg1_base_hi =0 , + }; +cxl_component_create_dvsec (cxl_cstate,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void ct3_instance_init (Object *obj) +{ +/* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */ +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); + +g_free (ct3d->cxl_dstate.pmem); +} + +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset;
hw/mem/cxl_type3.c line 110
Quoted 10 lines
hw/mem/cxl_type3.c new file +10
-0+if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size); + } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);
this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?
hw/mem/cxl_type3.c line 115
Quoted 5 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif
What this hunk is supposed to do, why it's ifdef-ed?
Quoted 193 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +13
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444
Quoted 10 lines
hw/pci/pcie.c +10
-0+* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } +* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }
why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?
Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Quoted 8 lines
Review
Quoted 4 lines
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?
A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.
This isn't implemented in the code yet, but it is planned.
Quoted 578 lines
Quoted 237 lines
Signed-off-by Ben Widawsky <[email protected]>--- hw/core/numa.c | 3 + hw/cxl/cxl-mailbox-utils.c | 41 ++++++ hw/i386/pc.c | 1 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 262 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + hw/pci/pcie.c | 30 +++++ include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 ++++ include/hw/pci/pci_ids.h | 1 + monitor/hmp-cmds.c | 15 +++ qapi/machine.json | 1 + 12 files changed, 384 insertions(+) create mode 100644 hw/mem/cxl_type3.chw/core/numa.c +3
-0@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[pcdimm_info->node].node_plugged_mem += pcdimm_info->size;break ; +case MEMORY_DEVICE_INFO_KIND_CXL : +/* FINISHME */ +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data;/* TODO: once we support numa, assign to right node */ hw/cxl/cxl-mailbox-utils.c +41
-0@@ -49,6 +49,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG ); +declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ), + CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0), };#undef CXL_CMD@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +define_mailbox_handler (IDENTIFY_MEMORY_DEVICE ) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +if (memory_region_size (cxl_dstate->pmem) < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + id->total_capacity =memory_region_size (cxl_dstate->pmem); + id->persistent_capacity =memory_region_size (cxl_dstate->pmem); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;hw/i386/pc.c +1
-0@@ -79,6 +79,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/visitor.h" hw/mem/Kconfig +5
-0@@ -10,3 +10,8 @@ config NVDIMM default y depends on (PC || PSERIES || ARM_VIRT) select MEM_DEVICE + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICEhw/mem/cxl_type3.c new file +100
-0@@ -0,0 +1,262 @@ +#include "qemu/osdep.h" +#include "qemu/units.h" +#include "qemu/error-report.h" +#include "hw/mem/memory-device.h" +#include "hw/mem/pc-dimm.h" +#include "hw/pci/pci.h" +#include "hw/qdev-properties.h" +#include "qapi/error.h" +#include "qemu/log.h" +#include "qemu/module.h" +#include "qemu/range.h" +#include "qemu/rcu.h" +#include "sysemu/hostmem.h" +#include "hw/cxl/cxl.h" + +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) + +static void build_dvsecs (CXLType3Dev *ct3d) +{ +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct dvsec_device ){ + .cap =0x1e , + .ctrl =0x6 , + .status2 =0x2 , + .range1_size_hi =0 , + .range1_size_lo = (2 <<5 ) | (2 <<2 ) |0x3 | ct3d->size, + .range1_base_hi =0 , + .range1_base_lo =0 , + }; +cxl_component_create_dvsec (cxl_cstate,PCIE_CXL_DEVICE_DVSEC_LENGTH , +PCIE_CXL_DEVICE_DVSEC , +PCIE_CXL2_DEVICE_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + .reg1_base_lo =RBI_CXL_DEVICE_REG |DEVICE_REG_BAR_IDX , + .reg1_base_hi =0 , + }; +cxl_component_create_dvsec (cxl_cstate,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void ct3_instance_init (Object *obj) +{ +/* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */ +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); + +g_free (ct3d->cxl_dstate.pmem); +} + +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset;hw/mem/cxl_type3.c line 110
Quoted 10 lines
hw/mem/cxl_type3.c new file +10
-0+if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size); + } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?hw/mem/cxl_type3.c line 115
Quoted 5 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
Quoted 193 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +13
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444
Quoted 10 lines
hw/pci/pcie.c +10
-0+* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } +* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 18 lines · On Wed, 27 Jan 2021 13:11:16 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Quoted 8 lines
Review
Quoted 4 lines
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?
Quoted 4 lines
A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.
This isn't implemented in the code yet, but it is planned.
could you add here how it supposed to look like CLI interface wise?
also see other questions below.
Quoted 579 lines
Quoted 578 lines
Quoted 237 lines
Signed-off-by Ben Widawsky <[email protected]>--- hw/core/numa.c | 3 + hw/cxl/cxl-mailbox-utils.c | 41 ++++++ hw/i386/pc.c | 1 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 262 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + hw/pci/pcie.c | 30 +++++ include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 ++++ include/hw/pci/pci_ids.h | 1 + monitor/hmp-cmds.c | 15 +++ qapi/machine.json | 1 + 12 files changed, 384 insertions(+) create mode 100644 hw/mem/cxl_type3.chw/core/numa.c +3
-0@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[pcdimm_info->node].node_plugged_mem += pcdimm_info->size;break ; +case MEMORY_DEVICE_INFO_KIND_CXL : +/* FINISHME */ +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data;/* TODO: once we support numa, assign to right node */ hw/cxl/cxl-mailbox-utils.c +41
-0@@ -49,6 +49,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG ); +declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ), + CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0), };#undef CXL_CMD@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +define_mailbox_handler (IDENTIFY_MEMORY_DEVICE ) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +if (memory_region_size (cxl_dstate->pmem) < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + id->total_capacity =memory_region_size (cxl_dstate->pmem); + id->persistent_capacity =memory_region_size (cxl_dstate->pmem); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;hw/i386/pc.c +1
-0@@ -79,6 +79,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/visitor.h" hw/mem/Kconfig +5
-0@@ -10,3 +10,8 @@ config NVDIMM default y depends on (PC || PSERIES || ARM_VIRT) select MEM_DEVICE + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICEhw/mem/cxl_type3.c new file +100
-0@@ -0,0 +1,262 @@ +#include "qemu/osdep.h" +#include "qemu/units.h" +#include "qemu/error-report.h" +#include "hw/mem/memory-device.h" +#include "hw/mem/pc-dimm.h" +#include "hw/pci/pci.h" +#include "hw/qdev-properties.h" +#include "qapi/error.h" +#include "qemu/log.h" +#include "qemu/module.h" +#include "qemu/range.h" +#include "qemu/rcu.h" +#include "sysemu/hostmem.h" +#include "hw/cxl/cxl.h" + +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) + +static void build_dvsecs (CXLType3Dev *ct3d) +{ +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct dvsec_device ){ + .cap =0x1e , + .ctrl =0x6 , + .status2 =0x2 , + .range1_size_hi =0 , + .range1_size_lo = (2 <<5 ) | (2 <<2 ) |0x3 | ct3d->size, + .range1_base_hi =0 , + .range1_base_lo =0 , + }; +cxl_component_create_dvsec (cxl_cstate,PCIE_CXL_DEVICE_DVSEC_LENGTH , +PCIE_CXL_DEVICE_DVSEC , +PCIE_CXL2_DEVICE_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + .reg1_base_lo =RBI_CXL_DEVICE_REG |DEVICE_REG_BAR_IDX , + .reg1_base_hi =0 , + }; +cxl_component_create_dvsec (cxl_cstate,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void ct3_instance_init (Object *obj) +{ +/* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */ +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); + +g_free (ct3d->cxl_dstate.pmem); +} + +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset;hw/mem/cxl_type3.c line 110
Quoted 10 lines
hw/mem/cxl_type3.c new file +10
-0+if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size); + } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?hw/mem/cxl_type3.c line 115
Quoted 5 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
Quoted 193 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +13
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444
Quoted 10 lines
hw/pci/pcie.c +10
-0+* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } +* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 21 lines · Igor Mammedov
Quoted 18 lines · On Wed, 27 Jan 2021 13:11:16 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 16 lines · Igor Mammedov
Review
Quoted 9 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?
Quoted 9 lines
Quoted 4 lines
A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.
This isn't implemented in the code yet, but it is planned.
could you add here how it supposed to look like CLI interface wise?
also see other questions below.
My mistake on the other questions. I forked another thread for those.
Interleave is still being fleshed out. But generally to set up a 512M address range interleaves across 2 devices, each 256M, and each connected to a root port on the host bridge:
# Memory backend
-object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
# Host Bridge
-device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c0000000 memdev[0]=cxl-mem1
# 2 root ports
-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 -device cxl-rp,id=rp1,bus=cxl.0,addr=0.1,chassis=0,slot=1,memdev=cxl-mem1
# 2 PMEM devices
-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
In reply to Igor MammedovQuoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
Quoted 8 lines
Review
Quoted 4 lines
The following example will create a 256M device in a 512M window:
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?
Answered in a separate thread...
Quoted 242 lines
Quoted 242 lines
Signed-off-by Ben Widawsky <[email protected]>--- hw/core/numa.c | 3 + hw/cxl/cxl-mailbox-utils.c | 41 ++++++ hw/i386/pc.c | 1 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 262 +++++++++++++++++++++++++++++++++++++ hw/mem/meson.build | 1 + hw/pci/pcie.c | 30 +++++ include/hw/cxl/cxl.h | 2 + include/hw/cxl/cxl_pci.h | 22 ++++ include/hw/pci/pci_ids.h | 1 + monitor/hmp-cmds.c | 15 +++ qapi/machine.json | 1 + 12 files changed, 384 insertions(+) create mode 100644 hw/mem/cxl_type3.chw/core/numa.c +3
-0@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[pcdimm_info->node].node_plugged_mem += pcdimm_info->size;break ; +case MEMORY_DEVICE_INFO_KIND_CXL : +/* FINISHME */ +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data;/* TODO: once we support numa, assign to right node */ hw/cxl/cxl-mailbox-utils.c +41
-0@@ -49,6 +49,8 @@ enum { LOGS = 0x04,#define GET_SUPPORTED 0x0#define GET_LOG 0x1 + IDENTIFY = 0x40, +#define MEMORY_DEVICE 0x0 };/* 8.2.8.4.5.1 Command Return Codes */ @@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);declare_mailbox_handler (TIMESTAMP_SET );declare_mailbox_handler (LOGS_GET_SUPPORTED );declare_mailbox_handler (LOGS_GET_LOG ); +declare_mailbox_handler (IDENTIFY_MEMORY_DEVICE );#define IMMEDIATE_CONFIG_CHANGE (1 << 1)#define IMMEDIATE_POLICY_CHANGE (1 << 3)@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {CXL_CMD (TIMESTAMP ,SET ,8 ,IMMEDIATE_POLICY_CHANGE ),CXL_CMD (LOGS ,GET_SUPPORTED ,0 ,0 ),CXL_CMD (LOGS ,GET_LOG ,0x18 ,0 ), + CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0), };#undef CXL_CMD@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)return CXL_MBOX_SUCCESS ; } +/* 8.2.9.5.1.1 */ +define_mailbox_handler (IDENTIFY_MEMORY_DEVICE ) +{ +struct { +char fw_revision[0x10 ]; +uint64_t total_capacity; +uint64_t volatile_capacity; +uint64_t persistent_capacity; +uint64_t partition_align; +uint16_t info_event_log_size; +uint16_t warning_event_log_size; +uint16_t failure_event_log_size; +uint16_t fatal_event_log_size; +uint32_t lsa_size; +uint8_t poison_list_max_mer[3 ]; +uint16_t inject_poison_limit; +uint8_t poison_caps; +uint8_t qos_telemetry_caps; + } __attribute__((packed)) *id; +_Static_assert (sizeof (*id) ==0x43 ,"Bad identify size" ); + +if (memory_region_size (cxl_dstate->pmem) < (256 <<20 )) { +return CXL_MBOX_INTERNAL_ERROR ; + } + + id = (void *)cmd->payload; +memset (id,0 ,sizeof (*id)); + +/* PMEM only */ +snprintf (id->fw_revision,0x10 ,"BWFW VERSION %02d" ,0 ); + id->total_capacity =memory_region_size (cxl_dstate->pmem); + id->persistent_capacity =memory_region_size (cxl_dstate->pmem); + + *len =sizeof (*id); +return CXL_MBOX_SUCCESS ; +} +void cxl_process_mailbox (CXLDeviceState *cxl_dstate) {uint16_t ret =CXL_MBOX_SUCCESS ;hw/i386/pc.c +1
-0@@ -79,6 +79,7 @@#include "acpi-build.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "qapi/error.h" #include "qapi/qapi-visit-common.h" #include "qapi/visitor.h" hw/mem/Kconfig +5
-0@@ -10,3 +10,8 @@ config NVDIMM default y depends on (PC || PSERIES || ARM_VIRT) select MEM_DEVICE + +config CXL_MEM_DEVICE + bool + default y if CXL + select MEM_DEVICEhw/mem/cxl_type3.c new file +105
-0@@ -0,0 +1,262 @@ +#include "qemu/osdep.h" +#include "qemu/units.h" +#include "qemu/error-report.h" +#include "hw/mem/memory-device.h" +#include "hw/mem/pc-dimm.h" +#include "hw/pci/pci.h" +#include "hw/qdev-properties.h" +#include "qapi/error.h" +#include "qemu/log.h" +#include "qemu/module.h" +#include "qemu/range.h" +#include "qemu/rcu.h" +#include "sysemu/hostmem.h" +#include "hw/cxl/cxl.h" + +typedef struct cxl_type3_dev { +/* Private */ +PCIDevice parent_obj; + +/* Properties */ +uint64_t size; +HostMemoryBackend *hostmem; + +/* State */ +CXLComponentState cxl_cstate; +CXLDeviceState cxl_dstate; +}CXLType3Dev ; + +#define CT3 (obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV) + +static void build_dvsecs (CXLType3Dev *ct3d) +{ +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +uint8_t *dvsec; + + dvsec = (uint8_t *)&(struct dvsec_device ){ + .cap =0x1e , + .ctrl =0x6 , + .status2 =0x2 , + .range1_size_hi =0 , + .range1_size_lo = (2 <<5 ) | (2 <<2 ) |0x3 | ct3d->size, + .range1_base_hi =0 , + .range1_base_lo =0 , + }; +cxl_component_create_dvsec (cxl_cstate,PCIE_CXL_DEVICE_DVSEC_LENGTH , +PCIE_CXL_DEVICE_DVSEC , +PCIE_CXL2_DEVICE_DVSEC_REVID , dvsec); + + dvsec = (uint8_t *)&(struct dvsec_register_locator ){ + .rsvd =0 , + .reg0_base_lo =RBI_COMPONENT_REG |COMPONENT_REG_BAR_IDX , + .reg0_base_hi =0 , + .reg1_base_lo =RBI_CXL_DEVICE_REG |DEVICE_REG_BAR_IDX , + .reg1_base_hi =0 , + }; +cxl_component_create_dvsec (cxl_cstate,REG_LOC_DVSEC_LENGTH ,REG_LOC_DVSEC , +REG_LOC_DVSEC_REVID , dvsec); +} + +static void ct3_instance_init (Object *obj) +{ +/* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */ +} + +static void ct3_finalize (Object *obj) +{ +CXLType3Dev *ct3d =CT3 (obj); + +g_free (ct3d->cxl_dstate.pmem); +} + +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset; +if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size);
Quoted 8 lines
hw/mem/cxl_type3.c line 110
Quoted 5 lines
hw/mem/cxl_type3.c new file +5
-0+ } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?
Quoted 7 lines
hw/mem/cxl_type3.c line 115
Quoted 6 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.
Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +17
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); +* if (local_err) { +* error_propagate(errp, local_err); +* return; +* }
Quoted 9 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445
Quoted 6 lines
hw/pci/pcie.c +6
-0+* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?
A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.
Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35
-0+ +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset; +if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size);Quoted 8 lines
hw/mem/cxl_type3.c line 110
Quoted 5 lines
hw/mem/cxl_type3.c new file +5
-0+ } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?
Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?
Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
"cxl_type3-memory", mr, offset, ct3d->size);
I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call memory_region_set_address() to put it in a particular location.
Quoted 340 lines
Quoted 9 lines
hw/mem/cxl_type3.c line 115
Quoted 8 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.
Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +17
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); +* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } Quoted 9 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445
Quoted 6 lines
hw/pci/pcie.c +6
-0+* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.
Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35
-0+ +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset; +if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size);Quoted 8 lines
hw/mem/cxl_type3.c line 110
Quoted 5 lines
hw/mem/cxl_type3.c new file +5
-0+ } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
Quoted 8 lines
"cxl_type3-memory", mr, offset, ct3d->size);I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call
memory_region_set_address()to put it in a particular location.
Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.
range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.
I was digging into this yesterday, but opted to start a new thread on the matter.
Quoted 342 lines
Quoted 340 lines
Quoted 9 lines
hw/mem/cxl_type3.c line 115
Quoted 8 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.
Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +17
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); +* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } Quoted 9 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445
Quoted 6 lines
hw/pci/pcie.c +6
-0+* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.
Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 85 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35
-0+ +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset; +if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size);Quoted 8 lines
hw/mem/cxl_type3.c line 110
Quoted 5 lines
hw/mem/cxl_type3.c new file +5
-0+ } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
Quoted 8 lines
"cxl_type3-memory", mr, offset, ct3d->size);I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call
memory_region_set_address()to put it in a particular location.
Quoted 8 lines
Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.
range_invariant: Assertion `range->lob <=range->upb||range->lob==range->upb+ 1' failed.I was digging into this yesterday, but opted to start a new thread on the matter.
Hmm. I think I need to figure out the right add_subregion after this and it might work. I'll keep digging, but if you have ideas, let me know.
Quoted 343 lines
Quoted 342 lines
Quoted 340 lines
Quoted 9 lines
hw/mem/cxl_type3.c line 115
Quoted 8 lines
hw/mem/cxl_type3.c new file +4
-0+memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); +#ifdef SET_PMEM_PADDR +memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem); +#endif What this hunk is supposed to do, why it's ifdef-ed?
In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.
Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147
-0+} + +static MemoryRegion *cxl_md_get_memory_region (MemoryDeviceState *md, +Error **errp) +{ +CXLType3Dev *ct3d =CT3 (md); + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +return ct3d->cxl_dstate.pmem; +} + +static void ct3_realize (PCIDevice *pci_dev,Error **errp) +{ +CXLType3Dev *ct3d =CT3 (pci_dev); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +MemoryRegion *mr = ®s->component_registers; +uint8_t *pci_conf = pci_dev->config; + +if (!ct3d->cxl_dstate.pmem) { +cxl_setup_memory (ct3d, errp); + } + +pci_config_set_prog_interface (pci_conf,0x10 ); +pci_config_set_class (pci_conf,PCI_CLASS_MEMORY_CXL ); + +pcie_endpoint_cap_init (pci_dev,0x80 ); + cxl_cstate->dvsec_offset =0x100 ; + + ct3d->cxl_cstate.pdev = pci_dev; +build_dvsecs (ct3d); + +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate, +TYPE_CXL_TYPE3_DEV ); + +pci_register_bar ( + pci_dev,COMPONENT_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , mr); + +cxl_device_register_block_init (OBJECT (pci_dev), &ct3d->cxl_dstate); +pci_register_bar (pci_dev,DEVICE_REG_BAR_IDX , +PCI_BASE_ADDRESS_SPACE_MEMORY | +PCI_BASE_ADDRESS_MEM_TYPE_64 , + &ct3d->cxl_dstate.device_registers); +} + +static uint64_t cxl_md_get_addr (const MemoryDeviceState *md) +{ +CXLType3Dev *ct3d =CT3 (md); + +return memory_region_get_ram_addr (ct3d->cxl_dstate.pmem); +} + +static void cxl_md_set_addr (MemoryDeviceState *md,uint64_t addr,Error **errp) +{ +object_property_set_uint (OBJECT (md),"paddr" , addr, errp); +} + +static void ct3d_reset (DeviceState *dev) +{ +CXLType3Dev *ct3d =CT3 (dev); +uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; + +cxl_component_register_init_common (reg_state,CXL2_TYPE3_DEVICE ); +cxl_device_register_init_common (&ct3d->cxl_dstate); +} + +static Property ct3_props[] = { +DEFINE_PROP_SIZE ("size" , CXLType3Dev, size,-1 ), +DEFINE_PROP_LINK ("memdev" , CXLType3Dev, hostmem,TYPE_MEMORY_BACKEND , + HostMemoryBackend *), +DEFINE_PROP_END_OF_LIST (), +}; + +static void pc_dimm_md_fill_device_info (const MemoryDeviceState *md, +MemoryDeviceInfo *info) +{ +PCDIMMDeviceInfo *di =g_new0 (PCDIMMDeviceInfo,1 ); +const DeviceClass *dc =DEVICE_GET_CLASS (md); +const DeviceState *dev =DEVICE (md); +CXLType3Dev *ct3d =CT3 (md); + +if (dev->id) { + di->has_id = true; + di->id =g_strdup (dev->id); + } + di->hotplugged = dev->hotplugged; + di->hotpluggable = dc->hotpluggable; + di->addr =cxl_md_get_addr (md); + di->slot =0 ; + di->node =0 ; + di->size =memory_device_get_region_size (md,NULL ); + di->memdev =object_get_canonical_path (OBJECT (ct3d->hostmem)); + + + info->u.cxl.data = di; + info->type =MEMORY_DEVICE_INFO_KIND_CXL ; +} + +static void ct3_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +PCIDeviceClass *pc =PCI_DEVICE_CLASS (oc); +MemoryDeviceClass *mdc =MEMORY_DEVICE_CLASS (oc); + + pc->realize = ct3_realize; + pc->class_id =PCI_CLASS_STORAGE_EXPRESS ; + pc->vendor_id =PCI_VENDOR_ID_INTEL ; + pc->device_id =0xd93 ;/* LVF for now */ + pc->revision =1 ; + +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); + dc->desc ="CXL PMEM Device (Type 3)" ; + dc->reset = ct3d_reset; +device_class_set_props (dc, ct3_props); + + mdc->get_memory_region = cxl_md_get_memory_region; + mdc->get_addr = cxl_md_get_addr; + mdc->fill_device_info = pc_dimm_md_fill_device_info; + mdc->get_plugged_size = memory_device_get_region_size; + mdc->set_addr = cxl_md_set_addr; +} + +static const TypeInfo ct3d_info = { + .name =TYPE_CXL_TYPE3_DEV , + .parent =TYPE_PCI_DEVICE , + .class_init = ct3_class_init, + .instance_size =sizeof (CXLType3Dev), + .instance_init = ct3_instance_init, + .instance_finalize = ct3_finalize, + .interfaces = (InterfaceInfo []) { + {TYPE_MEMORY_DEVICE }, + {INTERFACE_CXL_DEVICE }, + {INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void ct3d_registers (void ) +{ +type_register_static (&ct3d_info); +} + +type_init (ct3d_registers);hw/mem/meson.build +1
-0@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c')) mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c')) mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c')) mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c')) +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c')) softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)hw/pci/pcie.c +17
-0@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/mem/memory-device.h" #include "hw/pci/pci_bridge.h" #include "hw/pci/pcie.h" #include "hw/pci/msix.h" @@ -27,6 +28,8 @@#include "hw/pci/pci_bus.h" #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" +#include "hw/cxl/cxl.h" +#include "hw/boards.h" #include "qemu/range.h" //#define DEBUG_PCIE @@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, }pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); + +#ifdef CXL_MEM_DEVICE +/* +* FIXME: +* if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) { +* HotplugHandler *hotplug_ctrl; +* Error *local_err = NULL; +* hotplug_ctrl = qdev_get_hotplug_handler(dev); +* if (hotplug_ctrl) { +* hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err); +* if (local_err) { +* error_propagate(errp, local_err); +* return; +* } Quoted 9 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445
Quoted 6 lines
hw/pci/pcie.c +6
-0+* } +*/ +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) { +memory_device_pre_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ()), +NULL , errp); + }why use
MEMORY_DEVICEinterface instead of exposing memory as PCI BAR?A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.
Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6
-0+#endif } void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,pcie_cap_slot_event (hotplug_pdev,PCI_EXP_HP_EV_PDC |PCI_EXP_HP_EV_ABP ); } + +#ifdef CXL_MEM_DEVICE +if (object_dynamic_cast (OBJECT (dev),TYPE_CXL_TYPE3_DEV )) +memory_device_plug (MEMORY_DEVICE (dev),MACHINE (qdev_get_machine ())); +#endif } void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,include/hw/cxl/cxl.h +2
-0@@ -17,6 +17,8 @@#define COMPONENT_REG_BAR_IDX 0#define DEVICE_REG_BAR_IDX 2 +#define TYPE_CXL_TYPE3_DEV "cxl-type3" +#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10include/hw/cxl/cxl_pci.h +22
-0@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,* CXL 2.0 Downstream Port: 3, 4, 7, 8 */ +/* CXL 2.0 - 8.1.3 (ID 0001) */ +struct dvsec_device { +struct dvsec_header hdr; +uint16_t cap; +uint16_t ctrl; +uint16_t status; +uint16_t ctrl2; +uint16_t status2; +uint16_t lock; +uint16_t cap2; +uint32_t range1_size_hi; +uint32_t range1_size_lo; +uint32_t range1_base_hi; +uint32_t range1_base_lo; +uint32_t range2_size_hi; +uint32_t range2_size_lo; +uint32_t range2_base_hi; +uint32_t range2_base_lo; +}; +_Static_assert (sizeof (struct dvsec_device ) ==0x38 , +"dvsec device size incorrect" ); +/* CXL 2.0 - 8.1.5 (ID 0003) */ struct extensions_dvsec_port {struct dvsec_header hdr;include/hw/pci/pci_ids.h +1
-0@@ -53,6 +53,7 @@#define PCI_BASE_CLASS_MEMORY 0x05#define PCI_CLASS_MEMORY_RAM 0x0500#define PCI_CLASS_MEMORY_FLASH 0x0501 +#define PCI_CLASS_MEMORY_CXL 0x0502#define PCI_CLASS_MEMORY_OTHER 0x0580#define PCI_BASE_CLASS_BRIDGE 0x06monitor/hmp-cmds.c +15
-0@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)monitor_printf (mon," hotpluggable: %s\n" , di->hotpluggable ?"true" :"false" );break ; +case MEMORY_DEVICE_INFO_KIND_CXL : + di = value->u.cxl.data; +monitor_printf (mon,"Memory device [%s]: \"%s\"\n" , +MemoryDeviceInfoKind_str (value->type), + di->id ? di->id :"" ); +monitor_printf (mon," addr: 0x%" PRIx64"\n" , di->addr); +monitor_printf (mon," slot: %" PRId64"\n" , di->slot); +monitor_printf (mon," node: %" PRId64"\n" , di->node); +monitor_printf (mon," size: %" PRIu64"\n" , di->size); +monitor_printf (mon," memdev: %s\n" , di->memdev); +monitor_printf (mon," hotplugged: %s\n" , + di->hotplugged ?"true" :"false" ); +monitor_printf (mon," hotpluggable: %s\n" , + di->hotpluggable ?"true" :"false" ); +break ;case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM : vpi = value->u.virtio_pmem.data; monitor_printf(mon, "Memory device [%s]: \"%s\"\n",qapi/machine.json +1
-0@@ -1394,6 +1394,7 @@ { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', + 'cxl': 'PCDIMMDeviceInfo', 'virtio-pmem': 'VirtioPMEMDeviceInfo', 'virtio-mem': 'VirtioMEMDeviceInfo' }
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35
-0+ +static void cxl_setup_memory (CXLType3Dev *ct3d,Error **errp) +{ +MemoryRegionSection mrs; +MemoryRegion *mr; +uint64_t offset =0 ; +size_t remaining_size; + +if (!ct3d->hostmem) { +error_setg (errp,"memdev property must be set" ); +return ; + } + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem); + +/* Create our new subregion */ + ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); + +/* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs =memory_region_find (mr, offset,1 ); +while (mrs.mr && mrs.mr != mr) { + offset +=memory_region_size (mrs.mr); + mrs =memory_region_find (mr, offset,1 ); + } + } + + remaining_size =memory_region_size (mr) - offset; +if (remaining_size < ct3d->size) { +g_free (ct3d->cxl_dstate.pmem); +error_setg (errp, +"Not enough free space (%zd) required for device (%" PRId64")" , + remaining_size, ct3d->size);Quoted 8 lines
hw/mem/cxl_type3.c line 110
Quoted 5 lines
hw/mem/cxl_type3.c new file +5
-0+ } + +/* Register our subregion as non-volatile */ +memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d), +"cxl_type3-memory" , ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
Quoted 8 lines
"cxl_type3-memory", mr, offset, ct3d->size);I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call
memory_region_set_address()to put it in a particular location.Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.
Quoted 8 lines
Quoted 5 lines
range_invariant: Assertion `range->lob <=range->upb||range->lob==range->upb+ 1' failed.I was digging into this yesterday, but opted to start a new thread on the matter.
Hmm. I think I need to figure out the right
add_subregionafter this and it might work. I'll keep digging, but if you have ideas, let me know.
[snip]
I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.
memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
"cxl_type3-memory", mr, mr->addr + offset, ct3d->size);
memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 104 lines · Ben Widawsky
Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
(quoted diff) +35
-0+ +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) +{ + MemoryRegionSection mrs; + MemoryRegion *mr; + uint64_t offset = 0; + size_t remaining_size; + + if (!ct3d->hostmem) { + error_setg(errp, "memdev property must be set"); + return; + } + + /* FIXME: need to check mr is the host bridge's MR */ + mr = host_memory_backend_get_memory(ct3d->hostmem); + + /* Create our new subregion */ + ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1); + + /* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs = memory_region_find(mr, offset, 1); + while (mrs.mr && mrs.mr != mr) { + offset += memory_region_size(mrs.mr); + mrs = memory_region_find(mr, offset, 1); + } + } + + remaining_size = memory_region_size(mr) - offset; + if (remaining_size < ct3d->size) { + g_free(ct3d->cxl_dstate.pmem); + error_setg(errp, + "Not enough free space (%zd) required for device (%" PRId64 ")", + remaining_size, ct3d->size);Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ } + + /* Register our subregion as non-volatile */ + memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d), + "cxl_type3-memory", ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
Quoted 8 lines
"cxl_type3-memory", mr, offset, ct3d->size);I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call
memory_region_set_address()to put it in a particular location.Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.
Quoted 8 lines
Quoted 5 lines
range_invariant: Assertion `range->lob <=range->upb||range->lob==range->upb+ 1' failed.I was digging into this yesterday, but opted to start a new thread on the matter.
Hmm. I think I need to figure out the right
add_subregionafter this and it might work. I'll keep digging, but if you have ideas, let me know.[snip]
I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.
Quoted 7 lines
memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d), "cxl_type3-memory", mr, mr->addr + offset, ct3d->size); memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true); memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments
Ignore this. It was a problem with my commandline.
I think I have something limping along now.
Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)
Quoted 110 lines · On Thu, 28 Jan 2021 08:58:01 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 109 lines · Ben Widawsky
Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:
A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.
The following example will create a 256M device in a 512M window:
Quoted 8 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +0
-2-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?Answered in a separate thread...
One possible suggestion inline.
Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
(quoted diff) +35
-0+ +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) +{ + MemoryRegionSection mrs; + MemoryRegion *mr; + uint64_t offset = 0; + size_t remaining_size; + + if (!ct3d->hostmem) { + error_setg(errp, "memdev property must be set"); + return; + } + + /* FIXME: need to check mr is the host bridge's MR */ + mr = host_memory_backend_get_memory(ct3d->hostmem); + + /* Create our new subregion */ + ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1); + + /* Find the first free space in the window */ + WITH_RCU_READ_LOCK_GUARD() + { + mrs = memory_region_find(mr, offset, 1); + while (mrs.mr && mrs.mr != mr) { + offset += memory_region_size(mrs.mr); + mrs = memory_region_find(mr, offset, 1); + } + } + + remaining_size = memory_region_size(mr) - offset; + if (remaining_size < ct3d->size) { + g_free(ct3d->cxl_dstate.pmem); + error_setg(errp, + "Not enough free space (%zd) required for device (%" PRId64 ")", + remaining_size, ct3d->size);Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ } + + /* Register our subregion as non-volatile */ + memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d), + "cxl_type3-memory", ct3d->size, errp);this allocates
ct3d->sizeof anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?Quoted 3 lines
It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is
add_subregion()sufficient?Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,
Quoted 8 lines
"cxl_type3-memory", mr, offset, ct3d->size);I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)
Then for the pmem addr, call
memory_region_set_address()to put it in a particular location.Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.
Quoted 8 lines
Quoted 5 lines
range_invariant: Assertion `range->lob <=range->upb||range->lob==range->upb+ 1' failed.I was digging into this yesterday, but opted to start a new thread on the matter.
Hmm. I think I need to figure out the right
add_subregionafter this and it might work. I'll keep digging, but if you have ideas, let me know.[snip]
I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.
memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d), "cxl_type3-memory", mr, mr->addr + offset, ct3d->size); memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true); memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
Quoted 6 lines
Quoted 2 lines
-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments
Ignore this. It was a problem with my commandline.
I think I have something limping along now.
Great. It's going to get more interesting to do this 'right' though.
We'll need to define a memory window similar to device_mem which is used for normal memory hotplug then export that window via the relevant ACPI tables (once defined somewhere public). Finally we'll then want to actually do the memory_region_add_subregion() only on the OS / firmware having configured the type3 device by setting it's PA base.
Going to be fiddly - but should be doable. With it basically working should be a series of sensible (ish) steps to get there.
Jonathan
RFC PATCH v2 25/32 hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
In reply to Ben WidawskyA device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. The generic cxl-utils probably should be the correct place to put this since HDM decoders aren't unique to a type3 device. It is however easier at the moment, and requires less design consideration to simply implement it in the device, and figure out how to consolidate it later.
--- hw/mem/cxl_type3.c | 82 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 77 insertions(+), 5 deletions(-)
hw/mem/cxl_type3.c +77 -5
@@ -57,6 +57,71 @@ static void build_dvsecs(CXLType3Dev *ct3d) REG_LOC_DVSEC_REVID, dvsec); } +static void hdm_decoder_commit (CXLType3Dev *ct3d,int which) +{ +MemoryRegion *pmem = ct3d->cxl_dstate.pmem; +MemoryRegion *mr =host_memory_backend_get_memory (ct3d->hostmem); +Range window, device; +ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; +uint32_t *cache_mem = cregs->cache_mem_registers; +uint64_t offset, size; +Error *err =NULL ; + +assert (which ==0 ); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMIT ,0 ); +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERROR ,0 ); + + offset = ((uint64_t )cache_mem[R_CXL_HDM_DECODER0_BASE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_BASE_LO ]; + size = ((uint64_t )cache_mem[R_CXL_HDM_DECODER0_SIZE_HI ] <<32 ) | + cache_mem[R_CXL_HDM_DECODER0_SIZE_LO ]; + +range_init_nofail (&window, mr->addr,memory_region_size (mr)); +range_init_nofail (&device, offset, size); + +if (!range_contains_range (&window, &device)) { +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERROR ,1 ); +return ; + } + +memory_region_ram_resize (pmem, size, &err); +if (err) { +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,ERROR ,1 ); +return ; + } + + offset -= mr->addr; +memory_region_add_subregion (mr, offset, pmem); + +ARRAY_FIELD_DP32 (cache_mem,CXL_HDM_DECODER0_CTRL ,COMMITTED ,1 ); +} + +static void ct3d_reg_write (void *opaque,hwaddr offset,uint64_t value,unsigned size) +{ +CXLComponentState *cxl_cstate = opaque; +ComponentRegisters *cregs = &cxl_cstate->crb; +CXLType3Dev *ct3d =container_of (cxl_cstate, CXLType3Dev, cxl_cstate); +uint32_t *cache_mem = cregs->cache_mem_registers; +bool should_commit = false; +int which_hdm =-1 ; + +assert (size ==4 ); + +switch (offset) { +case A_CXL_HDM_DECODER0_CTRL : + should_commit =FIELD_EX32 (value,CXL_HDM_DECODER0_CTRL ,COMMIT ); + which_hdm =0 ; +break ; +default : +break ; + } + +stl_le_p ((uint8_t *)cache_mem + offset, value); +if (should_commit) +hdm_decoder_commit (ct3d, which_hdm); +} +static void ct3_instance_init (Object *obj) {/* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
@@ -65,7 +130,10 @@ static void ct3_instance_init(Object *obj)static void ct3_finalize (Object *obj) {CXLType3Dev *ct3d =CT3 (obj); +CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; +ComponentRegisters *regs = &cxl_cstate->crb; +g_free ((void *)regs->special_ops);g_free (ct3d->cxl_dstate.pmem); }
@@ -81,11 +149,12 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)return ; } -/* FIXME: need to check mr is the host bridge's MR */ - mr =host_memory_backend_get_memory (ct3d->hostmem); -/* Create our new subregion */ ct3d->cxl_dstate.pmem =g_new (MemoryRegion,1 ); +memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); + +/* FIXME: need to check mr is the host bridge's MR */ + mr =host_memory_backend_get_memory (ct3d->hostmem);/* Find the first free space in the window */ WITH_RCU_READ_LOCK_GUARD ()
@@ -108,8 +177,6 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)/* Register our subregion as non-volatile */ memory_region_init_ram (ct3d->cxl_dstate.pmem,OBJECT (ct3d),"cxl_type3-memory" , ct3d->size, errp); -memory_region_set_nonvolatile (ct3d->cxl_dstate.pmem, true); -#ifdef SET_PMEM_PADDR memory_region_add_subregion (mr, offset, ct3d->cxl_dstate.pmem);#endif
@@ -148,6 +215,11 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) ct3d->cxl_cstate.pdev = pci_dev;build_dvsecs (ct3d); +#ifndef SET_PMEM_PADDR + regs->special_ops =g_new0 (MemoryRegionOps,1 ); + regs->special_ops->write = ct3d_reg_write; +#endif +cxl_component_register_block_init (OBJECT (pci_dev), cxl_cstate,TYPE_CXL_TYPE3_DEV );
Signature · 1 line
2.30.0
RFC PATCH v2 26/32 acpi/cxl: Add _OSC implementation (9.14.2)
In reply to Ben WidawskyCXL 2.0 specification adds 2 new dwords to the existing _OSC definition from PCIe. The new dwords are accessed with a new uuid. This implementation supports what is in the specification.
We are currently in the process of trying to define a new definition for _OSC. See later work for an explanation.
--- hw/acpi/Kconfig | 5 ++ hw/acpi/cxl.c | 104 ++++++++++++++++++++++++++++++++++++++++++ hw/acpi/meson.build | 1 + hw/i386/acpi-build.c | 12 ++++- include/hw/acpi/cxl.h | 23 ++++++++++ 5 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 hw/acpi/cxl.c create mode 100644 include/hw/acpi/cxl.h
hw/acpi/Kconfig +5 -0
@@ -5,6 +5,7 @@ config ACPI_X86 bool select ACPI select ACPI_NVDIMM + select ACPI_CXL select ACPI_CPU_HOTPLUG select ACPI_MEMORY_HOTPLUG select ACPI_HMAT
@@ -42,3 +43,7 @@ config ACPI_VMGENID depends on PC config ACPI_HW_REDUCED + +config ACPI_CXL + bool + depends on ACPI
hw/acpi/cxl.c new file +104 -0
104 lines not on this page · raw message
hw/acpi/meson.build +1 -0
@@ -10,6 +10,7 @@ acpi_ss.add(when: 'CONFIG_ACPI_CPU_HOTPLUG', if_true: files('cpu_hotplug.c')) acpi_ss.add(when: 'CONFIG_ACPI_MEMORY_HOTPLUG', if_true: files('memory_hotplug.c')) acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_true: files('nvdimm.c')) acpi_ss.add(when: 'CONFIG_ACPI_PCI', if_true: files('pci.c')) +acpi_ss.add(when: 'CONFIG_ACPI_CXL', if_true: files('cxl.c')) acpi_ss.add(when: 'CONFIG_ACPI_VMGENID', if_true: files('vmgenid.c')) acpi_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device.c')) acpi_ss.add(when: 'CONFIG_ACPI_HMAT', if_true: files('hmat.c'))
hw/i386/acpi-build.c +11 -1
@@ -66,6 +66,7 @@#include "hw/acpi/aml-build.h" #include "hw/acpi/utils.h" #include "hw/acpi/pci.h" +#include "hw/acpi/cxl.h" #include "qom/qom-qobject.h" #include "hw/i386/amd_iommu.h"
@@ -1201,11 +1202,20 @@ static void init_pci_acpi(Aml *dev, int uid, int type)if (type ==PCI ) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); - }else { + }else if (type ==PCIE ) {aml_append (dev,aml_name_decl ("_HID" ,aml_eisaid ("PNP0A08" )));aml_append (dev,aml_name_decl ("_CID" ,aml_eisaid ("PNP0A03" )));aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid)));aml_append (dev,build_q35_osc_method ()); + }else /* CXL */ { +struct Aml *pkg =aml_package (2 ); + +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0016" ))); +aml_append (pkg,aml_eisaid ("PNP0A08" )); +aml_append (pkg,aml_eisaid ("PNP0A03" )); +aml_append (dev,aml_name_decl ("_CID" , pkg)); +aml_append (dev,aml_name_decl ("_UID" ,aml_int (uid))); +build_cxl_osc_method (dev); } }
include/hw/acpi/cxl.h new file +23 -0
@@ -0,0 +1,23 @@ +/* +* Copyright (C) 2020 Intel Corporation +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. + +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. + +* You should have received a copy of the GNU General Public License along +* with this program; if not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef HW_ACPI_CXL_H +#define HW_ACPI_CXL_H + +void build_cxl_osc_method (Aml *dev); + +#endif
Signature · 1 line
2.30.0
RFC PATCH v2 27/32 tests/acpi: allow CEDT table addition
In reply to Ben Widawsky--- tests/data/acpi/pc/CEDT | 0 tests/data/acpi/q35/CEDT | 0 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++ 3 files changed, 2 insertions(+) create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT
tests/data/acpi/pc/CEDT new file +0 -0
tests/data/acpi/q35/CEDT new file +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +2 -0
@@ -1 +1,3 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/pc/CEDT" , +"tests/data/acpi/q35/CEDT",
Signature · 1 line
2.30.0
RFC PATCH v2 28/32 acpi/cxl: Create the CEDT (9.14.1)
In reply to Ben WidawskyThe CXL Early Discovery Table is defined in the CXL 2.0 specification as a way for the OS to get CXL specific information from the system firmware.
As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge Structure (CHBS) which is primarily useful for telling the OS exactly where the MMIO for the host bridge is.
v2: Update CHBS to spec released definition
--- hw/acpi/cxl.c | 69 +++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 6 ++- hw/pci-bridge/pci_expander_bridge.c | 21 +-------- include/hw/acpi/cxl.h | 4 ++ include/hw/pci/pci_bridge.h | 25 +++++++++++ 5 files changed, 104 insertions(+), 21 deletions(-)
hw/acpi/cxl.c +69 -0
@@ -18,14 +18,83 @@*/ #include "qemu/osdep.h" +#include "hw/sysbus.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_host.h" #include "hw/cxl/cxl.h" +#include "hw/mem/memory-device.h" #include "hw/acpi/acpi.h" #include "hw/acpi/aml-build.h" #include "hw/acpi/bios-linker-loader.h" #include "hw/acpi/cxl.h" +#include "hw/acpi/cxl.h" #include "qapi/error.h" #include "qemu/uuid.h" +static void cedt_build_chbs (GArray *table_data,PXBDev *cxl) +{ +SysBusDevice *sbd =SYS_BUS_DEVICE (cxl->cxl.cxl_host_bridge); +struct MemoryRegion *mr = sbd->mmio[0 ].memory; + +/* Type */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,1 ); + +/* Record Length */ +build_append_int_noprefix (table_data,32 ,2 ); + +/* UID */ +build_append_int_noprefix (table_data, cxl->uid,4 ); + +/* Version */ +build_append_int_noprefix (table_data,1 ,4 ); + +/* Reserved */ +build_append_int_noprefix (table_data,0 ,4 ); + +/* Base */ +build_append_int_noprefix (table_data, mr->addr,8 ); + +/* Length */ +build_append_int_noprefix (table_data,memory_region_size (mr),8 ); +} + +static int cxl_foreach_pxb_hb (Object *obj,void *opaque) +{ +Aml *cedt = opaque; + +if (object_dynamic_cast (obj,TYPE_PXB_CXL_DEVICE )) { +PXBDev *pxb =PXB_CXL_DEV (obj); + +cedt_build_chbs (cedt->buf, pxb); + } + +return 0 ; +} + +void cxl_build_cedt (GArray *table_offsets,GArray *table_data, +BIOSLinker *linker) +{ +const int cedt_start = table_data->len; +Aml *cedt; + + cedt =init_aml_allocator (); + +/* reserve space for CEDT header */ +acpi_add_table (table_offsets, table_data); +acpi_data_push (cedt->buf,sizeof (AcpiTableHeader)); + +object_child_foreach_recursive (object_get_root (), cxl_foreach_pxb_hb, cedt); + +/* copy AML table into ACPI tables blob and patch header there */ +g_array_append_vals (table_data, cedt->buf->data, cedt->buf->len); +build_header (linker, table_data, (void *)(table_data->data + cedt_start), +"CEDT" , table_data->len - cedt_start,1 ,NULL ,NULL ); +free_aml_allocator (); +} +static Aml *__build_cxl_osc_method (void ) {Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
hw/i386/acpi-build.c +5 -1
@@ -75,6 +75,8 @@#include "hw/acpi/ipmi.h" #include "hw/acpi/hmat.h" +#include "hw/acpi/cxl.h" +/* These are used to size the ACPI tables for -M pc-i440fx-1.7 and * -M pc-i440fx-2.0. Even if the actual amount of AML generated grows * a little bit, there should be plenty of free space since the DSDT
@@ -1371,7 +1373,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, scope =aml_scope ("\\_SB" );if (type ==CXL ) { - dev =aml_device ("CXL%.01X" ,pci_bus_uid (bus)); + dev =aml_device ("CXL%.01X" , uid); }else { dev =aml_device ("PC%.02X" , bus_num); }
@@ -2277,6 +2279,8 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine) machine->nvdimms_state, machine->ram_slots); } +cxl_build_cedt (table_offsets, tables_blob, tables->linker); +acpi_add_table (table_offsets, tables_blob);build_waet (tables_blob, tables->linker);
hw/pci-bridge/pci_expander_bridge.c +1 -20
@@ -57,26 +57,6 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,DECLARE_INSTANCE_CHECKER (PXBDev,PXB_PCIE_DEV , TYPE_PXB_PCIE_DEVICE) -#define TYPE_PXB_CXL_DEVICE "pxb-cxl" -DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , - TYPE_PXB_CXL_DEVICE) - -struct PXBDev { -/*< private >*/ -PCIDevice parent_obj; -/*< public >*/ - -uint8_t bus_nr; -uint16_t numa_node; -int32_t uid; -struct cxl_dev { -HostMemoryBackend *memory_window[CXL_WINDOW_MAX ]; - -uint32_t num_windows; -hwaddr *window_base[CXL_WINDOW_MAX ]; - } cxl; -}; - typedef struct CXLHost { PCIHostState parent_obj;
@@ -351,6 +331,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type, bus =pci_root_bus_new (ds, dev_name,NULL ,NULL ,0 ,TYPE_PXB_CXL_BUS ); bus->flags |=PCI_BUS_CXL ;PXB_CXL_HOST (ds)->dev =PXB_CXL_DEV (dev); +PXB_CXL_DEV (dev)->cxl.cxl_host_bridge = ds; }else { bus =pci_root_bus_new (ds,"pxb-internal" ,NULL ,NULL ,0 ,TYPE_PXB_BUS ); bds =qdev_new ("pci-bridge" );
include/hw/acpi/cxl.h +4 -0
@@ -18,6 +18,10 @@#ifndef HW_ACPI_CXL_H #define HW_ACPI_CXL_H +#include "hw/acpi/bios-linker-loader.h" + +void cxl_build_cedt (GArray *table_offsets,GArray *table_data, +BIOSLinker *linker);void build_cxl_osc_method (Aml *dev);#endif
include/hw/pci/pci_bridge.h +25 -0
@@ -28,6 +28,7 @@#include "hw/pci/pci.h" #include "hw/pci/pci_bus.h" +#include "hw/cxl/cxl.h" #include "qom/object.h" typedef struct PCIBridgeWindows PCIBridgeWindows ;
@@ -81,6 +82,30 @@ struct PCIBridge {#define PCI_BRIDGE_DEV_PROP_MSI "msi"#define PCI_BRIDGE_DEV_PROP_SHPC "shpc" +struct PXBDev { +/*< private >*/ +PCIDevice parent_obj; +/*< public >*/ + +uint8_t bus_nr; +uint16_t numa_node; +int32_t uid; + +struct cxl_dev { +HostMemoryBackend *memory_window[CXL_WINDOW_MAX ]; + +uint32_t num_windows; +hwaddr *window_base[CXL_WINDOW_MAX ]; + +void *cxl_host_bridge;/* Pointer to a CXLHost */ + } cxl; +}; + +typedef struct PXBDev PXBDev ; +#define TYPE_PXB_CXL_DEVICE "pxb-cxl" +DECLARE_INSTANCE_CHECKER (PXBDev,PXB_CXL_DEV , + TYPE_PXB_CXL_DEVICE) +int pci_bridge_ssvid_init (PCIDevice *dev,uint8_t offset,uint16_t svid,uint16_t ssid,Error **errp);
Signature · 1 line
2.30.0
RFC PATCH v2 29/32 Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
In reply to Ben WidawskyThis represents Intel's proposal for how the system firmware can notify Linux that the CEDT exists and provides a driver attach point. It is not in the CXL 2.0 specification as of now.
CXL 2.0 specification adds an _HID, ACPI0016, for CXL capable host bridges, with a _CID of PNP0A08 (PCIe host bridge). CXL aware software is able to use this initiate the proper _OSC method, and get the _UID which is referenced by the CEDT. Therefore the existence of an ACPI0016 device allows a CXL aware driver perform the necessary actions. For a CXL capable OS, this works. For a CXL unaware OS, this works.
The motivation for ACPI0017 is to provide the possibility of having a Linux CXL module that can work on a legacy Linux kernel. Linux core PCI/ACPI which won't be built as a module, will see the _CID of PNP0A08 and bind a driver to it. If we later loaded a driver for ACPI0016, Linux won't be able to bind it to the hardware because it has already bound the PNP0A08 driver. The ACPI0017 device is an opportunity to have an object to bind a driver will be used by a Linux driver to walk the CXL topology and do everything that we would have preferred to do with ACPI0016.
There is another motivation for an ACPI0017 device which isn't implemented here. An operating system needs an attach point for a non-volatile region provider that understands cross-hostbridge interleaving. Since QEMU emulation doesn't support interleaving yet, this is more important on the OS side, for now.
--- hw/i386/acpi-build.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
hw/i386/acpi-build.c +19 -0
@@ -1221,6 +1221,19 @@ static void init_pci_acpi(Aml *dev, int uid, int type) } } +static void build_acpi0017 (Aml *table) +{ +Aml *dev; +Aml *scope; + + scope =aml_scope ("_SB" ); + dev =aml_device ("CXLM" ); +aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0017" ))); + +aml_append (scope, dev); +aml_append (table, scope); +} + static void build_dsdt(GArray *table_data, BIOSLinker *linker, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1237,6 +1250,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,int root_bus_limit =0xFF ;PCIBus *bus =NULL ;TPMIf *tpm =tpm_find (); +bool cxl_present = false;int i;VMBusBridge *vmbus_bridge =vmbus_bridge_find ();
@@ -1392,6 +1406,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,/* Handle the ranges for the PXB expanders */ if (type ==CXL ) { + cxl_present = true;uint64_t base =CXL_HOST_BASE + uid *0x10000 ;crs_range_insert (crs_range_set.mem_ranges, base, base +0x10000 -1 );
@@ -1399,6 +1414,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, } } +if (cxl_present) { +build_acpi0017 (dsdt); + } +/* * At this point crs_range_set has all the ranges used by pci * busses *other* than PCI0. These ranges will be excluded from
Signature · 1 line
2.30.0
RFC PATCH v2 30/32 tests/acpi: Add new CEDT files
In reply to Ben Widawsky--- tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/qtest/bios-tables-test-allowed-diff.h | 2 -- 3 files changed, 2 deletions(-)
tests/data/acpi/pc/CEDT binary +0 -0
tests/data/acpi/q35/CEDT binary +0 -0
tests/qtest/bios-tables-test-allowed-diff.h +0 -2
@@ -1,3 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/pc/CEDT" , -"tests/data/acpi/q35/CEDT",
Signature · 1 line
2.30.0
RFC PATCH v2 31/32 WIP: i386/cxl: Initialize a host bridge
In reply to Ben WidawskyThis patch allows initializing the primary host bridge as a CXL capable hostbridge.
--
This patch is WIP.
--- hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++++++++++++++++++++++++++ hw/i386/acpi-build.c | 8 +++++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 1 + hw/ppc/spapr.c | 2 ++ include/hw/boards.h | 2 ++ include/hw/cxl/cxl.h | 4 ++++ 8 files changed, 44 insertions(+), 1 deletion(-)
hw/arm/virt.c +1 -0
@@ -2481,6 +2481,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) hc->unplug_request = virt_machine_device_unplug_request_cb; hc->unplug = virt_machine_device_unplug_cb; mc->nvdimm_supported = true; + mc->cxl_supported = false; mc->auto_enable_numa_with_memhp = true; mc->auto_enable_numa_with_memdev = true; mc->default_ram_id ="mach-virt.ram" ;
hw/core/machine.c +26 -0
@@ -30,6 +30,7 @@#include "sysemu/qtest.h" #include "hw/pci/pci.h" #include "hw/mem/nvdimm.h" +#include "hw/cxl/cxl.h" #include "migration/global_state.h" #include "migration/vmstate.h"
@@ -502,6 +503,20 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value, nvdimms_state->persistence_string =g_strdup (value); } +static bool machine_get_cxl (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->cxl_devices_state->is_enabled; +} + +static void machine_set_cxl (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->cxl_devices_state->is_enabled = value; +} +void machine_class_allow_dynamic_sysbus_dev (MachineClass *mc,const char *type) {QAPI_LIST_PREPEND (mc->allowed_dynamic_sysbus_devices,g_strdup (type));
@@ -903,6 +918,16 @@ static void machine_initfn(Object *obj) "Valid values are cpu, mem-ctrl"); } +if (mc->cxl_supported) { +Object *obj =OBJECT (ms); + + ms->cxl_devices_state =g_new0 (CXLState,1 ); +object_property_add_bool (obj,"cxl" , machine_get_cxl, machine_set_cxl); +object_property_set_description (obj,"cxl" , +"Set on/off to enable/disable " +"CXL instantiation" ); + } +if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) { ms->numa_state =g_new0 (NumaState,1 ); object_property_add_bool(obj, "hmat",
@@ -939,6 +964,7 @@ static void machine_finalize(Object *obj)g_free (ms->device_memory);g_free (ms->nvdimms_state);g_free (ms->numa_state); +g_free (ms->cxl_devices_state); }bool machine_usb (MachineState *machine)
hw/i386/acpi-build.c +7 -1
@@ -53,6 +53,7 @@#include "sysemu/numa.h" #include "sysemu/reset.h" #include "hw/hyperv/vmbus-bridge.h" +#include "hw/cxl/cxl.h" /* Supported chipsets: */ #include "hw/southbridge/piix.h"
@@ -1277,8 +1278,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,build_piix4_pci0_int (dsdt); }else { sb_scope =aml_scope ("_SB" ); +/* +* XXX: CXL spec calls this "CXL0", but that would require lots of +* changes throughout and so even for CXL enabled, we call it "PCI0" +*/ dev =aml_device ("PCI0" ); -init_pci_acpi (dev,0 ,PCIE ); +init_pci_acpi (dev,0 , + machine->cxl_devices_state->is_enabled ?CXL :PCIE );aml_append (dev,aml_name_decl ("_ADR" ,aml_int (0 )));aml_append (sb_scope, dev);
hw/i386/microvm.c +1 -0
@@ -688,6 +688,7 @@ static void microvm_class_init(ObjectClass *oc, void *data) mc->auto_enable_numa_with_memdev = false; mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = false; + mc->cxl_supported = false; mc->default_ram_id ="microvm.ram" ;/* Avoid relying too much on kernel components */
hw/i386/pc.c +1 -0
@@ -1725,6 +1725,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) hc->unplug = pc_machine_device_unplug_cb; mc->default_cpu_type =TARGET_DEFAULT_CPU_TYPE ; mc->nvdimm_supported = true; + mc->cxl_supported = true; mc->default_ram_id ="pc.ram" ; object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
hw/ppc/spapr.c +2 -0
@@ -4442,6 +4442,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) mc->default_cpu_type =POWERPC_CPU_TYPE_NAME ("power9_v2.0" ); mc->has_hotpluggable_cpus = true; mc->nvdimm_supported = true; + mc->cxl_supported = false; smc->resize_hpt_default =SPAPR_RESIZE_HPT_ENABLED ; fwc->get_dev_path = spapr_get_fw_dev_path; nc->nmi_monitor_handler = spapr_nmi;
@@ -4602,6 +4603,7 @@ static void spapr_machine_4_2_class_options(MachineClass *mc) smc->default_caps.caps[SPAPR_CAP_FWNMI ] =SPAPR_CAP_OFF ; smc->rma_limit =16 * GiB; mc->nvdimm_supported = false; + mc->cxl_supported = false; } DEFINE_SPAPR_MACHINE(4_2, "4.2", false);
include/hw/boards.h +2 -0
@@ -206,6 +206,7 @@ struct MachineClass {bool ignore_boot_device_suffixes;bool smbus_no_migration_support;bool nvdimm_supported; +bool cxl_supported;bool numa_mem_supported;bool auto_enable_numa;const char *default_ram_id;
@@ -292,6 +293,7 @@ struct MachineState {CPUArchIdList *possible_cpus;CpuTopology smp;struct NVDIMMState *nvdimms_state; +struct CXLState *cxl_devices_state;struct NumaState *numa_state; };
include/hw/cxl/cxl.h +4 -0
@@ -22,5 +22,9 @@#define CXL_HOST_BASE 0xD0000000#define CXL_WINDOW_MAX 10 +typedef struct CXLState { +bool is_enabled; +}CXLState ; +#endif
Signature · 1 line
2.30.0
RFC PATCH v2 32/32 qtest/cxl: Add very basic sanity tests
In reply to Ben Widawsky--- tests/qtest/cxl-test.c | 93 +++++++++++++++++++++++++++++++++++++++++ tests/qtest/meson.build | 4 ++ 2 files changed, 97 insertions(+) create mode 100644 tests/qtest/cxl-test.c
tests/qtest/cxl-test.c new file +93 -0
@@ -0,0 +1,93 @@ +/* +* QTest testcase for CXL +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#include "qemu/osdep.h" +#include "libqtest-single.h" + +#define QEMU_PXB_CMD "-machine q35 -object memory-backend-file,id=cxl-mem1," \ + "share,mem-path=%s,size=512M " \ + "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0," \ + "len-window-base=1,window-base[0]=0x4c0000000,memdev[0]=cxl-mem1" +#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0" + +#define QEMU_T3D "-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M" + +static void cxl_basic_hb (void ) +{ +qtest_start ("-machine q35,cxl" ); +qtest_end (); +} + +static void cxl_basic_pxb (void ) +{ +qtest_start ("-machine q35 -device pxb-cxl,bus=pcie.0,uid=0" ); +qtest_end (); +} + +static void cxl_pxb_with_window (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); +g_string_printf (cmdline,QEMU_PXB_CMD , tmpfs); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_root_port (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); +g_string_printf (cmdline,QEMU_PXB_CMD " %s" , tmpfs,QEMU_RP ); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +static void cxl_t3d (void ) +{ +GString *cmdline; +char template[] ="/tmp/cxl-test-XXXXXX" ; +const char *tmpfs; + + tmpfs =mkdtemp (template); + + cmdline =g_string_new (NULL ); +g_string_printf (cmdline,QEMU_PXB_CMD " %s %s" , tmpfs,QEMU_RP ,QEMU_T3D ); + +qtest_start (cmdline->str); +qtest_end (); + +g_string_free (cmdline, TRUE); +} + +int main (int argc,char **argv) +{ +g_test_init (&argc, &argv,NULL ); + +qtest_add_func ("/pci/cxl/basic_hostbridge" , cxl_basic_hb); +qtest_add_func ("/pci/cxl/basic_pxb" , cxl_basic_pxb); +qtest_add_func ("/pci/cxl/pxb_with_window" , cxl_pxb_with_window); +qtest_add_func ("/pci/cxl/root_port" , cxl_root_port); +qtest_add_func ("/pci/cxl/type3_device" , cxl_t3d); + +return g_test_run (); +}
tests/qtest/meson.build +4 -0
@@ -22,6 +22,9 @@ qtests_pci = \ (config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : []) + \ (config_all_devices.has_key('CONFIG_IVSHMEM_DEVICE') ? ['ivshmem-test'] : []) +qtests_cxl = \ + (config_all_devices.has_key('CONFIG_CXL') ? ['cxl-test'] : []) + qtests_i386 = \ (slirp.found() ? ['pxe-test', 'test-netfilter'] : []) + \ (config_host.has_key('CONFIG_POSIX') ? ['test-filter-mirror'] : []) + \
@@ -47,6 +50,7 @@ qtests_i386 = \ (config_all_devices.has_key('CONFIG_TPM_TIS_ISA') ? ['tpm-tis-swtpm-test'] : []) + \ (config_all_devices.has_key('CONFIG_RTL8139_PCI') ? ['rtl8139-test'] : []) + \ qtests_pci + \ + qtests_cxl + \ ['fdc-test', 'ide-test', 'hd-geo-test',
Signature · 1 line
2.30.0
Quoted 20 lines · On Tue, 5 Jan 2021 08:52:51 -0800 Ben Widawsky <[email protected]> wrote:
Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbeddevice_reg_init_common()(Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support
Quoted 4 lines
(There are some new patches that I just named 'v2' for simplicity's sake)
Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).
Hi Ben,
In interests of avoid duplication, I thought I'd give a quick update of where I am. I have what is here working on arm64 (enabling is fairly simple). I'll have a bit more feedback next week and hopefully have time for an indepth review.
I did run into some issues around alignment for the persistent memory that (I think) are down to the fact the memory-backend isn't quite tied up to the device.
Flows wise, I initially started hacking in NFIT building changes in qemu but I'm not yet sure how the hotplug flows are supposed to work and who is responsible for setting up a hotplugged device (OS or firmware).
Plans wise, I was thinking I could look at DOE emulation as seems that we'll be needing that fairly soon to get any useful memory usecases up and running. Let me know if you already have that underway.
Jonathan
Quoted 193 lines
The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.
Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).
We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMsAs mentioned above there are three benefits to carrying this enabling in upstream QEMU:
1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.
2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (
nfit_test). In retrospect a QEMU emulation ofnfit_testcapabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?
There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacityThe flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.
The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb_UIDthrough host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add_OSCimplementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity testsJonathan Cameron (1):
Temp: Add thePCI_EXT_ID_DVSECdefinition to the qemupci_regs.hcopy.MAINTAINERS | 6 + hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl.c | 173 ++++++++++ hw/acpi/meson.build | 1 + hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++ hw/core/numa.c | 3 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 193 +++++++++++ hw/cxl/cxl-device-utils.c | 278 ++++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 386 ++++++++++++++++++++++ hw/cxl/meson.build | 5 + hw/i386/acpi-build.c | 87 ++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 2 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 334 +++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 32 +- hw/pci/pcie.c | 30 ++ hw/ppc/spapr.c | 2 + include/hw/acpi/cxl.h | 27 ++ include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 30 ++ include/hw/cxl/cxl_component.h | 187 +++++++++++ include/hw/cxl/cxl_device.h | 210 ++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 15 + include/hw/pci/pci_bridge.h | 25 ++ include/hw/pci/pci_bus.h | 8 + include/hw/pci/pci_ids.h | 1 + include/standard-headers/linux/pci_regs.h | 1 + monitor/hmp-cmds.c | 15 + qapi/machine.json | 1 + tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/pc/DSDT | Bin 5065 -> 5065 bytes tests/data/acpi/pc/DSDT.acpihmat | Bin 6390 -> 6390 bytes tests/data/acpi/pc/DSDT.bridge | Bin 6924 -> 6924 bytes tests/data/acpi/pc/DSDT.cphp | Bin 5529 -> 5529 bytes tests/data/acpi/pc/DSDT.dimmpxm | Bin 6719 -> 6719 bytes tests/data/acpi/pc/DSDT.hpbridge | Bin 5026 -> 5026 bytes tests/data/acpi/pc/DSDT.hpbrroot | Bin 3084 -> 3084 bytes tests/data/acpi/pc/DSDT.ipmikcs | Bin 5137 -> 5137 bytes tests/data/acpi/pc/DSDT.memhp | Bin 6424 -> 6424 bytes tests/data/acpi/pc/DSDT.numamem | Bin 5071 -> 5071 bytes tests/data/acpi/pc/DSDT.roothp | Bin 5261 -> 5261 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/DSDT | Bin 7801 -> 7801 bytes tests/data/acpi/q35/DSDT.acpihmat | Bin 9126 -> 9126 bytes tests/data/acpi/q35/DSDT.bridge | Bin 7819 -> 7819 bytes tests/data/acpi/q35/DSDT.cphp | Bin 8265 -> 8265 bytes tests/data/acpi/q35/DSDT.dimmpxm | Bin 9455 -> 9455 bytes tests/data/acpi/q35/DSDT.ipmibt | Bin 7876 -> 7876 bytes tests/data/acpi/q35/DSDT.memhp | Bin 9160 -> 9160 bytes tests/data/acpi/q35/DSDT.mmio64 | Bin 8932 -> 8932 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7807 -> 7807 bytes tests/qtest/cxl-test.c | 93 ++++++ tests/qtest/meson.build | 4 + 65 files changed, 2777 insertions(+), 30 deletions(-) create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT create mode 100644 tests/qtest/cxl-test.c
Quoted 25 lines · Jonathan Cameron
Quoted 22 lines · On Tue, 5 Jan 2021 08:52:51 -0800 Ben Widawsky <[email protected]> wrote:
Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbeddevice_reg_init_common()(Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support(There are some new patches that I just named 'v2' for simplicity's sake)
Quoted 8 lines
Quoted 2 lines
Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).Hi Ben,
In interests of avoid duplication, I thought I'd give a quick update of where I am. I have what is here working on arm64 (enabling is fairly simple). I'll have a bit more feedback next week and hopefully have time for an indepth review.
Thanks for this. Dan recommended I send out an email with plans for what's needed, what we intend to work on and so on. I'll try to get that out next week also.
Quoted 4 lines
I did run into some issues around alignment for the persistent memory that (I think) are down to the fact the memory-backend isn't quite tied up to the device.
I did have plans to change this to support interleaving.
Quoted 4 lines
Flows wise, I initially started hacking in NFIT building changes in qemu but I'm not yet sure how the hotplug flows are supposed to work and who is responsible for setting up a hotplugged device (OS or firmware).
It won't be NFIT :-). I'd redirect this question to the linux-cxl mailing list, as it's a good one. The spec is somewhat handwavy around this.
Quoted 7 lines
Plans wise, I was thinking I could look at DOE emulation as seems that we'll be needing that fairly soon to get any useful memory usecases up and running. Let me know if you already have that underway.
Jonathan
Chris in the Cc had some plans to enable DOE for their purposes. It'd be good to sync with him. AFAIK though, nobody has plans to enable DOE in Linux, which might be a good target next.
Quoted 194 lines
Quoted 193 lines
The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.
Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).
We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMsAs mentioned above there are three benefits to carrying this enabling in upstream QEMU:
1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.
2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (
nfit_test). In retrospect a QEMU emulation ofnfit_testcapabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?
There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacityThe flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.
The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb_UIDthrough host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add_OSCimplementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity testsJonathan Cameron (1):
Temp: Add thePCI_EXT_ID_DVSECdefinition to the qemupci_regs.hcopy.MAINTAINERS | 6 + hw/Kconfig | 1 + hw/acpi/Kconfig | 5 + hw/acpi/cxl.c | 173 ++++++++++ hw/acpi/meson.build | 1 + hw/arm/virt.c | 1 + hw/core/machine.c | 26 ++ hw/core/numa.c | 3 + hw/cxl/Kconfig | 3 + hw/cxl/cxl-component-utils.c | 193 +++++++++++ hw/cxl/cxl-device-utils.c | 278 ++++++++++++++++ hw/cxl/cxl-mailbox-utils.c | 386 ++++++++++++++++++++++ hw/cxl/meson.build | 5 + hw/i386/acpi-build.c | 87 ++++- hw/i386/microvm.c | 1 + hw/i386/pc.c | 2 + hw/mem/Kconfig | 5 + hw/mem/cxl_type3.c | 334 +++++++++++++++++++ hw/mem/meson.build | 1 + hw/meson.build | 1 + hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++ hw/pci-bridge/meson.build | 1 + hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++- hw/pci-bridge/pcie_root_port.c | 6 +- hw/pci/pci.c | 32 +- hw/pci/pcie.c | 30 ++ hw/ppc/spapr.c | 2 + include/hw/acpi/cxl.h | 27 ++ include/hw/boards.h | 2 + include/hw/cxl/cxl.h | 30 ++ include/hw/cxl/cxl_component.h | 187 +++++++++++ include/hw/cxl/cxl_device.h | 210 ++++++++++++ include/hw/cxl/cxl_pci.h | 160 +++++++++ include/hw/pci/pci.h | 15 + include/hw/pci/pci_bridge.h | 25 ++ include/hw/pci/pci_bus.h | 8 + include/hw/pci/pci_ids.h | 1 + include/standard-headers/linux/pci_regs.h | 1 + monitor/hmp-cmds.c | 15 + qapi/machine.json | 1 + tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes tests/data/acpi/pc/DSDT | Bin 5065 -> 5065 bytes tests/data/acpi/pc/DSDT.acpihmat | Bin 6390 -> 6390 bytes tests/data/acpi/pc/DSDT.bridge | Bin 6924 -> 6924 bytes tests/data/acpi/pc/DSDT.cphp | Bin 5529 -> 5529 bytes tests/data/acpi/pc/DSDT.dimmpxm | Bin 6719 -> 6719 bytes tests/data/acpi/pc/DSDT.hpbridge | Bin 5026 -> 5026 bytes tests/data/acpi/pc/DSDT.hpbrroot | Bin 3084 -> 3084 bytes tests/data/acpi/pc/DSDT.ipmikcs | Bin 5137 -> 5137 bytes tests/data/acpi/pc/DSDT.memhp | Bin 6424 -> 6424 bytes tests/data/acpi/pc/DSDT.numamem | Bin 5071 -> 5071 bytes tests/data/acpi/pc/DSDT.roothp | Bin 5261 -> 5261 bytes tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes tests/data/acpi/q35/DSDT | Bin 7801 -> 7801 bytes tests/data/acpi/q35/DSDT.acpihmat | Bin 9126 -> 9126 bytes tests/data/acpi/q35/DSDT.bridge | Bin 7819 -> 7819 bytes tests/data/acpi/q35/DSDT.cphp | Bin 8265 -> 8265 bytes tests/data/acpi/q35/DSDT.dimmpxm | Bin 9455 -> 9455 bytes tests/data/acpi/q35/DSDT.ipmibt | Bin 7876 -> 7876 bytes tests/data/acpi/q35/DSDT.memhp | Bin 9160 -> 9160 bytes tests/data/acpi/q35/DSDT.mmio64 | Bin 8932 -> 8932 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7807 -> 7807 bytes tests/qtest/cxl-test.c | 93 ++++++ tests/qtest/meson.build | 4 + 65 files changed, 2777 insertions(+), 30 deletions(-) create mode 100644 hw/acpi/cxl.c create mode 100644 hw/cxl/Kconfig create mode 100644 hw/cxl/cxl-component-utils.c create mode 100644 hw/cxl/cxl-device-utils.c create mode 100644 hw/cxl/cxl-mailbox-utils.c create mode 100644 hw/cxl/meson.build create mode 100644 hw/mem/cxl_type3.c create mode 100644 hw/pci-bridge/cxl_root_port.c create mode 100644 include/hw/acpi/cxl.h create mode 100644 include/hw/cxl/cxl.h create mode 100644 include/hw/cxl/cxl_component.h create mode 100644 include/hw/cxl/cxl_device.h create mode 100644 include/hw/cxl/cxl_pci.h create mode 100644 tests/data/acpi/pc/CEDT create mode 100644 tests/data/acpi/q35/CEDT create mode 100644 tests/qtest/cxl-test.c