Skip to contentSkip to conversation

CXL 2.0 Support

RFC PATCH Superseded Ben Widawsky 4 participants 57 messages Jan 5, 2021 – Jan 28, 2021 Arm/Amlogic Meson SoC support

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover CXL 2.0 Support

Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbed device_reg_init_common() (Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)

New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support

(There are some new patches that I just named 'v2' for simplicity's sake)

Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).

The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.

Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).

CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).

We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs

As mentioned above there are three benefits to carrying this enabling in upstream QEMU:

1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.

2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (nfit_test). In retrospect a QEMU emulation of nfit_test capabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.

3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?

There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity

The flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.

The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case

[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/

Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb _UID through host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests

Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.

 MAINTAINERS                               |   6 +
 hw/Kconfig                                |   1 +
 hw/acpi/Kconfig                           |   5 +
 hw/acpi/cxl.c                             | 173 ++++++++++
 hw/acpi/meson.build                       |   1 +
 hw/arm/virt.c                             |   1 +
 hw/core/machine.c                         |  26 ++
 hw/core/numa.c                            |   3 +
 hw/cxl/Kconfig                            |   3 +
 hw/cxl/cxl-component-utils.c              | 193 +++++++++++
 hw/cxl/cxl-device-utils.c                 | 278 ++++++++++++++++
 hw/cxl/cxl-mailbox-utils.c                | 386 ++++++++++++++++++++++
 hw/cxl/meson.build                        |   5 +
 hw/i386/acpi-build.c                      |  87 ++++-
 hw/i386/microvm.c                         |   1 +
 hw/i386/pc.c                              |   2 +
 hw/mem/Kconfig                            |   5 +
 hw/mem/cxl_type3.c                        | 334 +++++++++++++++++++
 hw/mem/meson.build                        |   1 +
 hw/meson.build                            |   1 +
 hw/pci-bridge/Kconfig                     |   5 +
 hw/pci-bridge/cxl_root_port.c             | 231 +++++++++++++
 hw/pci-bridge/meson.build                 |   1 +
 hw/pci-bridge/pci_expander_bridge.c       | 209 +++++++++++-
 hw/pci-bridge/pcie_root_port.c            |   6 +-
 hw/pci/pci.c                              |  32 +-
 hw/pci/pcie.c                             |  30 ++
 hw/ppc/spapr.c                            |   2 +
 include/hw/acpi/cxl.h                     |  27 ++
 include/hw/boards.h                       |   2 +
 include/hw/cxl/cxl.h                      |  30 ++
 include/hw/cxl/cxl_component.h            | 187 +++++++++++
 include/hw/cxl/cxl_device.h               | 210 ++++++++++++
 include/hw/cxl/cxl_pci.h                  | 160 +++++++++
 include/hw/pci/pci.h                      |  15 +
 include/hw/pci/pci_bridge.h               |  25 ++
 include/hw/pci/pci_bus.h                  |   8 +
 include/hw/pci/pci_ids.h                  |   1 +
 include/standard-headers/linux/pci_regs.h |   1 +
 monitor/hmp-cmds.c                        |  15 +
 qapi/machine.json                         |   1 +
 tests/data/acpi/pc/CEDT                   | Bin 0 -> 36 bytes
 tests/data/acpi/pc/DSDT                   | Bin 5065 -> 5065 bytes
 tests/data/acpi/pc/DSDT.acpihmat          | Bin 6390 -> 6390 bytes
 tests/data/acpi/pc/DSDT.bridge            | Bin 6924 -> 6924 bytes
 tests/data/acpi/pc/DSDT.cphp              | Bin 5529 -> 5529 bytes
 tests/data/acpi/pc/DSDT.dimmpxm           | Bin 6719 -> 6719 bytes
 tests/data/acpi/pc/DSDT.hpbridge          | Bin 5026 -> 5026 bytes
 tests/data/acpi/pc/DSDT.hpbrroot          | Bin 3084 -> 3084 bytes
 tests/data/acpi/pc/DSDT.ipmikcs           | Bin 5137 -> 5137 bytes
 tests/data/acpi/pc/DSDT.memhp             | Bin 6424 -> 6424 bytes
 tests/data/acpi/pc/DSDT.numamem           | Bin 5071 -> 5071 bytes
 tests/data/acpi/pc/DSDT.roothp            | Bin 5261 -> 5261 bytes
 tests/data/acpi/q35/CEDT                  | Bin 0 -> 36 bytes
 tests/data/acpi/q35/DSDT                  | Bin 7801 -> 7801 bytes
 tests/data/acpi/q35/DSDT.acpihmat         | Bin 9126 -> 9126 bytes
 tests/data/acpi/q35/DSDT.bridge           | Bin 7819 -> 7819 bytes
 tests/data/acpi/q35/DSDT.cphp             | Bin 8265 -> 8265 bytes
 tests/data/acpi/q35/DSDT.dimmpxm          | Bin 9455 -> 9455 bytes
 tests/data/acpi/q35/DSDT.ipmibt           | Bin 7876 -> 7876 bytes
 tests/data/acpi/q35/DSDT.memhp            | Bin 9160 -> 9160 bytes
 tests/data/acpi/q35/DSDT.mmio64           | Bin 8932 -> 8932 bytes
 tests/data/acpi/q35/DSDT.numamem          | Bin 7807 -> 7807 bytes
 tests/qtest/cxl-test.c                    |  93 ++++++
 tests/qtest/meson.build                   |   4 +
 65 files changed, 2777 insertions(+), 30 deletions(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/cxl-device-utils.c
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 hw/mem/cxl_type3.c
 create mode 100644 hw/pci-bridge/cxl_root_port.c
 create mode 100644 include/hw/acpi/cxl.h
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_device.h
 create mode 100644 include/hw/cxl/cxl_pci.h
 create mode 100644 tests/data/acpi/pc/CEDT
 create mode 100644 tests/data/acpi/q35/CEDT
 create mode 100644 tests/qtest/cxl-test.c
Signature · 1 line
2.30.0
#1 lore raw

RFC PATCH v2 1/32 Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.

From: Jonathan Cameron <[email protected]>

This hasn't yet been added to the linux kernel tree, so for purposes of this RFC just add it locally.

---
 include/standard-headers/linux/pci_regs.h | 1 +
 1 file changed, 1 insertion(+)
include/standard-headers/linux/pci_regs.h +1 -0
index a95d55f9f2..5d0b79b9da 100644
@@ -723,6 +723,7 @@
 #define PCI_EXT_CAP_ID_DPC	0x1D	/* Downstream Port Containment */
 #define PCI_EXT_CAP_ID_L1SS	0x1E	/* L1 PM Substates */
 #define PCI_EXT_CAP_ID_PTM	0x1F	/* Precision Time Measurement */
+#define PCI_EXT_CAP_ID_DVSEC	0x23    /* Designated Vendor-Specific */
 #define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
 #define PCI_EXT_CAP_ID_PL_16GT	0x26	/* Physical Layer 16.0 GT/s */
 #define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL_16GT
Signature · 1 line
2.30.0
#2 lore raw

RFC PATCH v2 2/32 hw/pci/cxl: Add a CXL component type (interface)

In reply to Ben Widawsky

A CXL component is a hardware entity that implements CXL component registers from the CXL 2.0 spec (8.2.3). Currently these represent 3 general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)

A CXL component can be conceptually thought of as a PCIe device with extra functionality when enumerated and enabled. For this reason, CXL does here, and will continue to add on to existing PCI code paths.

Host bridges will typically need to be handled specially and so they can implement this newly introduced interface or not. All other components should implement this interface. Implementing this interface allows the core pci code to treat these devices as special where appropriate.

---
 hw/pci/pci.c         | 10 ++++++++++
 include/hw/pci/pci.h |  8 ++++++++
 2 files changed, 18 insertions(+)
hw/pci/pci.c +10 -0
index a6b0c5602e..50fc61c81f 100644
@@ -194,6 +194,11 @@ static const TypeInfo pci_bus_info = {
     .class_init = pci_bus_class_init,
 };
 
+static const TypeInfo cxl_interface_info = {
+    .name          = INTERFACE_CXL_DEVICE,
+    .parent        = TYPE_INTERFACE,
+};
+
 static const TypeInfo pcie_interface_info = {
     .name          = INTERFACE_PCIE_DEVICE,
     .parent        = TYPE_INTERFACE,
@@ -2115,6 +2120,10 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
         pci_dev->cap_present |= QEMU_PCI_CAP_EXPRESS;
     }
 
+    if (object_class_dynamic_cast(klass, INTERFACE_CXL_DEVICE)) {
+        pci_dev->cap_present |= QEMU_PCIE_CAP_CXL;
+    }
+
     pci_dev = do_pci_register_device(pci_dev,
                                      object_get_typename(OBJECT(qdev)),
                                      pci_dev->devfn, errp);
@@ -2841,6 +2850,7 @@ static void pci_register_types(void)
     type_register_static(&pci_bus_info);
     type_register_static(&pcie_bus_info);
     type_register_static(&conventional_pci_interface_info);
+    type_register_static(&cxl_interface_info);
     type_register_static(&pcie_interface_info);
     type_register_static(&pci_device_type_info);
 }
include/hw/pci/pci.h +8 -0
index 259f9c992d..f6b1eff446 100644
@@ -194,6 +194,8 @@ enum {
     QEMU_PCIE_LNKSTA_DLLLA = (1 << QEMU_PCIE_LNKSTA_DLLLA_BITNR),
 #define QEMU_PCIE_EXTCAP_INIT_BITNR 9
     QEMU_PCIE_EXTCAP_INIT = (1 << QEMU_PCIE_EXTCAP_INIT_BITNR),
+#define QEMU_PCIE_CXL_BITNR 10
+    QEMU_PCIE_CAP_CXL = (1 << QEMU_PCIE_CXL_BITNR),
 };
 
 #define TYPE_PCI_DEVICE "pci-device"
@@ -201,6 +203,12 @@ typedef struct PCIDeviceClass PCIDeviceClass;
 DECLARE_OBJ_CHECKERS(PCIDevice, PCIDeviceClass,
                      PCI_DEVICE, TYPE_PCI_DEVICE)
 
+/*
+ * Implemented by devices that can be plugged on CXL buses. In the spec, this is
+ * actually a "CXL Component, but we name it device to match the PCI naming.
+ */
+#define INTERFACE_CXL_DEVICE "cxl-device"
+
 /* Implemented by devices that can be plugged on PCI Express buses */
 #define INTERFACE_PCIE_DEVICE "pci-express-device"
 
Signature · 1 line
2.30.0
#3 lore raw

RFC PATCH v2 3/32 hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)

In reply to Ben Widawsky

A CXL 2.0 component is any entity in the CXL topology. All components have a analogous function in PCIe. Except for the CXL host bridge, all have a PCIe config space that is accessible via the common PCIe mechanisms. CXL components are enumerated via DVSEC fields in the extended PCIe header space. CXL components will minimally implement some subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL 2.0 specification. Two headers and a utility library are introduced to support the minimum functionality needed to enumerate components.

The cxl_pci header manages bits associated with PCI, specifically the DVSEC and related fields. The cxl_component.h variant has data structures and APIs that are useful for drivers implementing any of the CXL 2.0 components. The library takes care of making use of the DVSEC bits and the CXL.[mem|cache] regisetrs.

None of the mechanisms required to enumerate a CXL capable hostbridge are introduced at this point.

Note that the CXL.mem and CXL.cache registers used are always 4B wide. It's possible in the future that this constraint will not hold.

v2: Improve assertions in DVSEC creation (Jonathan)
Use 'n' for HDM register offsets (Jonathan) Correct revision ID for extensions (Jonathan) Minor cleanups and clarifications (Jonathan)

---
 MAINTAINERS                    |   6 +
 hw/Kconfig                     |   1 +
 hw/cxl/Kconfig                 |   3 +
 hw/cxl/cxl-component-utils.c   | 193 +++++++++++++++++++++++++++++++++
 hw/cxl/meson.build             |   3 +
 hw/meson.build                 |   1 +
 include/hw/cxl/cxl.h           |  17 +++
 include/hw/cxl/cxl_component.h | 187 ++++++++++++++++++++++++++++++++
 include/hw/cxl/cxl_pci.h       | 138 +++++++++++++++++++++++
 9 files changed, 549 insertions(+)
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_pci.h
MAINTAINERS +6 -0
index 42fedf91e7..1f5fc033b4 100644
@@ -2225,6 +2225,12 @@ F: qapi/block*.json
 F: qapi/transaction.json
 T: git https://repo.or.cz/qemu/armbru.git block-next
 
+Compute Express Link
+M: Ben Widawsky <[email protected]>
+S: Supported
+F: hw/cxl/
+F: include/hw/cxl/
+
 Dirty Bitmaps
 M: Eric Blake <[email protected]>
 M: Vladimir Sementsov-Ogievskiy <[email protected]>
hw/Kconfig +1 -0
index 5ad3c6b5a4..c03650c5ed 100644
@@ -6,6 +6,7 @@ source audio/Kconfig
 source block/Kconfig
 source char/Kconfig
 source core/Kconfig
+source cxl/Kconfig
 source display/Kconfig
 source dma/Kconfig
 source gpio/Kconfig
hw/cxl/Kconfig new file +3 -0
new file mode 100644
index 0000000000..8e67519b16
@@ -0,0 +1,3 @@
+config CXL
+    bool
+    default y if PCI_EXPRESS
hw/cxl/cxl-component-utils.c new file +193 -0

193 lines not on this page · raw message

hw/cxl/meson.build new file +3 -0
new file mode 100644
index 0000000000..00c3876a0f
@@ -0,0 +1,3 @@
+softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
+  'cxl-component-utils.c',
+))
hw/meson.build +1 -0
index 010de7219c..3e440c341a 100644
@@ -6,6 +6,7 @@ subdir('block')
 subdir('char')
 subdir('core')
 subdir('cpu')
+subdir('cxl')
 subdir('display')
 subdir('dma')
 subdir('gpio')
include/hw/cxl/cxl.h new file +17 -0
new file mode 100644
index 0000000000..55f6cc30a5
@@ -0,0 +1,17 @@
+/*
+ * QEMU CXL Support
+ *
+ * Copyright (c) 2020 Intel
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef CXL_H
+#define CXL_H
+
+#include "cxl_pci.h"
+#include "cxl_component.h"
+
+#endif
+
include/hw/cxl/cxl_component.h new file +187 -0

187 lines not on this page · raw message

include/hw/cxl/cxl_pci.h new file +138 -0

138 lines not on this page · raw message

Signature · 1 line
2.30.0
#4 lore raw

RFC PATCH v2 4/32 hw/cxl/device: Introduce a CXL device (8.2.8)

In reply to Ben Widawsky

A CXL device is a type of CXL component. Conceptually, a CXL device would be a leaf node in a CXL topology. From an emulation perspective, CXL devices are the most complex and so the actual implementation is reserved for discrete commits.

This new device type is specifically catered towards the eventually implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0 specification.

v2: Remove error codes not yet used (Jonathan)
Fix interrupt enable bit width (Jonathan) Add comment for weird register size (Jonathan) Remove commands/registers not yet used (Ben)

---
 include/hw/cxl/cxl.h        |   1 +
 include/hw/cxl/cxl_device.h | 151 ++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
 create mode 100644 include/hw/cxl/cxl_device.h
include/hw/cxl/cxl.h +1 -0
index 55f6cc30a5..23f52c4cf9 100644
@@ -12,6 +12,7 @@
 
 #include "cxl_pci.h"
 #include "cxl_component.h"
+#include "cxl_device.h"
 
 #endif
 
include/hw/cxl/cxl_device.h new file +151 -0

151 lines not on this page · raw message

Signature · 1 line
2.30.0
#5 lore raw

RFC PATCH v2 5/32 hw/cxl/device: Implement the CAP array (8.2.8.1-2)

In reply to Ben Widawsky

This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.

v2: Break out register alignment checks (Jonathan)

---
 hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build        |  1 +
 2 files changed, 73 insertions(+)
 create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +72 -0
new file mode 100644
index 0000000000..d1b1371e66
@@ -0,0 +1,72 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+static int cxl_device_check_register_alignment(hwaddr offset, unsigned size)
+{
+    if (unlikely(offset & (size - 1))) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
+}
+
+static const MemoryRegionOps caps_ops = {
+    .read = caps_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+    /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
+    memory_region_init(
+        &cxl_dstate->device_registers, obj, "device-registers",
+        pow2ceil(CXL_MAILBOX_REGISTERS_LENGTH + CXL_MAILBOX_REGISTERS_OFFSET));
+
+    memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+                          "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+
+    memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+                                &cxl_dstate->caps);
+}
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+    uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
+    const int cap_count = 0;
+
+    /* CXL Device Capabilities Array Register */
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+}
hw/cxl/meson.build +1 -0
index 00c3876a0f..47154d6850 100644
@@ -1,3 +1,4 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
+  'cxl-device-utils.c',
 ))
Signature · 1 line
2.30.0
#6 lore raw

Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)

Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:

This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.

v2: Break out register alignment checks (Jonathan)

Hi Ben,

One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.

Will get to a full review after messing around ('testing') this a bit more ;)

Jonathan

Quoted 56 lines
---
 hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build        |  1 +
 2 files changed, 73 insertions(+)
 create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +44 -0
new file mode 100644
index 0000000000..d1b1371e66
@@ -0,0 +1,72 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+static int cxl_device_check_register_alignment(hwaddr offset, unsigned size)
+{
+    if (unlikely(offset & (size - 1))) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
+}
+
+static const MemoryRegionOps caps_ops = {
+    .read = caps_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,

hw/cxl/cxl-device-utils.c line 54

Quoted 10 lines
hw/cxl/cxl-device-utils.c new file +10 -0
+        .max_access_size = 8,
+    },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+    /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
+    memory_region_init(
+        &cxl_dstate->device_registers, obj, "device-registers",
+        pow2ceil(CXL_MAILBOX_REGISTERS_LENGTH + CXL_MAILBOX_REGISTERS_OFFSET));

I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).

Quoted 27 lines
hw/cxl/cxl-device-utils.c new file +18 -0
+
+    memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+                          "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+
+    memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+                                &cxl_dstate->caps);
+}
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+    uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
+    const int cap_count = 0;
+
+    /* CXL Device Capabilities Array Register */
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+}
hw/cxl/meson.build +1 -0
index 00c3876a0f..47154d6850 100644
@@ -1,3 +1,4 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
+  'cxl-device-utils.c',
 ))
#7 lore raw

Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)

Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:

This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.

v2: Break out register alignment checks (Jonathan)

Hi Ben,

One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.

Will get to a full review after messing around ('testing') this a bit more ;)

Jonathan

Quoted 63 lines
---
 hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build        |  1 +
 2 files changed, 73 insertions(+)
 create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +51 -0
new file mode 100644
index 0000000000..d1b1371e66
@@ -0,0 +1,72 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+static int cxl_device_check_register_alignment(hwaddr offset, unsigned size)
+{
+    if (unlikely(offset & (size - 1))) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
+}
+
+static const MemoryRegionOps caps_ops = {
+    .read = caps_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+    /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
Quoted 10 lines

hw/cxl/cxl-device-utils.c line 54

Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3 -0
+    memory_region_init(
+        &cxl_dstate->device_registers, obj, "device-registers",
+        pow2ceil(CXL_MAILBOX_REGISTERS_LENGTH + CXL_MAILBOX_REGISTERS_OFFSET));

I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).

Maybe I misunderstand, but this is the intended behavior. cxl_dstate->device_registers is the MemoryRegion container for all the subregions that are the actual device MMIO.

                     +------------------+
                  ^  |                  |
                  |  |      unused      |
                  |  --------------------
                  |  |   memdev regs    |
                  |  --------------------
                  |  |                  |
                  |  | +--------------+ |
 cxl_dstate->     |  | |              | |
 device_registers |  | |              | |
                  |  | |payload       | |
                  |  | |(2k currently)| |
                  |  | |              | |
                  |  | |              | |
                  |  | +--------------+ |
                  |  |   mailbox regs   |
                  |  --------------------
                  |  |    device regs   |
                  v  --------------------
                     |     caps regs    |
         BAR ------> +------------------+

Perhaps I should add this as a comment in the code?

Quoted 28 lines
Quoted 27 lines
hw/cxl/cxl-device-utils.c new file +18 -0
+
+    memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+                          "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+
+    memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+                                &cxl_dstate->caps);
+}
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+    uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
+    const int cap_count = 0;
+
+    /* CXL Device Capabilities Array Register */
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+}
hw/cxl/meson.build +1 -0
index 00c3876a0f..47154d6850 100644
@@ -1,3 +1,4 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
+  'cxl-device-utils.c',
 ))
#8 lore raw

Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)

Quoted 122 lines · On Wed, 6 Jan 2021 08:49:48 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:

This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.

v2: Break out register alignment checks (Jonathan)

Hi Ben,

One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.

Will get to a full review after messing around ('testing') this a bit more ;)

Jonathan

Quoted 63 lines
---
 hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build        |  1 +
 2 files changed, 73 insertions(+)
 create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +51 -0
new file mode 100644
index 0000000000..d1b1371e66
@@ -0,0 +1,72 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+static int cxl_device_check_register_alignment(hwaddr offset, unsigned size)
+{
+    if (unlikely(offset & (size - 1))) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
+}
+
+static const MemoryRegionOps caps_ops = {
+    .read = caps_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+    /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
Quoted 10 lines

hw/cxl/cxl-device-utils.c line 54

Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3 -0
+    memory_region_init(
+        &cxl_dstate->device_registers, obj, "device-registers",
+        pow2ceil(CXL_MAILBOX_REGISTERS_LENGTH + CXL_MAILBOX_REGISTERS_OFFSET));  

I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).

Maybe I misunderstand, but this is the intended behavior. cxl_dstate->device_registers is the MemoryRegion container for all the subregions that are the actual device MMIO.

                     +------------------+
                  ^  |                  |
                  |  |      unused      |
                  |  --------------------
                  |  |   memdev regs    |
                  |  --------------------
                  |  |                  |
                  |  | +--------------+ |
 cxl_dstate->     |  | |              | |
 device_registers |  | |              | |
                  |  | |payload       | |
                  |  | |(2k currently)| |
                  |  | |              | |
                  |  | |              | |
                  |  | +--------------+ |
                  |  |   mailbox regs   |
                  |  --------------------
                  |  |    device regs   |
                  v  --------------------
                     |     caps regs    |
         BAR ------> +------------------+
Quoted 1 line

Perhaps I should add this as a comment in the code?

I agree with intent but above it is setting limit to the top of the mailbox regs, not the memdev regs which I'd expect to see.

Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
hw/cxl/cxl-device-utils.c new file +18 -0
+
+    memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+                          "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+
+    memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+                                &cxl_dstate->caps);
+}
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+    uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
+    const int cap_count = 0;
+
+    /* CXL Device Capabilities Array Register */
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+}
hw/cxl/meson.build +1 -0
index 00c3876a0f..47154d6850 100644
@@ -1,3 +1,4 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
+  'cxl-device-utils.c',
 ))  
#9 lore raw

Re: [RFC PATCH v2 05/32] hw/cxl/device: Implement the CAP array (8.2.8.1-2)

Quoted 125 lines · Jonathan Cameron
Quoted 122 lines · On Wed, 6 Jan 2021 08:49:48 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 83 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 5 Jan 2021 08:52:56 -0800 Ben Widawsky <[email protected]> wrote:

This implements all device MMIO up to the first capability .That includes the CXL Device Capabilities Array Register, as well as all of the CXL Device Capability Header Registers. The latter are filled in as they are implemented in the following patches.

v2: Break out register alignment checks (Jonathan)

Hi Ben,

One buglet / inconsistency inline that I spotted whilst chasing that issue with size of reads.

Will get to a full review after messing around ('testing') this a bit more ;)

Jonathan

Quoted 63 lines
---
 hw/cxl/cxl-device-utils.c | 72 +++++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build        |  1 +
 2 files changed, 73 insertions(+)
 create mode 100644 hw/cxl/cxl-device-utils.c
hw/cxl/cxl-device-utils.c new file +51 -0
new file mode 100644
index 0000000000..d1b1371e66
@@ -0,0 +1,72 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+static int cxl_device_check_register_alignment(hwaddr offset, unsigned size)
+{
+    if (unlikely(offset & (size - 1))) {
+        return 1;
+    }
+
+    return 0;
+}
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
+}
+
+static const MemoryRegionOps caps_ops = {
+    .read = caps_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+    /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
Quoted 10 lines

hw/cxl/cxl-device-utils.c line 54

Quoted 3 lines
hw/cxl/cxl-device-utils.c new file +3 -0
+    memory_region_init(
+        &cxl_dstate->device_registers, obj, "device-registers",
+        pow2ceil(CXL_MAILBOX_REGISTERS_LENGTH + CXL_MAILBOX_REGISTERS_OFFSET));  

I can see why you jumped directly to sizing this for the whole region, but the snag is that means I think you missed the fact that patch 8 adds a region after the end
of the mailbox. Doesn't result in an actual bug because the ceil above takes
you way past the space needed though (the memory device region is only 8 bytes long).

Maybe I misunderstand, but this is the intended behavior. cxl_dstate->device_registers is the MemoryRegion container for all the subregions that are the actual device MMIO.

                     +------------------+
                  ^  |                  |
                  |  |      unused      |
                  |  --------------------
                  |  |   memdev regs    |
                  |  --------------------
                  |  |                  |
                  |  | +--------------+ |
 cxl_dstate->     |  | |              | |
 device_registers |  | |              | |
                  |  | |payload       | |
                  |  | |(2k currently)| |
                  |  | |              | |
                  |  | |              | |
                  |  | +--------------+ |
                  |  |   mailbox regs   |
                  |  --------------------
                  |  |    device regs   |
                  v  --------------------
                     |     caps regs    |
         BAR ------> +------------------+
Quoted 5 lines
Quoted 1 line

Perhaps I should add this as a comment in the code?

I agree with intent but above it is setting limit to the top of the mailbox regs, not the memdev regs which I'd expect to see.

oops :D - you're correct

Quoted 30 lines
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
hw/cxl/cxl-device-utils.c new file +18 -0
+
+    memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+                          "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+
+    memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+                                &cxl_dstate->caps);
+}
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+    uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
+    const int cap_count = 0;
+
+    /* CXL Device Capabilities Array Register */
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+    ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+}
hw/cxl/meson.build +1 -0
index 00c3876a0f..47154d6850 100644
@@ -1,3 +1,4 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
+  'cxl-device-utils.c',
 ))  
#10 lore raw

RFC PATCH v2 6/32 hw/cxl/device: Add device status (8.2.8.3)

In reply to Ben Widawsky

This implements the CXL device status registers from 8.2.8.3.1 in the CXL 2.0 specification. It is capability ID 0001h.

v2: Use the reg alignment helper (Jonathan)
Add stubbed device_reg_init_common() (Ben)

---
 hw/cxl/cxl-device-utils.c   | 38 ++++++++++++++++++++++++++++++++++++-
 include/hw/cxl/cxl_device.h | 16 ++++++++++++++++
 2 files changed, 53 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c +37 -1
index d1b1371e66..b86e5466bd 100644
@@ -32,6 +32,32 @@ static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(cxl_dstate->caps_reg_state + offset, size);
 }
 
+static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    uint64_t retval = 0;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(&retval, size);
+}
+
+static const MemoryRegionOps dev_ops = {
+    .read = dev_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps caps_ops = {
     .read = caps_reg_read,
     .write = NULL,
@@ -55,18 +81,28 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
 
     memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
+    memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
+                          "device-status", CXL_DEVICE_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_DEVICE_REGISTERS_OFFSET,
+                                &cxl_dstate->device);
 }
 
+static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 0;
+    const int cap_count = 1;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY2, CAP_COUNT, cap_count);
+
+    cxl_device_cap_init(cxl_dstate, DEVICE, 1);
+    device_reg_init_common(cxl_dstate);
 }
include/hw/cxl/cxl_device.h +16 -0
index 38af43c1a6..3380fc9e7a 100644
@@ -122,6 +122,22 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+#define cxl_device_cap_init(dstate, reg, cap_id)                                   \
+    do {                                                                           \
+        uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
+        int which = R_CXL_DEV_##reg##_CAP_HDR0;                                    \
+        cap_hdrs[which] =                                                          \
+            FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_ID, cap_id); \
+        cap_hdrs[which] = FIELD_DP32(                                              \
+            cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1);            \
+        cap_hdrs[which + 1] =                                                      \
+            FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1,              \
+                       CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET);                  \
+        cap_hdrs[which + 2] =                                                      \
+            FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2,              \
+                       CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH);                  \
+    } while (0)
+
 REG32(CXL_DEV_MAILBOX_CAP, 0)
     FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)
     FIELD(CXL_DEV_MAILBOX_CAP, INT_CAP, 5, 1)
Signature · 1 line
2.30.0
#11 lore raw

RFC PATCH v2 7/32 hw/cxl/device: Implement basic mailbox (8.2.8.4)

In reply to Ben Widawsky

This is the beginning of implementing mailbox support for CXL 2.0 devices.

v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)

---
 hw/cxl/cxl-device-utils.c   | 122 ++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 173 ++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build          |   1 +
 include/hw/cxl/cxl.h        |   3 +
 include/hw/cxl/cxl_device.h |  27 +++++-
 5 files changed, 322 insertions(+), 4 deletions(-)
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +121 -1
index b86e5466bd..642e3c2617 100644
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(&retval, size);
 }
 
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->mbox_reg_state + offset, size);
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CTRL:
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_CAP:
+        /* RO register */
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        break;
+    }
+
+    stl_le_p((uint8_t *)reg_state + offset, value);
+}
+
+static void mailbox_mem_writeq(uint8_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CMD:
+        break;
+    case A_CXL_DEV_BG_CMD_STS:
+        /* BG not supported */
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_STS:
+        /* Read only register, will get updated by the state machine */
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        return;
+    }
+
+    stq_le_p(reg_state + offset, value);
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+                              unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    /*
+     * Lock is needed to prevent concurrent writes as well as to prevent writes
+     * coming in while the firmware is processing. Without background commands
+     * or the second mailbox implemented, this serves no purpose since the
+     * memory access is synchronized at a higher level (per memory region).
+     */
+    RCU_READ_LOCK_GUARD();
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writeq(cxl_dstate->mbox_reg_state, offset, value);
+        break;
+    }
+
+    if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                         DOORBELL))
+        cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+    .read = mailbox_reg_read,
+    .write = mailbox_reg_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps dev_ops = {
     .read = dev_reg_read,
     .write = NULL,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
     memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+                          "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_DEVICE_REGISTERS_OFFSET,
                                 &cxl_dstate->device);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MAILBOX_REGISTERS_OFFSET,
+                                &cxl_dstate->mailbox);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
 
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+    /* 2048 payload size, with no interrupt or background support */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+                     PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+    cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 1;
+    const int cap_count = 2;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 
     cxl_device_cap_init(cxl_dstate, DEVICE, 1);
     device_reg_init_common(cxl_dstate);
+
+    cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+    mailbox_reg_init_common(cxl_dstate);
+
+    assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c new file +173 -0

173 lines not on this page · raw message

hw/cxl/meson.build +1 -0
index 47154d6850..0eca715d10 100644
@@ -1,4 +1,5 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
   'cxl-device-utils.c',
+  'cxl-mailbox-utils.c',
 ))
include/hw/cxl/cxl.h +3 -0
index 23f52c4cf9..362cda40de 100644
@@ -14,5 +14,8 @@
 #include "cxl_component.h"
 #include "cxl_device.h"
 
+#define COMPONENT_REG_BAR_IDX 0
+#define DEVICE_REG_BAR_IDX 2
+
 #endif
 
include/hw/cxl/cxl_device.h +24 -3
index 3380fc9e7a..bdf990cec2 100644
@@ -70,9 +70,10 @@
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
 typedef struct cxl_device_state {
-    /* Boss container and caps registers */
+    /* Main register container */
     MemoryRegion device_registers;
 
+    MemoryRegion device;
     struct {
         MemoryRegion caps;
         union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {
             uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >> 2];
         };
     };
-    MemoryRegion device;
-    MemoryRegion mailbox;
+    struct {
+        MemoryRegion mailbox;
+        uint16_t payload_size;
+        union {
+            uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+            uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >> 2];
+        };
+        struct {
+            uint16_t opcode;
+            uint16_t effect;
+        } cel_log[1 << 16];
+        size_t cel_size;
+    };
 
     MemoryRegion *pmem;
     MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
 #define cxl_device_cap_init(dstate, reg, cap_id)                                   \
     do {                                                                           \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4)
     FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
     FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)
 
+/* XXX: actually a 64b register */
+REG32(CXL_DEV_MAILBOX_CMD, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
 /* XXX: actually a 64b register */
 REG32(CXL_DEV_MAILBOX_STS, 0x10)
     FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)
Signature · 1 line
2.30.0
#12 lore raw

Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:

This is the beginning of implementing mailbox support for CXL 2.0 devices.

v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)

Hi Ben,

I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.

The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()

If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.

I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)

Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).

Quoted 101 lines
---
 hw/cxl/cxl-device-utils.c   | 122 ++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 173 ++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build          |   1 +
 include/hw/cxl/cxl.h        |   3 +
 include/hw/cxl/cxl_device.h |  27 +++++-
 5 files changed, 322 insertions(+), 4 deletions(-)
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +84 -0
index b86e5466bd..642e3c2617 100644
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(&retval, size);
 }
 
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->mbox_reg_state + offset, size);
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CTRL:
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_CAP:
+        /* RO register */
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        break;
+    }
+
+    stl_le_p((uint8_t *)reg_state + offset, value);
+}
+
+static void mailbox_mem_writeq(uint8_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CMD:
+        break;
+    case A_CXL_DEV_BG_CMD_STS:
+        /* BG not supported */
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_STS:
+        /* Read only register, will get updated by the state machine */
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        return;
+    }
+
+    stq_le_p(reg_state + offset, value);
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+                              unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    /*
+     * Lock is needed to prevent concurrent writes as well as to prevent writes
+     * coming in while the firmware is processing. Without background commands
+     * or the second mailbox implemented, this serves no purpose since the
+     * memory access is synchronized at a higher level (per memory region).
+     */
+    RCU_READ_LOCK_GUARD();
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writeq(cxl_dstate->mbox_reg_state, offset, value);
+        break;
+    }
+
+    if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140

Quoted 10 lines
hw/cxl/cxl-device-utils.c +10 -0
+                         DOORBELL))
+        cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+    .read = mailbox_reg_read,
+    .write = mailbox_reg_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,

Hack was to set this to 1 (and the impl below - no idea if both are needed).

Quoted 319 lines
hw/cxl/cxl-device-utils.c +27 -1
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps dev_ops = {
     .read = dev_reg_read,
     .write = NULL,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
     memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+                          "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_DEVICE_REGISTERS_OFFSET,
                                 &cxl_dstate->device);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MAILBOX_REGISTERS_OFFSET,
+                                &cxl_dstate->mailbox);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
 
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+    /* 2048 payload size, with no interrupt or background support */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+                     PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+    cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 1;
+    const int cap_count = 2;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 
     cxl_device_cap_init(cxl_dstate, DEVICE, 1);
     device_reg_init_common(cxl_dstate);
+
+    cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+    mailbox_reg_init_common(cxl_dstate);
+
+    assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c new file +173 -0
new file mode 100644
index 0000000000..3a39c936de
@@ -0,0 +1,173 @@
+/*
+ * CXL Utility library for mailbox interface
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci.h"
+#include "qemu/log.h"
+#include "qemu/uuid.h"
+
+/*
+ * How to add a new command, example. The command set FOO, with cmd BAR.
+ *  1. Add the command set and cmd to the enum.
+ *     FOO    = 0x7f,
+ *          #define BAR 0
+ *  2. Forward declare the handler.
+ *     declare_mailbox_handler(FOO_BAR);
+ *  3. Add the command to the cxl_cmd_set[][]
+ *     CXL_CMD(FOO, BAR, 0, 0),
+ *  4. Implement your handler
+ *     define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; }
+ *
+ *
+ *  Writing the handler:
+ *    The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the
+ *    in/out length of the payload. The handler is responsible for consuming the
+ *    payload from cmd->payload and operating upon it as necessary. It must then
+ *    fill the output data into cmd->payload (overwriting what was there),
+ *    setting the length, and returning a valid return code.
+ *
+ *  XXX: The handler need not worry about endianess. The payload is read out of
+ *  a register interface that already deals with it.
+ */
+
+/* 8.2.8.4.5.1 Command Return Codes */
+typedef enum {
+    CXL_MBOX_SUCCESS = 0x0,
+    CXL_MBOX_BG_STARTED = 0x1,
+    CXL_MBOX_INVALID_INPUT = 0x2,
+    CXL_MBOX_UNSUPPORTED = 0x3,
+    CXL_MBOX_INTERNAL_ERROR = 0x4,
+    CXL_MBOX_RETRY_REQUIRED = 0x5,
+    CXL_MBOX_BUSY = 0x6,
+    CXL_MBOX_MEDIA_DISABLED = 0x7,
+    CXL_MBOX_FW_XFER_IN_PROGRESS = 0x8,
+    CXL_MBOX_FW_XFER_OUT_OF_ORDER = 0x9,
+    CXL_MBOX_FW_AUTH_FAILED = 0xa,
+    CXL_MBOX_FW_INVALID_SLOT = 0xb,
+    CXL_MBOX_FW_ROLLEDBACK = 0xc,
+    CXL_MBOX_FW_REST_REQD = 0xd,
+    CXL_MBOX_INVALID_HANDLE = 0xe,
+    CXL_MBOX_INVALID_PA = 0xf,
+    CXL_MBOX_INJECT_POISON_LIMIT = 0x10,
+    CXL_MBOX_PERMANENT_MEDIA_FAILURE = 0x11,
+    CXL_MBOX_ABORTED = 0x12,
+    CXL_MBOX_INVALID_SECURITY_STATE = 0x13,
+    CXL_MBOX_INCORRECT_PASSPHRASE = 0x14,
+    CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15,
+    CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16,
+    CXL_MBOX_MAX = 0x17
+} ret_code;
+
+struct cxl_cmd;
+typedef ret_code (*opcode_handler)(struct cxl_cmd *cmd,
+                                   CXLDeviceState *cxl_dstate, uint16_t *len);
+struct cxl_cmd {
+    const char *name;
+    opcode_handler handler;
+    ssize_t in;
+    uint16_t effect; /* Reported in CEL */
+    uint8_t *payload;
+};
+
+#define define_mailbox_handler(name)                \
+    static ret_code cmd_##name(struct cxl_cmd *cmd, \
+                               CXLDeviceState *cxl_dstate, uint16_t *len)
+#define declare_mailbox_handler(name) define_mailbox_handler(name)
+
+#define CXL_CMD(s, c, in, cel_effect) \
+    [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {};
+
+#undef CXL_CMD
+
+QemuUUID cel_uuid;
+
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
+{
+    uint16_t ret = CXL_MBOX_SUCCESS;
+    struct cxl_cmd *cxl_cmd;
+    uint64_t status_reg;
+    opcode_handler h;
+
+    /*
+     * current state of mailbox interface
+     *  mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP];
+     *  mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL];
+     *  status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS];
+     */
+    uint64_t command_reg =
+        *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD];
+
+    /* Check if we have to do anything */
+    if (!ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                          DOORBELL)) {
+        qemu_log_mask(LOG_UNIMP, "Corrupt internal state for firmware\n");
+        return;
+    }
+
+    uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET);
+    uint8_t cmd = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND);
+    uint16_t len = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH);
+    cxl_cmd = &cxl_cmd_set[set][cmd];
+    h = cxl_cmd->handler;
+    if (!h) {
+        goto handled;
+    }
+
+    if (len != cxl_cmd->in) {
+        ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
+    }
+
+    cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD;
+    ret = (*h)(cxl_cmd, cxl_dstate, &len);
+    assert(len <= cxl_dstate->payload_size);
+
+handled:
+    /*
+     * Set the return code
+     * XXX: it's a 64b register, but we're not setting the vendor, so we can get
+     * away with this
+     */
+    status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret);
+
+    /*
+     * Set the return length
+     */
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH, len);
+
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_CMD, command_reg);
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_STS, status_reg);
+
+    /* Tell the host we're done */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                     DOORBELL, 0);
+}
+
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate)
+{
+    const char *cel_uuidstr = "0da9c0b5-bf41-4b78-8f79-96b1623b3f17";
+
+    for (int i = 0; i < 256; i++) {
+        for (int j = 0; j < 256; j++) {
+            if (cxl_cmd_set[i][j].handler) {
+                struct cxl_cmd *c = &cxl_cmd_set[i][j];
+
+                cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i << 8) | j;
+                cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect;
+                cxl_dstate->cel_size++;
+            }
+        }
+    }
+
+    return qemu_uuid_parse(cel_uuidstr, &cel_uuid);
+}
hw/cxl/meson.build +1 -0
index 47154d6850..0eca715d10 100644
@@ -1,4 +1,5 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
   'cxl-device-utils.c',
+  'cxl-mailbox-utils.c',
 ))
include/hw/cxl/cxl.h +3 -0
index 23f52c4cf9..362cda40de 100644
@@ -14,5 +14,8 @@
 #include "cxl_component.h"
 #include "cxl_device.h"
 
+#define COMPONENT_REG_BAR_IDX 0
+#define DEVICE_REG_BAR_IDX 2
+
 #endif
 
include/hw/cxl/cxl_device.h +24 -3
index 3380fc9e7a..bdf990cec2 100644
@@ -70,9 +70,10 @@
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
 typedef struct cxl_device_state {
-    /* Boss container and caps registers */
+    /* Main register container */
     MemoryRegion device_registers;
 
+    MemoryRegion device;
     struct {
         MemoryRegion caps;
         union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {
             uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >> 2];
         };
     };
-    MemoryRegion device;
-    MemoryRegion mailbox;
+    struct {
+        MemoryRegion mailbox;
+        uint16_t payload_size;
+        union {
+            uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+            uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >> 2];
+        };
+        struct {
+            uint16_t opcode;
+            uint16_t effect;
+        } cel_log[1 << 16];
+        size_t cel_size;
+    };
 
     MemoryRegion *pmem;
     MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
 #define cxl_device_cap_init(dstate, reg, cap_id)                                   \
     do {                                                                           \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4)
     FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
     FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)
 
+/* XXX: actually a 64b register */
+REG32(CXL_DEV_MAILBOX_CMD, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
 /* XXX: actually a 64b register */
 REG32(CXL_DEV_MAILBOX_STS, 0x10)
     FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)
#13 lore raw

Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

Quoted 13 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:

This is the beginning of implementing mailbox support for CXL 2.0 devices.

v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)

Quoted 6 lines
Quoted 1 line

Hi Ben,

I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.

Cool!

Quoted 5 lines

The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()

https://gitlab.com/bwidawsk/linux/-/tree/cxl-2.0v3

Vishal in the Cc is finishing up some work on adding support to ndctl, after which I'll be sending that (or something closely related) to the mailing list.

Quoted 4 lines

If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.

Could you add the backtrace here? You caused me to go look at what x86 does. I actually assumed on modern platforms it did movsb (byte moves), which are highly optimized in microcode. In fact, it does some amount of word moves too, which was new info to me.

So I'm not really sure yet why ARM64 is special here.

Quoted 5 lines

I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)

8.2
• A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes quantity.
• A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8 Bytes quantity

Interesting though, I really thought QEMU should do the right thing here and you wouldn't need to allow 1byte accesses.

Quoted 5 lines

Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).

I think the right solution is to support/allow byte and word access. I just thought it should work in the existing code. I'd like to try to figure out what's going on.

How much effort would it be for me to reproduce what you've done?

Quoted 435 lines
Quoted 102 lines
---
 hw/cxl/cxl-device-utils.c   | 122 ++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 173 ++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build          |   1 +
 include/hw/cxl/cxl.h        |   3 +
 include/hw/cxl/cxl_device.h |  27 +++++-
 5 files changed, 322 insertions(+), 4 deletions(-)
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +84 -0
index b86e5466bd..642e3c2617 100644
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(&retval, size);
 }
 
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->mbox_reg_state + offset, size);
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CTRL:
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_CAP:
+        /* RO register */
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        break;
+    }
+
+    stl_le_p((uint8_t *)reg_state + offset, value);
+}
+
+static void mailbox_mem_writeq(uint8_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CMD:
+        break;
+    case A_CXL_DEV_BG_CMD_STS:
+        /* BG not supported */
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_STS:
+        /* Read only register, will get updated by the state machine */
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        return;
+    }
+
+    stq_le_p(reg_state + offset, value);
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+                              unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    /*
+     * Lock is needed to prevent concurrent writes as well as to prevent writes
+     * coming in while the firmware is processing. Without background commands
+     * or the second mailbox implemented, this serves no purpose since the
+     * memory access is synchronized at a higher level (per memory region).
+     */
+    RCU_READ_LOCK_GUARD();
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writeq(cxl_dstate->mbox_reg_state, offset, value);
+        break;
+    }
+
+    if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140

Quoted 10 lines
hw/cxl/cxl-device-utils.c +10 -0
+                         DOORBELL))
+        cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+    .read = mailbox_reg_read,
+    .write = mailbox_reg_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,

Hack was to set this to 1 (and the impl below - no idea if both are needed).

Quoted 319 lines
hw/cxl/cxl-device-utils.c +27 -1
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps dev_ops = {
     .read = dev_reg_read,
     .write = NULL,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
     memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+                          "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_DEVICE_REGISTERS_OFFSET,
                                 &cxl_dstate->device);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MAILBOX_REGISTERS_OFFSET,
+                                &cxl_dstate->mailbox);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
 
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+    /* 2048 payload size, with no interrupt or background support */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+                     PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+    cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 1;
+    const int cap_count = 2;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 
     cxl_device_cap_init(cxl_dstate, DEVICE, 1);
     device_reg_init_common(cxl_dstate);
+
+    cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+    mailbox_reg_init_common(cxl_dstate);
+
+    assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c new file +173 -0
new file mode 100644
index 0000000000..3a39c936de
@@ -0,0 +1,173 @@
+/*
+ * CXL Utility library for mailbox interface
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci.h"
+#include "qemu/log.h"
+#include "qemu/uuid.h"
+
+/*
+ * How to add a new command, example. The command set FOO, with cmd BAR.
+ *  1. Add the command set and cmd to the enum.
+ *     FOO    = 0x7f,
+ *          #define BAR 0
+ *  2. Forward declare the handler.
+ *     declare_mailbox_handler(FOO_BAR);
+ *  3. Add the command to the cxl_cmd_set[][]
+ *     CXL_CMD(FOO, BAR, 0, 0),
+ *  4. Implement your handler
+ *     define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; }
+ *
+ *
+ *  Writing the handler:
+ *    The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the
+ *    in/out length of the payload. The handler is responsible for consuming the
+ *    payload from cmd->payload and operating upon it as necessary. It must then
+ *    fill the output data into cmd->payload (overwriting what was there),
+ *    setting the length, and returning a valid return code.
+ *
+ *  XXX: The handler need not worry about endianess. The payload is read out of
+ *  a register interface that already deals with it.
+ */
+
+/* 8.2.8.4.5.1 Command Return Codes */
+typedef enum {
+    CXL_MBOX_SUCCESS = 0x0,
+    CXL_MBOX_BG_STARTED = 0x1,
+    CXL_MBOX_INVALID_INPUT = 0x2,
+    CXL_MBOX_UNSUPPORTED = 0x3,
+    CXL_MBOX_INTERNAL_ERROR = 0x4,
+    CXL_MBOX_RETRY_REQUIRED = 0x5,
+    CXL_MBOX_BUSY = 0x6,
+    CXL_MBOX_MEDIA_DISABLED = 0x7,
+    CXL_MBOX_FW_XFER_IN_PROGRESS = 0x8,
+    CXL_MBOX_FW_XFER_OUT_OF_ORDER = 0x9,
+    CXL_MBOX_FW_AUTH_FAILED = 0xa,
+    CXL_MBOX_FW_INVALID_SLOT = 0xb,
+    CXL_MBOX_FW_ROLLEDBACK = 0xc,
+    CXL_MBOX_FW_REST_REQD = 0xd,
+    CXL_MBOX_INVALID_HANDLE = 0xe,
+    CXL_MBOX_INVALID_PA = 0xf,
+    CXL_MBOX_INJECT_POISON_LIMIT = 0x10,
+    CXL_MBOX_PERMANENT_MEDIA_FAILURE = 0x11,
+    CXL_MBOX_ABORTED = 0x12,
+    CXL_MBOX_INVALID_SECURITY_STATE = 0x13,
+    CXL_MBOX_INCORRECT_PASSPHRASE = 0x14,
+    CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15,
+    CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16,
+    CXL_MBOX_MAX = 0x17
+} ret_code;
+
+struct cxl_cmd;
+typedef ret_code (*opcode_handler)(struct cxl_cmd *cmd,
+                                   CXLDeviceState *cxl_dstate, uint16_t *len);
+struct cxl_cmd {
+    const char *name;
+    opcode_handler handler;
+    ssize_t in;
+    uint16_t effect; /* Reported in CEL */
+    uint8_t *payload;
+};
+
+#define define_mailbox_handler(name)                \
+    static ret_code cmd_##name(struct cxl_cmd *cmd, \
+                               CXLDeviceState *cxl_dstate, uint16_t *len)
+#define declare_mailbox_handler(name) define_mailbox_handler(name)
+
+#define CXL_CMD(s, c, in, cel_effect) \
+    [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {};
+
+#undef CXL_CMD
+
+QemuUUID cel_uuid;
+
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
+{
+    uint16_t ret = CXL_MBOX_SUCCESS;
+    struct cxl_cmd *cxl_cmd;
+    uint64_t status_reg;
+    opcode_handler h;
+
+    /*
+     * current state of mailbox interface
+     *  mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP];
+     *  mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL];
+     *  status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS];
+     */
+    uint64_t command_reg =
+        *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD];
+
+    /* Check if we have to do anything */
+    if (!ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                          DOORBELL)) {
+        qemu_log_mask(LOG_UNIMP, "Corrupt internal state for firmware\n");
+        return;
+    }
+
+    uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET);
+    uint8_t cmd = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND);
+    uint16_t len = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH);
+    cxl_cmd = &cxl_cmd_set[set][cmd];
+    h = cxl_cmd->handler;
+    if (!h) {
+        goto handled;
+    }
+
+    if (len != cxl_cmd->in) {
+        ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
+    }
+
+    cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD;
+    ret = (*h)(cxl_cmd, cxl_dstate, &len);
+    assert(len <= cxl_dstate->payload_size);
+
+handled:
+    /*
+     * Set the return code
+     * XXX: it's a 64b register, but we're not setting the vendor, so we can get
+     * away with this
+     */
+    status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret);
+
+    /*
+     * Set the return length
+     */
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH, len);
+
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_CMD, command_reg);
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_STS, status_reg);
+
+    /* Tell the host we're done */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                     DOORBELL, 0);
+}
+
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate)
+{
+    const char *cel_uuidstr = "0da9c0b5-bf41-4b78-8f79-96b1623b3f17";
+
+    for (int i = 0; i < 256; i++) {
+        for (int j = 0; j < 256; j++) {
+            if (cxl_cmd_set[i][j].handler) {
+                struct cxl_cmd *c = &cxl_cmd_set[i][j];
+
+                cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i << 8) | j;
+                cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect;
+                cxl_dstate->cel_size++;
+            }
+        }
+    }
+
+    return qemu_uuid_parse(cel_uuidstr, &cel_uuid);
+}
hw/cxl/meson.build +1 -0
index 47154d6850..0eca715d10 100644
@@ -1,4 +1,5 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
   'cxl-device-utils.c',
+  'cxl-mailbox-utils.c',
 ))
include/hw/cxl/cxl.h +3 -0
index 23f52c4cf9..362cda40de 100644
@@ -14,5 +14,8 @@
 #include "cxl_component.h"
 #include "cxl_device.h"
 
+#define COMPONENT_REG_BAR_IDX 0
+#define DEVICE_REG_BAR_IDX 2
+
 #endif
 
include/hw/cxl/cxl_device.h +24 -3
index 3380fc9e7a..bdf990cec2 100644
@@ -70,9 +70,10 @@
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
 typedef struct cxl_device_state {
-    /* Boss container and caps registers */
+    /* Main register container */
     MemoryRegion device_registers;
 
+    MemoryRegion device;
     struct {
         MemoryRegion caps;
         union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {
             uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >> 2];
         };
     };
-    MemoryRegion device;
-    MemoryRegion mailbox;
+    struct {
+        MemoryRegion mailbox;
+        uint16_t payload_size;
+        union {
+            uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+            uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >> 2];
+        };
+        struct {
+            uint16_t opcode;
+            uint16_t effect;
+        } cel_log[1 << 16];
+        size_t cel_size;
+    };
 
     MemoryRegion *pmem;
     MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
 #define cxl_device_cap_init(dstate, reg, cap_id)                                   \
     do {                                                                           \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4)
     FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
     FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)
 
+/* XXX: actually a 64b register */
+REG32(CXL_DEV_MAILBOX_CMD, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
 /* XXX: actually a 64b register */
 REG32(CXL_DEV_MAILBOX_STS, 0x10)
     FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)
#14 lore raw

Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

In reply to Jonathan Cameron
Quoted 34 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:

This is the beginning of implementing mailbox support for CXL 2.0 devices.

v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)

Hi Ben,

I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.

The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()

If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.

I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)

Quoted 4 lines

Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).

Bit more testing an another little thing below.

J

Quoted 57 lines
Quoted 57 lines
---
 hw/cxl/cxl-device-utils.c   | 122 ++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 173 ++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build          |   1 +
 include/hw/cxl/cxl.h        |   3 +
 include/hw/cxl/cxl_device.h |  27 +++++-
 5 files changed, 322 insertions(+), 4 deletions(-)
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +39 -0
index b86e5466bd..642e3c2617 100644
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(&retval, size);
 }
 
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->mbox_reg_state + offset, size);
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CTRL:
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_CAP:
+        /* RO register */
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        break;
+    }
+
+    stl_le_p((uint8_t *)reg_state + offset, value);
+}
+
+static void mailbox_mem_writeq(uint8_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CMD:
+        break;
+    case A_CXL_DEV_BG_CMD_STS:
+        /* BG not supported */
Quoted 8 lines

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 93

Quoted 8 lines
hw/cxl/cxl-device-utils.c +8 -0
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_STS:
+        /* Read only register, will get updated by the state machine */
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);

I've been debugging mail box issues and it seems we can hit this path if a payload is written by the OS. Result is it never gets written into the memory and hence isn't available when we try to read it below.

Quoted 233 lines
Quoted 37 lines
hw/cxl/cxl-device-utils.c +37 -0
+        return;
+    }
+
+    stq_le_p(reg_state + offset, value);
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+                              unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    /*
+     * Lock is needed to prevent concurrent writes as well as to prevent writes
+     * coming in while the firmware is processing. Without background commands
+     * or the second mailbox implemented, this serves no purpose since the
+     * memory access is synchronized at a higher level (per memory region).
+     */
+    RCU_READ_LOCK_GUARD();
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writeq(cxl_dstate->mbox_reg_state, offset, value);
+        break;
+    }
+
+    if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140

Quoted 10 lines
hw/cxl/cxl-device-utils.c +10 -0
+                         DOORBELL))
+        cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+    .read = mailbox_reg_read,
+    .write = mailbox_reg_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,  

Hack was to set this to 1 (and the impl below - no idea if both are needed).

Quoted 183 lines
hw/cxl/cxl-device-utils.c +27 -1
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps dev_ops = {
     .read = dev_reg_read,
     .write = NULL,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
     memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+                          "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_DEVICE_REGISTERS_OFFSET,
                                 &cxl_dstate->device);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MAILBOX_REGISTERS_OFFSET,
+                                &cxl_dstate->mailbox);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
 
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+    /* 2048 payload size, with no interrupt or background support */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+                     PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+    cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 1;
+    const int cap_count = 2;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 
     cxl_device_cap_init(cxl_dstate, DEVICE, 1);
     device_reg_init_common(cxl_dstate);
+
+    cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+    mailbox_reg_init_common(cxl_dstate);
+
+    assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c new file +121 -0
new file mode 100644
index 0000000000..3a39c936de
@@ -0,0 +1,173 @@
+/*
+ * CXL Utility library for mailbox interface
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci.h"
+#include "qemu/log.h"
+#include "qemu/uuid.h"
+
+/*
+ * How to add a new command, example. The command set FOO, with cmd BAR.
+ *  1. Add the command set and cmd to the enum.
+ *     FOO    = 0x7f,
+ *          #define BAR 0
+ *  2. Forward declare the handler.
+ *     declare_mailbox_handler(FOO_BAR);
+ *  3. Add the command to the cxl_cmd_set[][]
+ *     CXL_CMD(FOO, BAR, 0, 0),
+ *  4. Implement your handler
+ *     define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; }
+ *
+ *
+ *  Writing the handler:
+ *    The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the
+ *    in/out length of the payload. The handler is responsible for consuming the
+ *    payload from cmd->payload and operating upon it as necessary. It must then
+ *    fill the output data into cmd->payload (overwriting what was there),
+ *    setting the length, and returning a valid return code.
+ *
+ *  XXX: The handler need not worry about endianess. The payload is read out of
+ *  a register interface that already deals with it.
+ */
+
+/* 8.2.8.4.5.1 Command Return Codes */
+typedef enum {
+    CXL_MBOX_SUCCESS = 0x0,
+    CXL_MBOX_BG_STARTED = 0x1,
+    CXL_MBOX_INVALID_INPUT = 0x2,
+    CXL_MBOX_UNSUPPORTED = 0x3,
+    CXL_MBOX_INTERNAL_ERROR = 0x4,
+    CXL_MBOX_RETRY_REQUIRED = 0x5,
+    CXL_MBOX_BUSY = 0x6,
+    CXL_MBOX_MEDIA_DISABLED = 0x7,
+    CXL_MBOX_FW_XFER_IN_PROGRESS = 0x8,
+    CXL_MBOX_FW_XFER_OUT_OF_ORDER = 0x9,
+    CXL_MBOX_FW_AUTH_FAILED = 0xa,
+    CXL_MBOX_FW_INVALID_SLOT = 0xb,
+    CXL_MBOX_FW_ROLLEDBACK = 0xc,
+    CXL_MBOX_FW_REST_REQD = 0xd,
+    CXL_MBOX_INVALID_HANDLE = 0xe,
+    CXL_MBOX_INVALID_PA = 0xf,
+    CXL_MBOX_INJECT_POISON_LIMIT = 0x10,
+    CXL_MBOX_PERMANENT_MEDIA_FAILURE = 0x11,
+    CXL_MBOX_ABORTED = 0x12,
+    CXL_MBOX_INVALID_SECURITY_STATE = 0x13,
+    CXL_MBOX_INCORRECT_PASSPHRASE = 0x14,
+    CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15,
+    CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16,
+    CXL_MBOX_MAX = 0x17
+} ret_code;
+
+struct cxl_cmd;
+typedef ret_code (*opcode_handler)(struct cxl_cmd *cmd,
+                                   CXLDeviceState *cxl_dstate, uint16_t *len);
+struct cxl_cmd {
+    const char *name;
+    opcode_handler handler;
+    ssize_t in;
+    uint16_t effect; /* Reported in CEL */
+    uint8_t *payload;
+};
+
+#define define_mailbox_handler(name)                \
+    static ret_code cmd_##name(struct cxl_cmd *cmd, \
+                               CXLDeviceState *cxl_dstate, uint16_t *len)
+#define declare_mailbox_handler(name) define_mailbox_handler(name)
+
+#define CXL_CMD(s, c, in, cel_effect) \
+    [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {};
+
+#undef CXL_CMD
+
+QemuUUID cel_uuid;
+
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
+{
+    uint16_t ret = CXL_MBOX_SUCCESS;
+    struct cxl_cmd *cxl_cmd;
+    uint64_t status_reg;
+    opcode_handler h;
+
+    /*
+     * current state of mailbox interface
+     *  mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP];
+     *  mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL];
+     *  status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS];
+     */
+    uint64_t command_reg =
+        *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD];
+
+    /* Check if we have to do anything */
+    if (!ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                          DOORBELL)) {
+        qemu_log_mask(LOG_UNIMP, "Corrupt internal state for firmware\n");
+        return;
+    }
+
+    uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET);
+    uint8_t cmd = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND);
+    uint16_t len = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH);
+    cxl_cmd = &cxl_cmd_set[set][cmd];
+    h = cxl_cmd->handler;
+    if (!h) {
Quoted 8 lines

hw/cxl/cxl-mailbox-utils.c line 129

Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c new file +8 -0
+        goto handled;
+    }
+
+    if (len != cxl_cmd->in) {
+        ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
+    }
+
+    cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD;

The return highlighted above on which registers are writable meant the payload never has anything useful in it. Remove that and this seems to work and I can query implemented commands + enable them.

Quoted 131 lines
Quoted 118 lines
hw/cxl/cxl-mailbox-utils.c new file +44 -0
+    ret = (*h)(cxl_cmd, cxl_dstate, &len);
+    assert(len <= cxl_dstate->payload_size);
+
+handled:
+    /*
+     * Set the return code
+     * XXX: it's a 64b register, but we're not setting the vendor, so we can get
+     * away with this
+     */
+    status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret);
+
+    /*
+     * Set the return length
+     */
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH, len);
+
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_CMD, command_reg);
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_STS, status_reg);
+
+    /* Tell the host we're done */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                     DOORBELL, 0);
+}
+
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate)
+{
+    const char *cel_uuidstr = "0da9c0b5-bf41-4b78-8f79-96b1623b3f17";
+
+    for (int i = 0; i < 256; i++) {
+        for (int j = 0; j < 256; j++) {
+            if (cxl_cmd_set[i][j].handler) {
+                struct cxl_cmd *c = &cxl_cmd_set[i][j];
+
+                cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i << 8) | j;
+                cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect;
+                cxl_dstate->cel_size++;
+            }
+        }
+    }
+
+    return qemu_uuid_parse(cel_uuidstr, &cel_uuid);
+}
hw/cxl/meson.build +1 -0
index 47154d6850..0eca715d10 100644
@@ -1,4 +1,5 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
   'cxl-device-utils.c',
+  'cxl-mailbox-utils.c',
 ))
include/hw/cxl/cxl.h +3 -0
index 23f52c4cf9..362cda40de 100644
@@ -14,5 +14,8 @@
 #include "cxl_component.h"
 #include "cxl_device.h"
 
+#define COMPONENT_REG_BAR_IDX 0
+#define DEVICE_REG_BAR_IDX 2
+
 #endif
 
include/hw/cxl/cxl_device.h +18 -3
index 3380fc9e7a..bdf990cec2 100644
@@ -70,9 +70,10 @@
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
 typedef struct cxl_device_state {
-    /* Boss container and caps registers */
+    /* Main register container */
     MemoryRegion device_registers;
 
+    MemoryRegion device;
     struct {
         MemoryRegion caps;
         union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {
             uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >> 2];
         };
     };
-    MemoryRegion device;
-    MemoryRegion mailbox;
+    struct {
+        MemoryRegion mailbox;
+        uint16_t payload_size;
+        union {
+            uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+            uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >> 2];
+        };
+        struct {
+            uint16_t opcode;
+            uint16_t effect;
+        } cel_log[1 << 16];
+        size_t cel_size;
+    };
 
     MemoryRegion *pmem;
     MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
 #define cxl_device_cap_init(dstate, reg, cap_id)                                   \
     do {                                                                           \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4)
     FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
     FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)

include/hw/cxl/cxl_device.h REG32(CXL_DEV_MAILBOX_CTRL, 4) · line 175

Quoted 10 lines
include/hw/cxl/cxl_device.h +6 -0
 
+/* XXX: actually a 64b register */
+REG32(CXL_DEV_MAILBOX_CMD, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
 /* XXX: actually a 64b register */
 REG32(CXL_DEV_MAILBOX_STS, 0x10)
     FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)  

_______________________________________________
Linuxarm mailing list -- [email protected]
To unsubscribe send an email to [email protected]

#15 lore raw

Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

Quoted 107 lines · Jonathan Cameron
Quoted 34 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…
Quoted 11 lines · On Tue, 5 Jan 2021 08:52:58 -0800 Ben Widawsky <[email protected]> wrote:

This is the beginning of implementing mailbox support for CXL 2.0 devices.

v2: Use register alignment helper (Ben)
Minor cleanups (Jonathan)
Rename error codes to match spec (Jonathan)
Update cap count from 1 to 2 (Jonathan)
Add infra to support CEL (Ben)
Add more of the actual mailbox handling from later patch (Ben)

Hi Ben,

I hacked support in for ARM64 to give this a spin and ran into an interesting problem around read sizes.

The mailbox registers space allows 4 or 8 byte reads, but in the kernel driver (I think I have the right version from your github) you do the payload drain with
memcpy_from_io()

If the size of the payload is not a multiple of 8 bytes, on ARM64 that results in byte reads and an exception. This happens with some of the existing calls which happen to have non multiple of 8 payload sizes.

I hacked below to allow 1 byte reads from that region but that's probably not the right fix. I found a statement in the CXL spec saying maximum read size from this register block was 8 bytes but couldn't immediately see a minimum. (I haven't looked that hard yet though!)

Quoted 4 lines

Various approaches in kernel could also be used:
1) Change the payload drain to have specific handling for the end few bytes.
2) Pad the various structures to ensure payloads are always 8 byte multiples in length (nasty).

Bit more testing an another little thing below.

J

Quoted 61 lines
Quoted 61 lines
---
 hw/cxl/cxl-device-utils.c   | 122 ++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 173 ++++++++++++++++++++++++++++++++++++
 hw/cxl/meson.build          |   1 +
 include/hw/cxl/cxl.h        |   3 +
 include/hw/cxl/cxl_device.h |  27 +++++-
 5 files changed, 322 insertions(+), 4 deletions(-)
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
hw/cxl/cxl-device-utils.c +43 -0
index b86e5466bd..642e3c2617 100644
@@ -44,6 +44,108 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
     return ldn_le_p(&retval, size);
 }
 
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    if (cxl_device_check_register_alignment(offset, size)) {
+        qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+        return 0;
+    }
+
+    return ldn_le_p(cxl_dstate->mbox_reg_state + offset, size);
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CTRL:
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_CAP:
+        /* RO register */
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+        break;
+    }
+
+    stl_le_p((uint8_t *)reg_state + offset, value);
+}
+
+static void mailbox_mem_writeq(uint8_t *reg_state, hwaddr offset,
+                               uint64_t value)
+{
+    switch (offset) {
+    case A_CXL_DEV_MAILBOX_CMD:
+        break;
+    case A_CXL_DEV_BG_CMD_STS:
+        /* BG not supported */
+        /* fallthrough */
+    case A_CXL_DEV_MAILBOX_STS:
+        /* Read only register, will get updated by the state machine */
+        return;
Quoted 9 lines
Quoted 4 lines

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 93

Quoted 4 lines
hw/cxl/cxl-device-utils.c +4 -0
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);

I've been debugging mail box issues and it seems we can hit this path if a payload is written by the OS. Result is it never gets written into the memory and hence isn't available when we try to read it below.

I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.

I think the fix should be something like this, but I can't easily test at the moment:

hw/cxl/cxl-device-utils.c +3 -0
index c515d45d20..b38e9b4c17 100644
@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
 {
     CXLDeviceState *cxl_dstate = opaque;
 
+    if (offset >= A_CXL_DEV_CMD_PAYLOAD)
+        stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+
     /*
      * Lock is needed to prevent concurrent writes as well as to prevent writes
      * coming in while the firmware is processing. Without background commands
Quoted 378 lines
Quoted 233 lines
Quoted 37 lines
hw/cxl/cxl-device-utils.c +37 -0
+        return;
+    }
+
+    stq_le_p(reg_state + offset, value);
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+                              unsigned size)
+{
+    CXLDeviceState *cxl_dstate = opaque;
+
+    /*
+     * Lock is needed to prevent concurrent writes as well as to prevent writes
+     * coming in while the firmware is processing. Without background commands
+     * or the second mailbox implemented, this serves no purpose since the
+     * memory access is synchronized at a higher level (per memory region).
+     */
+    RCU_READ_LOCK_GUARD();
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return;
+        }
+        mailbox_mem_writeq(cxl_dstate->mbox_reg_state, offset, value);
+        break;
+    }
+
+    if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,

hw/cxl/cxl-device-utils.c static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size) · line 140

Quoted 10 lines
hw/cxl/cxl-device-utils.c +10 -0
+                         DOORBELL))
+        cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+    .read = mailbox_reg_read,
+    .write = mailbox_reg_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,  

Hack was to set this to 1 (and the impl below - no idea if both are needed).

Quoted 183 lines
hw/cxl/cxl-device-utils.c +27 -1
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps dev_ops = {
     .read = dev_reg_read,
     .write = NULL,
@@ -83,20 +185,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "cap-array", CXL_DEVICE_REGISTERS_OFFSET - 0);
     memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+                          "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_DEVICE_REGISTERS_OFFSET,
                                 &cxl_dstate->device);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MAILBOX_REGISTERS_OFFSET,
+                                &cxl_dstate->mailbox);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
 
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+    /* 2048 payload size, with no interrupt or background support */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+                     PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+    cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 1;
+    const int cap_count = 2;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -105,4 +220,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 
     cxl_device_cap_init(cxl_dstate, DEVICE, 1);
     device_reg_init_common(cxl_dstate);
+
+    cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+    mailbox_reg_init_common(cxl_dstate);
+
+    assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c new file +121 -0
new file mode 100644
index 0000000000..3a39c936de
@@ -0,0 +1,173 @@
+/*
+ * CXL Utility library for mailbox interface
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci.h"
+#include "qemu/log.h"
+#include "qemu/uuid.h"
+
+/*
+ * How to add a new command, example. The command set FOO, with cmd BAR.
+ *  1. Add the command set and cmd to the enum.
+ *     FOO    = 0x7f,
+ *          #define BAR 0
+ *  2. Forward declare the handler.
+ *     declare_mailbox_handler(FOO_BAR);
+ *  3. Add the command to the cxl_cmd_set[][]
+ *     CXL_CMD(FOO, BAR, 0, 0),
+ *  4. Implement your handler
+ *     define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; }
+ *
+ *
+ *  Writing the handler:
+ *    The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the
+ *    in/out length of the payload. The handler is responsible for consuming the
+ *    payload from cmd->payload and operating upon it as necessary. It must then
+ *    fill the output data into cmd->payload (overwriting what was there),
+ *    setting the length, and returning a valid return code.
+ *
+ *  XXX: The handler need not worry about endianess. The payload is read out of
+ *  a register interface that already deals with it.
+ */
+
+/* 8.2.8.4.5.1 Command Return Codes */
+typedef enum {
+    CXL_MBOX_SUCCESS = 0x0,
+    CXL_MBOX_BG_STARTED = 0x1,
+    CXL_MBOX_INVALID_INPUT = 0x2,
+    CXL_MBOX_UNSUPPORTED = 0x3,
+    CXL_MBOX_INTERNAL_ERROR = 0x4,
+    CXL_MBOX_RETRY_REQUIRED = 0x5,
+    CXL_MBOX_BUSY = 0x6,
+    CXL_MBOX_MEDIA_DISABLED = 0x7,
+    CXL_MBOX_FW_XFER_IN_PROGRESS = 0x8,
+    CXL_MBOX_FW_XFER_OUT_OF_ORDER = 0x9,
+    CXL_MBOX_FW_AUTH_FAILED = 0xa,
+    CXL_MBOX_FW_INVALID_SLOT = 0xb,
+    CXL_MBOX_FW_ROLLEDBACK = 0xc,
+    CXL_MBOX_FW_REST_REQD = 0xd,
+    CXL_MBOX_INVALID_HANDLE = 0xe,
+    CXL_MBOX_INVALID_PA = 0xf,
+    CXL_MBOX_INJECT_POISON_LIMIT = 0x10,
+    CXL_MBOX_PERMANENT_MEDIA_FAILURE = 0x11,
+    CXL_MBOX_ABORTED = 0x12,
+    CXL_MBOX_INVALID_SECURITY_STATE = 0x13,
+    CXL_MBOX_INCORRECT_PASSPHRASE = 0x14,
+    CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15,
+    CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16,
+    CXL_MBOX_MAX = 0x17
+} ret_code;
+
+struct cxl_cmd;
+typedef ret_code (*opcode_handler)(struct cxl_cmd *cmd,
+                                   CXLDeviceState *cxl_dstate, uint16_t *len);
+struct cxl_cmd {
+    const char *name;
+    opcode_handler handler;
+    ssize_t in;
+    uint16_t effect; /* Reported in CEL */
+    uint8_t *payload;
+};
+
+#define define_mailbox_handler(name)                \
+    static ret_code cmd_##name(struct cxl_cmd *cmd, \
+                               CXLDeviceState *cxl_dstate, uint16_t *len)
+#define declare_mailbox_handler(name) define_mailbox_handler(name)
+
+#define CXL_CMD(s, c, in, cel_effect) \
+    [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {};
+
+#undef CXL_CMD
+
+QemuUUID cel_uuid;
+
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
+{
+    uint16_t ret = CXL_MBOX_SUCCESS;
+    struct cxl_cmd *cxl_cmd;
+    uint64_t status_reg;
+    opcode_handler h;
+
+    /*
+     * current state of mailbox interface
+     *  mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP];
+     *  mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL];
+     *  status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS];
+     */
+    uint64_t command_reg =
+        *(uint64_t *)&cxl_dstate->mbox_reg_state[A_CXL_DEV_MAILBOX_CMD];
+
+    /* Check if we have to do anything */
+    if (!ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                          DOORBELL)) {
+        qemu_log_mask(LOG_UNIMP, "Corrupt internal state for firmware\n");
+        return;
+    }
+
+    uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET);
+    uint8_t cmd = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND);
+    uint16_t len = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH);
+    cxl_cmd = &cxl_cmd_set[set][cmd];
+    h = cxl_cmd->handler;
+    if (!h) {
Quoted 8 lines

hw/cxl/cxl-mailbox-utils.c line 129

Quoted 8 lines
hw/cxl/cxl-mailbox-utils.c new file +8 -0
+        goto handled;
+    }
+
+    if (len != cxl_cmd->in) {
+        ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
+    }
+
+    cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD;

The return highlighted above on which registers are writable meant the payload never has anything useful in it. Remove that and this seems to work and I can query implemented commands + enable them.

Quoted 131 lines
Quoted 118 lines
hw/cxl/cxl-mailbox-utils.c new file +44 -0
+    ret = (*h)(cxl_cmd, cxl_dstate, &len);
+    assert(len <= cxl_dstate->payload_size);
+
+handled:
+    /*
+     * Set the return code
+     * XXX: it's a 64b register, but we're not setting the vendor, so we can get
+     * away with this
+     */
+    status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret);
+
+    /*
+     * Set the return length
+     */
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND, 0);
+    command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH, len);
+
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_CMD, command_reg);
+    stq_le_p(cxl_dstate->mbox_reg_state + A_CXL_DEV_MAILBOX_STS, status_reg);
+
+    /* Tell the host we're done */
+    ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+                     DOORBELL, 0);
+}
+
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate)
+{
+    const char *cel_uuidstr = "0da9c0b5-bf41-4b78-8f79-96b1623b3f17";
+
+    for (int i = 0; i < 256; i++) {
+        for (int j = 0; j < 256; j++) {
+            if (cxl_cmd_set[i][j].handler) {
+                struct cxl_cmd *c = &cxl_cmd_set[i][j];
+
+                cxl_dstate->cel_log[cxl_dstate->cel_size].opcode = (i << 8) | j;
+                cxl_dstate->cel_log[cxl_dstate->cel_size].effect = c->effect;
+                cxl_dstate->cel_size++;
+            }
+        }
+    }
+
+    return qemu_uuid_parse(cel_uuidstr, &cel_uuid);
+}
hw/cxl/meson.build +1 -0
index 47154d6850..0eca715d10 100644
@@ -1,4 +1,5 @@
 softmmu_ss.add(when: 'CONFIG_CXL', if_true: files(
   'cxl-component-utils.c',
   'cxl-device-utils.c',
+  'cxl-mailbox-utils.c',
 ))
include/hw/cxl/cxl.h +3 -0
index 23f52c4cf9..362cda40de 100644
@@ -14,5 +14,8 @@
 #include "cxl_component.h"
 #include "cxl_device.h"
 
+#define COMPONENT_REG_BAR_IDX 0
+#define DEVICE_REG_BAR_IDX 2
+
 #endif
 
include/hw/cxl/cxl_device.h +18 -3
index 3380fc9e7a..bdf990cec2 100644
@@ -70,9 +70,10 @@
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
 typedef struct cxl_device_state {
-    /* Boss container and caps registers */
+    /* Main register container */
     MemoryRegion device_registers;
 
+    MemoryRegion device;
     struct {
         MemoryRegion caps;
         union {
@@ -80,8 +81,19 @@ typedef struct cxl_device_state {
             uint32_t caps_reg_state32[CXL_DEVICE_CAP_REG_SIZE >> 2];
         };
     };
-    MemoryRegion device;
-    MemoryRegion mailbox;
+    struct {
+        MemoryRegion mailbox;
+        uint16_t payload_size;
+        union {
+            uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+            uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH >> 2];
+        };
+        struct {
+            uint16_t opcode;
+            uint16_t effect;
+        } cel_log[1 << 16];
+        size_t cel_size;
+    };
 
     MemoryRegion *pmem;
     MemoryRegion *vmem;
@@ -122,6 +134,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
 
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
 #define cxl_device_cap_init(dstate, reg, cap_id)                                   \
     do {                                                                           \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                             \
@@ -149,6 +164,12 @@ REG32(CXL_DEV_MAILBOX_CTRL, 4)
     FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
     FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)

include/hw/cxl/cxl_device.h REG32(CXL_DEV_MAILBOX_CTRL, 4) · line 175

Quoted 10 lines
include/hw/cxl/cxl_device.h +6 -0
 
+/* XXX: actually a 64b register */
+REG32(CXL_DEV_MAILBOX_CMD, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+    FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
 /* XXX: actually a 64b register */
 REG32(CXL_DEV_MAILBOX_STS, 0x10)
     FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)  

_______________________________________________
Linuxarm mailing list -- [email protected]
To unsubscribe send an email to [email protected]

#16 lore raw

Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

Quoted 7 lines · Ben Widawsky
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…

On Tue, 5 Jan 2021 08:52:58 -0800
Ben Widawsky <[email protected]> wrote:

[snip]

Quoted 15 lines

I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.

I think the fix should be something like this, but I can't easily test at the moment:

hw/cxl/cxl-device-utils.c +0 -0
index c515d45d20..b38e9b4c17 100644

hw/cxl/cxl-device-utils.c static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value, · line 110

Quoted 9 lines
hw/cxl/cxl-device-utils.c +3 -0
@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
 {
     CXLDeviceState *cxl_dstate = opaque;
 
+    if (offset >= A_CXL_DEV_CMD_PAYLOAD)
+        stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+
     /*
      * Lock is needed to prevent concurrent writes as well as to prevent writes
      * coming in while the firmware is processing. Without background commands

+ if (offset >= A_CXL_DEV_CMD_PAYLOAD) {
+ stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+ return;
+ }
+

[snip]

Quoted 1 line
#17 lore raw

Re: [Linuxarm] Re: [RFC PATCH v2 07/32] hw/cxl/device: Implement basic mailbox (8.2.8.4)

Quoted 36 lines · Ben Widawsky
Quoted 7 lines · Ben Widawsky
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Wed, 6 Jan 2021 13:21:23 +0000 Jonathan Cameron…

On Tue, 5 Jan 2021 08:52:58 -0800
Ben Widawsky <[email protected]> wrote:

[snip]

Quoted 24 lines

I'm sorry you had to debug this. I had fixed this previously and it got lost. I'm currently between test applications, so my regression testing isn't great.

I think the fix should be something like this, but I can't easily test at the moment:

hw/cxl/cxl-device-utils.c +3 -0
index c515d45d20..b38e9b4c17 100644
@@ -102,6 +102,9 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
 {
     CXLDeviceState *cxl_dstate = opaque;
 
+    if (offset >= A_CXL_DEV_CMD_PAYLOAD)
+        stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+
     /*
      * Lock is needed to prevent concurrent writes as well as to prevent writes
      * coming in while the firmware is processing. Without background commands

Review

Quoted 5 lines
(quoted diff) +5 -0
+    if (offset >= A_CXL_DEV_CMD_PAYLOAD) {
+        stn_le_p(cxl_dstate->mbox_reg_state, size, value);
+        return;
+    }
+

Last time's a charm

         stn_le_p(cxl_dstate->mbox_reg_state + offset, size, value);
Quoted 4 lines

[snip]

Quoted 1 line
#18 lore raw

RFC PATCH v2 8/32 hw/cxl/device: Add memory devices (8.2.8.5)

In reply to Ben Widawsky

Memory devices implement extra capabilities on top of CXL devices. This adds support for that.

A large part of memory devices is the mailbox/command interface. All of the mailbox handling is done in the mailbox-utils library. Longer term, new CXL devices that are being emulated may want to handle commands differently, and therefore would need a mechanism to opt in/out of the specific generic handlers. As such, this is considered sufficient for now, but may need more depth in the future.

---
 hw/cxl/cxl-device-utils.c   | 52 ++++++++++++++++++++++++++++++++++++-
 hw/cxl/cxl-mailbox-utils.c  | 24 +++++++++++++++++
 include/hw/cxl/cxl_device.h | 16 ++++++++++++
 3 files changed, 91 insertions(+), 1 deletion(-)
hw/cxl/cxl-device-utils.c +51 -1
index 642e3c2617..c515d45d20 100644
@@ -132,6 +132,45 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
         cxl_process_mailbox(cxl_dstate);
 }
 
+static uint64_t mdev_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+    uint64_t retval = 0;
+
+    retval = FIELD_DP64(retval, CXL_MEM_DEV_STS, MEDIA_STATUS, 1);
+    retval = FIELD_DP64(retval, CXL_MEM_DEV_STS, MBOX_READY, 1);
+
+    switch (size) {
+    case 4:
+        if (unlikely(offset & (sizeof(uint32_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return 0;
+        }
+        break;
+    case 8:
+        if (unlikely(offset & (sizeof(uint64_t) - 1))) {
+            qemu_log_mask(LOG_UNIMP, "Unaligned register read\n");
+            return 0;
+        }
+        break;
+    }
+
+    return ldn_le_p(&retval, size);
+}
+
+static const MemoryRegionOps mdev_ops = {
+    .read = mdev_reg_read,
+    .write = NULL,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static const MemoryRegionOps mailbox_ops = {
     .read = mailbox_reg_read,
     .write = mailbox_reg_write,
@@ -187,6 +226,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
                           "device-status", CXL_DEVICE_REGISTERS_LENGTH);
     memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
                           "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
+    memory_region_init_io(&cxl_dstate->memory_device, obj, &mdev_ops,
+                          cxl_dstate, "memory device caps",
+                          CXL_MEMORY_DEVICE_REGISTERS_LENGTH);
 
     memory_region_add_subregion(&cxl_dstate->device_registers, 0,
                                 &cxl_dstate->caps);
@@ -196,6 +238,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
     memory_region_add_subregion(&cxl_dstate->device_registers,
                                 CXL_MAILBOX_REGISTERS_OFFSET,
                                 &cxl_dstate->mailbox);
+    memory_region_add_subregion(&cxl_dstate->device_registers,
+                                CXL_MEMORY_DEVICE_REGISTERS_OFFSET,
+                                &cxl_dstate->memory_device);
 }
 
 static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
@@ -208,10 +253,12 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
     cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
 }
 
+static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { }
+
 void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
 {
     uint32_t *cap_hdrs = cxl_dstate->caps_reg_state32;
-    const int cap_count = 2;
+    const int cap_count = 3;
 
     /* CXL Device Capabilities Array Register */
     ARRAY_FIELD_DP32(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -224,5 +271,8 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
     cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
     mailbox_reg_init_common(cxl_dstate);
 
+    cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000);
+    memdev_reg_init_common(cxl_dstate);
+
     assert(cxl_initialize_mailbox(cxl_dstate) == 0);
 }
hw/cxl/cxl-mailbox-utils.c +24 -0
index 3a39c936de..0e94f5a95f 100644
@@ -81,6 +81,30 @@ struct cxl_cmd {
                                CXLDeviceState *cxl_dstate, uint16_t *len)
 #define declare_mailbox_handler(name) define_mailbox_handler(name)
 
+#define define_mailbox_handler_zeroed(name, size)                         \
+    uint16_t __zero##name = size;                                         \
+    static ret_code cmd_##name(struct cxl_cmd *cmd,                       \
+                               CXLDeviceState *cxl_dstate, uint16_t *len) \
+    {                                                                     \
+        *len = __zero##name;                                              \
+        memset(cmd->payload, 0, *len);                                    \
+        return CXL_MBOX_SUCCESS;                                          \
+    }
+#define define_mailbox_handler_const(name, data)                          \
+    static ret_code cmd_##name(struct cxl_cmd *cmd,                       \
+                               CXLDeviceState *cxl_dstate, uint16_t *len) \
+    {                                                                     \
+        *len = sizeof(data);                                              \
+        memcpy(cmd->payload, data, *len);                                 \
+        return CXL_MBOX_SUCCESS;                                          \
+    }
+#define define_mailbox_handler_nop(name)                                  \
+    static ret_code cmd_##name(struct cxl_cmd *cmd,                       \
+                               CXLDeviceState *cxl_dstate, uint16_t *len) \
+    {                                                                     \
+        return CXL_MBOX_SUCCESS;                                          \
+    }
+
 #define CXL_CMD(s, c, in, cel_effect) \
     [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
 
include/hw/cxl/cxl_device.h +16 -0
index bdf990cec2..dd3f4572aa 100644
@@ -69,11 +69,16 @@
 #define CXL_MAILBOX_REGISTERS_LENGTH \
     (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
 
+#define CXL_MEMORY_DEVICE_REGISTERS_OFFSET \
+    (CXL_MAILBOX_REGISTERS_OFFSET + CXL_MAILBOX_REGISTERS_LENGTH)
+#define CXL_MEMORY_DEVICE_REGISTERS_LENGTH 0x8
+
 typedef struct cxl_device_state {
     /* Main register container */
     MemoryRegion device_registers;
 
     MemoryRegion device;
+    MemoryRegion memory_device;
     struct {
         MemoryRegion caps;
         union {
@@ -133,6 +138,9 @@ REG32(CXL_DEV_CAP_ARRAY2, 4) /* We're going to pretend it's 64b */
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
 CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
                                                CXL_DEVICE_CAP_REG_SIZE)
+CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MEMORY_DEVICE,
+                                      CXL_DEVICE_CAP_HDR1_OFFSET +
+                                          CXL_DEVICE_CAP_REG_SIZE * 2)
 
 int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
@@ -185,4 +193,12 @@ REG32(CXL_DEV_BG_CMD_STS, 0x18)
 
 REG32(CXL_DEV_CMD_PAYLOAD, 0x20)
 
+/* XXX: actually a 64b registers */
+REG32(CXL_MEM_DEV_STS, 0)
+    FIELD(CXL_MEM_DEV_STS, FATAL, 0, 1)
+    FIELD(CXL_MEM_DEV_STS, FW_HALT, 1, 1)
+    FIELD(CXL_MEM_DEV_STS, MEDIA_STATUS, 2, 2)
+    FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
+    FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
+
 #endif
Signature · 1 line
2.30.0
#19 lore raw

RFC PATCH v2 9/32 hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)

In reply to Ben Widawsky

Using the previously implemented stubbed helpers, it is now possible to easily add the missing, required commands to the implementation.

---
 hw/cxl/cxl-mailbox-utils.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
hw/cxl/cxl-mailbox-utils.c +22 -1
index 0e94f5a95f..890e486ea1 100644
@@ -37,6 +37,14 @@
  *  a register interface that already deals with it.
  */
 
+enum {
+    EVENTS      = 0x01,
+        #define GET_RECORDS   0x0
+        #define CLEAR_RECORDS   0x1
+        #define GET_INTERRUPT_POLICY   0x2
+        #define SET_INTERRUPT_POLICY   0x3
+};
+
 /* 8.2.8.4.5.1 Command Return Codes */
 typedef enum {
     CXL_MBOX_SUCCESS = 0x0,
@@ -105,10 +113,23 @@ struct cxl_cmd {
         return CXL_MBOX_SUCCESS;                                          \
     }
 
+define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);
+define_mailbox_handler_nop(EVENTS_CLEAR_RECORDS);
+define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);
+define_mailbox_handler_nop(EVENTS_SET_INTERRUPT_POLICY);
+
+#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_LOG_CHANGE (1 << 4)
+
 #define CXL_CMD(s, c, in, cel_effect) \
     [s][c] = { stringify(s##_##c), cmd_##s##_##c, in, cel_effect }
 
-static struct cxl_cmd cxl_cmd_set[256][256] = {};
+static struct cxl_cmd cxl_cmd_set[256][256] = {
+    CXL_CMD(EVENTS, GET_RECORDS, 1, 0),
+    CXL_CMD(EVENTS, CLEAR_RECORDS, ~0, IMMEDIATE_LOG_CHANGE),
+    CXL_CMD(EVENTS, GET_INTERRUPT_POLICY, 0, 0),
+    CXL_CMD(EVENTS, SET_INTERRUPT_POLICY, 4, IMMEDIATE_CONFIG_CHANGE),
+};
 
 #undef CXL_CMD
 
Signature · 1 line
2.30.0
#21 lore raw

RFC PATCH v2 11/32 hw/cxl/device: Timestamp implementation (8.2.9.3)

In reply to Ben Widawsky

Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.

---
 hw/cxl/cxl-mailbox-utils.c  | 52 +++++++++++++++++++++++++++++++++++++
 include/hw/cxl/cxl_device.h |  6 +++++
 2 files changed, 58 insertions(+)
hw/cxl/cxl-mailbox-utils.c +52 -0
index 890e486ea1..8b956d9e94 100644
@@ -43,6 +43,9 @@ enum {
         #define CLEAR_RECORDS   0x1
         #define GET_INTERRUPT_POLICY   0x2
         #define SET_INTERRUPT_POLICY   0x3
+    TIMESTAMP   = 0x03,
+        #define GET           0x0
+        #define SET           0x1
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);
 define_mailbox_handler_nop(EVENTS_CLEAR_RECORDS);
 define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);
 define_mailbox_handler_nop(EVENTS_SET_INTERRUPT_POLICY);
+declare_mailbox_handler(TIMESTAMP_GET);
+declare_mailbox_handler(TIMESTAMP_SET);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_POLICY_CHANGE (1 << 3)
 #define IMMEDIATE_LOG_CHANGE (1 << 4)
 
 #define CXL_CMD(s, c, in, cel_effect) \
@@ -129,10 +135,56 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(EVENTS, CLEAR_RECORDS, ~0, IMMEDIATE_LOG_CHANGE),
     CXL_CMD(EVENTS, GET_INTERRUPT_POLICY, 0, 0),
     CXL_CMD(EVENTS, SET_INTERRUPT_POLICY, 4, IMMEDIATE_CONFIG_CHANGE),
+    CXL_CMD(TIMESTAMP, GET, 0, 0),
+    CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
 };
 
 #undef CXL_CMD
 
+/*
+ * 8.2.9.3.1
+ */
+define_mailbox_handler(TIMESTAMP_GET)
+{
+    struct timespec ts;
+
+    clock_gettime(CLOCK_REALTIME, &ts);
+
+    cxl_dstate->timestamp.set = true;
+    cxl_dstate->timestamp.last_set =
+        ts.tv_sec * NANOSECONDS_PER_SECOND + ts.tv_nsec;
+    cxl_dstate->timestamp.host_set = *(uint64_t *)cmd->payload;
+
+    *len = 8;
+    return CXL_MBOX_SUCCESS;
+}
+
+/*
+ * 8.2.9.3.2
+ */
+define_mailbox_handler(TIMESTAMP_SET)
+{
+    struct timespec ts;
+    uint64_t delta;
+
+    if (!cxl_dstate->timestamp.set) {
+        *(uint64_t *)cmd->payload = 0;
+        goto done;
+    }
+
+    /* First find the delta from the last time the host set the time. */
+    clock_gettime(CLOCK_REALTIME, &ts);
+    delta = (ts.tv_sec * NANOSECONDS_PER_SECOND + ts.tv_nsec) -
+            cxl_dstate->timestamp.host_set;
+
+    /* Then adjust the actual time */
+    *(uint64_t *)cmd->payload = cxl_dstate->timestamp.host_set + delta;
+
+done:
+    *len = 8;
+    return CXL_MBOX_SUCCESS;
+}
+
 QemuUUID cel_uuid;
 
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
include/hw/cxl/cxl_device.h +6 -0
index dd3f4572aa..25974b2416 100644
@@ -100,6 +100,12 @@ typedef struct cxl_device_state {
         size_t cel_size;
     };
 
+    struct {
+        bool set;
+        uint64_t last_set;
+        uint64_t host_set;
+    } timestamp;
+
     MemoryRegion *pmem;
     MemoryRegion *vmem;
 } CXLDeviceState;
Signature · 1 line
2.30.0
#22 lore raw

Re: [RFC PATCH v2 11/32] hw/cxl/device: Timestamp implementation (8.2.9.3)

Quoted 82 lines · Ben Widawsky

Per spec, timestamp appears to be a free-running counter from a value set by the host via the Set Timestamp command (0301h). There are references to the epoch, which seem like a red herring. Therefore, the implementation implements the timestamp as freerunning counter from the last value that was issued by the Set Timestamp command.

---
 hw/cxl/cxl-mailbox-utils.c  | 52 +++++++++++++++++++++++++++++++++++++
 include/hw/cxl/cxl_device.h |  6 +++++
 2 files changed, 58 insertions(+)
hw/cxl/cxl-mailbox-utils.c +42 -0
index 890e486ea1..8b956d9e94 100644
@@ -43,6 +43,9 @@ enum {
         #define CLEAR_RECORDS   0x1
         #define GET_INTERRUPT_POLICY   0x2
         #define SET_INTERRUPT_POLICY   0x3
+    TIMESTAMP   = 0x03,
+        #define GET           0x0
+        #define SET           0x1
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -117,8 +120,11 @@ define_mailbox_handler_zeroed(EVENTS_GET_RECORDS, 0x20);
 define_mailbox_handler_nop(EVENTS_CLEAR_RECORDS);
 define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);
 define_mailbox_handler_nop(EVENTS_SET_INTERRUPT_POLICY);
+declare_mailbox_handler(TIMESTAMP_GET);
+declare_mailbox_handler(TIMESTAMP_SET);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_POLICY_CHANGE (1 << 3)
 #define IMMEDIATE_LOG_CHANGE (1 << 4)
 
 #define CXL_CMD(s, c, in, cel_effect) \
@@ -129,10 +135,56 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(EVENTS, CLEAR_RECORDS, ~0, IMMEDIATE_LOG_CHANGE),
     CXL_CMD(EVENTS, GET_INTERRUPT_POLICY, 0, 0),
     CXL_CMD(EVENTS, SET_INTERRUPT_POLICY, 4, IMMEDIATE_CONFIG_CHANGE),
+    CXL_CMD(TIMESTAMP, GET, 0, 0),
+    CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
 };
 
 #undef CXL_CMD
 
+/*
+ * 8.2.9.3.1
+ */
+define_mailbox_handler(TIMESTAMP_GET)
+{
+    struct timespec ts;
+
+    clock_gettime(CLOCK_REALTIME, &ts);
+
+    cxl_dstate->timestamp.set = true;
+    cxl_dstate->timestamp.last_set =
+        ts.tv_sec * NANOSECONDS_PER_SECOND + ts.tv_nsec;
+    cxl_dstate->timestamp.host_set = *(uint64_t *)cmd->payload;
+
+    *len = 8;
+    return CXL_MBOX_SUCCESS;
+}
+
+/*
+ * 8.2.9.3.2
+ */
+define_mailbox_handler(TIMESTAMP_SET)
+{
+    struct timespec ts;
+    uint64_t delta;
+
+    if (!cxl_dstate->timestamp.set) {
+        *(uint64_t *)cmd->payload = 0;
+        goto done;
+    }
+
+    /* First find the delta from the last time the host set the time. */
+    clock_gettime(CLOCK_REALTIME, &ts);
+    delta = (ts.tv_sec * NANOSECONDS_PER_SECOND + ts.tv_nsec) -

hw/cxl/cxl-mailbox-utils.c static struct cxl_cmd cxl_cmd_set[256][256] = { · line 187

Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10 -0
+            cxl_dstate->timestamp.host_set;
+
+    /* Then adjust the actual time */
+    *(uint64_t *)cmd->payload = cxl_dstate->timestamp.host_set + delta;
+
+done:
+    *len = 8;
+    return CXL_MBOX_SUCCESS;
+}
+

Set/get are reversed here. I missed the fixup before sending.

Quoted 23 lines
hw/cxl/cxl-mailbox-utils.c +0 -0
 QemuUUID cel_uuid;
 
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
include/hw/cxl/cxl_device.h +6 -0
index dd3f4572aa..25974b2416 100644
@@ -100,6 +100,12 @@ typedef struct cxl_device_state {
         size_t cel_size;
     };
 
+    struct {
+        bool set;
+        uint64_t last_set;
+        uint64_t host_set;
+    } timestamp;
+
     MemoryRegion *pmem;
     MemoryRegion *vmem;
 } CXLDeviceState;
Signature · 1 line
2.30.0
#23 lore raw

RFC PATCH v2 12/32 hw/cxl/device: Add log commands (8.2.9.4) + CEL

In reply to Ben Widawsky

CXL specification provides for the ability to obtain logs from the device. Logs are either spec defined, like the "Command Effects Log" (CEL), or vendor specific. UUIDs are defined for all log types.

The CEL is a mechanism to provide information to the host about which commands are supported. It is useful both to determine which spec'd optional commands are supported, as well as provide a list of vendor specified commands that might be used. The CEL is already created as part of mailbox initialization, but here it is now exported to hosts that use these log commands.

---
 hw/cxl/cxl-mailbox-utils.c | 75 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
hw/cxl/cxl-mailbox-utils.c +75 -0
index 8b956d9e94..f68ec5b5b9 100644
@@ -46,6 +46,9 @@ enum {
     TIMESTAMP   = 0x03,
         #define GET           0x0
         #define SET           0x1
+    LOGS        = 0x04,
+        #define GET_SUPPORTED 0x0
+        #define GET_LOG       0x1
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -122,6 +125,8 @@ define_mailbox_handler_zeroed(EVENTS_GET_INTERRUPT_POLICY, 4);
 define_mailbox_handler_nop(EVENTS_SET_INTERRUPT_POLICY);
 declare_mailbox_handler(TIMESTAMP_GET);
 declare_mailbox_handler(TIMESTAMP_SET);
+declare_mailbox_handler(LOGS_GET_SUPPORTED);
+declare_mailbox_handler(LOGS_GET_LOG);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
 #define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -137,6 +142,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(EVENTS, SET_INTERRUPT_POLICY, 4, IMMEDIATE_CONFIG_CHANGE),
     CXL_CMD(TIMESTAMP, GET, 0, 0),
     CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
+    CXL_CMD(LOGS, GET_SUPPORTED, 0, 0),
+    CXL_CMD(LOGS, GET_LOG, 0x18, 0),
 };
 
 #undef CXL_CMD
@@ -187,6 +194,74 @@ done:
 
 QemuUUID cel_uuid;
 
+/* 8.2.9.4.1 */
+define_mailbox_handler(LOGS_GET_SUPPORTED)
+{
+    struct {
+        uint16_t entries;
+        uint8_t rsvd[6];
+        struct {
+            QemuUUID uuid;
+            uint32_t size;
+        } log_entries[1];
+    } __attribute__((packed)) *supported_logs;
+    _Static_assert(sizeof(*supported_logs) == 0x1c, "Bad supported log size");
+
+    supported_logs = (void *)cmd->payload;
+
+    supported_logs->entries = 1;
+    supported_logs->log_entries[0].uuid = cel_uuid;
+    supported_logs->log_entries[0].size = 4 * cxl_dstate->cel_size;
+
+    *len = sizeof(*supported_logs);
+    return CXL_MBOX_SUCCESS;
+}
+
+/* 8.2.9.4.2 */
+define_mailbox_handler(LOGS_GET_LOG)
+{
+    struct {
+        union {
+            struct {
+                QemuUUID uuid;
+                uint32_t offset;
+                uint32_t length;
+            };
+        };
+    } __attribute__((packed, __aligned__(16))) *get_log;
+
+    get_log = (void *)cmd->payload;
+
+    /*
+     * 8.2.9.4.2
+     *   The device shall return Invalid Parameter if the Offset or Length
+     *   fields attempt to access beyond the size of the log as reported by Get
+     *   Supported Logs.
+     *
+     * XXX: Spec is wrong, "Invalid Parameter" isn't a thing.
+     * XXX: Spec doesn't address incorrect UUID incorrectness.
+     *
+     * The CEL buffer is large enough to fit all commands in the emulation, so
+     * the only possible failure would be if the mailbox itself isn't big
+     * enough.
+     */
+    if (get_log->offset + get_log->length > cxl_dstate->payload_size) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    if (!qemu_uuid_is_equal(&get_log->uuid, &cel_uuid)) {
+        return CXL_MBOX_UNSUPPORTED;
+    }
+
+    /* Store off everything to local variables so we can wipe out the payload */
+    *len = get_log->length;
+
+    memmove(cmd->payload, cxl_dstate->cel_log + get_log->offset,
+           get_log->length);
+
+    return CXL_MBOX_SUCCESS;
+}
+
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
 {
     uint16_t ret = CXL_MBOX_SUCCESS;
Signature · 1 line
2.30.0
#24 lore raw

RFC PATCH v2 13/32 hw/pxb: Use a type for realizing expanders

In reply to Ben Widawsky

This opens up the possibility for more types of expanders (other than PCI and PCIe). We'll need this to create a CXL expander.

---
 hw/pci-bridge/pci_expander_bridge.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +7 -4
index aedded1064..232b7ce305 100644
@@ -24,6 +24,8 @@
 #include "hw/boards.h"
 #include "qom/object.h"
 
+enum BusType { PCI, PCIE };
+
 #define TYPE_PXB_BUS "pxb-bus"
 typedef struct PXBBus PXBBus;
 DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
@@ -214,7 +216,8 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)
            0;
 }
 
-static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp)
+static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
+                                   Error **errp)
 {
     PXBDev *pxb = convert_to_pxb(dev);
     DeviceState *ds, *bds = NULL;
@@ -239,7 +242,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp)
     }
 
     ds = qdev_new(TYPE_PXB_HOST);
-    if (pcie) {
+    if (type == PCIE) {
         bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
     } else {
         bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
@@ -287,7 +290,7 @@ static void pxb_dev_realize(PCIDevice *dev, Error **errp)
         return;
     }
 
-    pxb_dev_realize_common(dev, false, errp);
+    pxb_dev_realize_common(dev, PCI, errp);
 }
 
 static void pxb_dev_exitfn(PCIDevice *pci_dev)
@@ -339,7 +342,7 @@ static void pxb_pcie_dev_realize(PCIDevice *dev, Error **errp)
         return;
     }
 
-    pxb_dev_realize_common(dev, true, errp);
+    pxb_dev_realize_common(dev, PCIE, errp);
 }
 
 static void pxb_pcie_dev_class_init(ObjectClass *klass, void *data)
Signature · 1 line
2.30.0
#25 lore raw

RFC PATCH v2 14/32 hw/pci/cxl: Create a CXL bus type

In reply to Ben Widawsky

The easiest way to differentiate a CXL bus, and a PCIE bus is using a flag. A CXL bus, in hardware, is backward compatible with PCIE, and therefore the code tries pretty hard to keep them in sync as much as possible.

The other way to implement this would be to try to cast the bus to the correct type. This is less code and useful for debugging via simply looking at the flags.

---
 hw/pci-bridge/pci_expander_bridge.c | 9 ++++++++-
 include/hw/pci/pci_bus.h            | 7 +++++++
 2 files changed, 15 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +8 -1
index 232b7ce305..88c45dc3b5 100644
@@ -24,7 +24,7 @@
 #include "hw/boards.h"
 #include "qom/object.h"
 
-enum BusType { PCI, PCIE };
+enum BusType { PCI, PCIE, CXL };
 
 #define TYPE_PXB_BUS "pxb-bus"
 typedef struct PXBBus PXBBus;
@@ -35,6 +35,10 @@ DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
 DECLARE_INSTANCE_CHECKER(PXBBus, PXB_PCIE_BUS,
                          TYPE_PXB_PCIE_BUS)
 
+#define TYPE_PXB_CXL_BUS "pxb-cxl-bus"
+DECLARE_INSTANCE_CHECKER(PXBBus, PXB_CXL_BUS,
+                         TYPE_PXB_CXL_BUS)
+
 struct PXBBus {
     /*< private >*/
     PCIBus parent_obj;
@@ -244,6 +248,9 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
     ds = qdev_new(TYPE_PXB_HOST);
     if (type == PCIE) {
         bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
+    } else if (type == CXL) {
+        bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_CXL_BUS);
+        bus->flags |= PCI_BUS_CXL;
     } else {
         bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
         bds = qdev_new("pci-bridge");
include/hw/pci/pci_bus.h +7 -0
index 347440d42c..eb94e7e85c 100644
@@ -24,6 +24,8 @@ enum PCIBusFlags {
     PCI_BUS_IS_ROOT                                         = 0x0001,
     /* PCIe extended configuration space is accessible on this bus */
     PCI_BUS_EXTENDED_CONFIG_SPACE                           = 0x0002,
+    /* This is a CXL Type BUS */
+    PCI_BUS_CXL                                             = 0x0004,
 };
 
 struct PCIBus {
@@ -53,6 +55,11 @@ struct PCIBus {
     Notifier machine_done;
 };
 
+static inline bool pci_bus_is_cxl(PCIBus *bus)
+{
+    return !!(bus->flags & PCI_BUS_CXL);
+}
+
 static inline bool pci_bus_is_root(PCIBus *bus)
 {
     return !!(bus->flags & PCI_BUS_IS_ROOT);
Signature · 1 line
2.30.0
#26 lore raw

RFC PATCH v2 15/32 hw/pxb: Allow creation of a CXL PXB (host bridge)

In reply to Ben Widawsky

This works like adding a typical pxb device, except the name is 'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1

A CXL PXB is backward compatible with PCIe. What this means in practice is that an operating system that is unaware of CXL should still be able to enumerate this topology as if it were PCIe.

One can create multiple CXL PXB host bridges, but a host bridge can only be connected to the main root bus. Host bridges cannot appear elsewhere in the topology.

Note that as of this patch, the ACPI tables needed for the host bridge (specifically, an ACPI object in _SB named ACPI0016 and the CEDT) aren't created. So while this patch internally creates it, it cannot be properly used by an operating system or other system software.

Upcoming patches will allow creating multiple host bridges.

v2: Remove vendor and device ID (Ben)

---
 hw/pci-bridge/pci_expander_bridge.c | 67 ++++++++++++++++++++++++++++-
 hw/pci/pci.c                        |  7 +++
 include/hw/pci/pci.h                |  6 +++
 3 files changed, 78 insertions(+), 2 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +65 -2
index 88c45dc3b5..b42592e1ff 100644
@@ -56,6 +56,10 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,
 DECLARE_INSTANCE_CHECKER(PXBDev, PXB_PCIE_DEV,
                          TYPE_PXB_PCIE_DEVICE)
 
+#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
+DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
+                         TYPE_PXB_CXL_DEVICE)
+
 struct PXBDev {
     /*< private >*/
     PCIDevice parent_obj;
@@ -67,6 +71,11 @@ struct PXBDev {
 
 static PXBDev *convert_to_pxb(PCIDevice *dev)
 {
+    /* A CXL PXB's parent bus is PCIe, so the normal check won't work */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_PXB_CXL_DEVICE)) {
+        return PXB_CXL_DEV(dev);
+    }
+
     return pci_bus_is_express(pci_get_bus(dev))
         ? PXB_PCIE_DEV(dev) : PXB_DEV(dev);
 }
@@ -111,11 +120,20 @@ static const TypeInfo pxb_pcie_bus_info = {
     .class_init    = pxb_bus_class_init,
 };
 
+static const TypeInfo pxb_cxl_bus_info = {
+    .name          = TYPE_PXB_CXL_BUS,
+    .parent        = TYPE_CXL_BUS,
+    .instance_size = sizeof(PXBBus),
+    .class_init    = pxb_bus_class_init,
+};
+
 static const char *pxb_host_root_bus_path(PCIHostState *host_bridge,
                                           PCIBus *rootbus)
 {
-    PXBBus *bus = pci_bus_is_express(rootbus) ?
-                  PXB_PCIE_BUS(rootbus) : PXB_BUS(rootbus);
+    PXBBus *bus = pci_bus_is_cxl(rootbus) ?
+                      PXB_CXL_BUS(rootbus) :
+                      pci_bus_is_express(rootbus) ? PXB_PCIE_BUS(rootbus) :
+                                                    PXB_BUS(rootbus);
 
     snprintf(bus->bus_path, 8, "0000:%02x", pxb_bus_num(rootbus));
     return bus->bus_path;
@@ -380,13 +398,58 @@ static const TypeInfo pxb_pcie_dev_info = {
     },
 };
 
+static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
+{
+    /* A CXL PXB's parent bus is still PCIe */
+    if (!pci_bus_is_express(pci_get_bus(dev))) {
+        error_setg(errp, "pxb-cxl devices cannot reside on a PCI bus");
+        return;
+    }
+
+    pxb_dev_realize_common(dev, CXL, errp);
+}
+
+static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc   = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->realize             = pxb_cxl_dev_realize;
+    k->exit                = pxb_dev_exitfn;
+    /*
+     * XXX: These types of bridges don't actually show up in the hierarchy so
+     * vendor, device, class, etc. ids are intentionally left out.
+     */
+
+    dc->desc = "CXL Host Bridge";
+    device_class_set_props(dc, pxb_dev_properties);
+    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+
+    /* Host bridges aren't hotpluggable. FIXME: spec reference */
+    dc->hotpluggable = false;
+}
+
+static const TypeInfo pxb_cxl_dev_info = {
+    .name          = TYPE_PXB_CXL_DEVICE,
+    .parent        = TYPE_PCI_DEVICE,
+    .instance_size = sizeof(PXBDev),
+    .class_init    = pxb_cxl_dev_class_init,
+    .interfaces =
+        (InterfaceInfo[]){
+            { INTERFACE_CONVENTIONAL_PCI_DEVICE },
+            {},
+        },
+};
+
 static void pxb_register_types(void)
 {
     type_register_static(&pxb_bus_info);
     type_register_static(&pxb_pcie_bus_info);
+    type_register_static(&pxb_cxl_bus_info);
     type_register_static(&pxb_host_info);
     type_register_static(&pxb_dev_info);
     type_register_static(&pxb_pcie_dev_info);
+    type_register_static(&pxb_cxl_dev_info);
 }
 
 type_init(pxb_register_types)
hw/pci/pci.c +7 -0
index 50fc61c81f..fb454e2317 100644
@@ -222,6 +222,12 @@ static const TypeInfo pcie_bus_info = {
     .class_init = pcie_bus_class_init,
 };
 
+static const TypeInfo cxl_bus_info = {
+    .name       = TYPE_CXL_BUS,
+    .parent     = TYPE_PCIE_BUS,
+    .class_init = pcie_bus_class_init,
+};
+
 static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
 static void pci_update_mappings(PCIDevice *d);
 static void pci_irq_handler(void *opaque, int irq_num, int level);
@@ -2849,6 +2855,7 @@ static void pci_register_types(void)
 {
     type_register_static(&pci_bus_info);
     type_register_static(&pcie_bus_info);
+    type_register_static(&cxl_bus_info);
     type_register_static(&conventional_pci_interface_info);
     type_register_static(&cxl_interface_info);
     type_register_static(&pcie_interface_info);
include/hw/pci/pci.h +6 -0
index f6b1eff446..c464597e53 100644
@@ -405,6 +405,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);
 #define TYPE_PCI_BUS "PCI"
 OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)
 #define TYPE_PCIE_BUS "PCIE"
+#define TYPE_CXL_BUS "CXL"
 
 bool pci_bus_is_express(PCIBus *bus);
 
@@ -753,6 +754,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)
     pci_irq_deassert(pci_dev);
 }
 
+static inline int pci_is_cxl(const PCIDevice *d)
+{
+    return d->cap_present & QEMU_PCIE_CAP_CXL;
+}
+
 static inline int pci_is_express(const PCIDevice *d)
 {
     return d->cap_present & QEMU_PCI_CAP_EXPRESS;
Signature · 1 line
2.30.0
#27 lore raw

RFC PATCH v2 16/32 qtest: allow DSDT acpi table changes

In reply to Ben Widawsky
---
 tests/qtest/bios-tables-test-allowed-diff.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
tests/qtest/bios-tables-test-allowed-diff.h +21 -0
index dfb8523c8b..5c695cdf37 100644
@@ -1 +1,22 @@
 /* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/pc/DSDT",
+"tests/data/acpi/pc/DSDT.acpihmat",
+"tests/data/acpi/pc/DSDT.bridge",
+"tests/data/acpi/pc/DSDT.cphp",
+"tests/data/acpi/pc/DSDT.dimmpxm",
+"tests/data/acpi/pc/DSDT.hpbridge",
+"tests/data/acpi/pc/DSDT.hpbrroot",
+"tests/data/acpi/pc/DSDT.ipmikcs",
+"tests/data/acpi/pc/DSDT.memhp",
+"tests/data/acpi/pc/DSDT.numamem",
+"tests/data/acpi/pc/DSDT.roothp",
+"tests/data/acpi/q35/DSDT",
+"tests/data/acpi/q35/DSDT.acpihmat",
+"tests/data/acpi/q35/DSDT.bridge",
+"tests/data/acpi/q35/DSDT.cphp",
+"tests/data/acpi/q35/DSDT.dimmpxm",
+"tests/data/acpi/q35/DSDT.ipmibt",
+"tests/data/acpi/q35/DSDT.memhp",
+"tests/data/acpi/q35/DSDT.mmio64",
+"tests/data/acpi/q35/DSDT.numamem",
+"tests/data/acpi/q35/DSDT.tis",
Signature · 1 line
2.30.0
#28 lore raw

RFC PATCH v2 17/32 acpi/pci: Consolidate host bridge setup

In reply to Ben Widawsky

This cleanup will make it easier to add support for CXL to the mix.

---
 hw/i386/acpi-build.c | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)
hw/i386/acpi-build.c +17 -14
index f18b71dea9..3c214bba67 100644
@@ -1194,6 +1194,20 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)
     aml_append(table, scope);
 }
 
+enum { PCI, PCIE };
+static void init_pci_acpi(Aml *dev, int uid, int type)
+{
+    if (type == PCI) {
+        aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
+        aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+    } else {
+        aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
+        aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
+        aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+        aml_append(dev, build_q35_osc_method());
+    }
+}
+
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker,
            AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1222,9 +1236,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
     if (misc->is_piix4) {
         sb_scope = aml_scope("_SB");
         dev = aml_device("PCI0");
-        aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
+        init_pci_acpi(dev, 0, PCI);
         aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
-        aml_append(dev, aml_name_decl("_UID", aml_int(0)));
         aml_append(sb_scope, dev);
         aml_append(dsdt, sb_scope);
 
@@ -1238,11 +1251,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
     } else {
         sb_scope = aml_scope("_SB");
         dev = aml_device("PCI0");
-        aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
-        aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
+        init_pci_acpi(dev, 0, PCIE);
         aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
-        aml_append(dev, aml_name_decl("_UID", aml_int(0)));
-        aml_append(dev, build_q35_osc_method());
         aml_append(sb_scope, dev);
 
         if (pm->smi_on_cpuhp) {
@@ -1345,15 +1355,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
 
             scope = aml_scope("\\_SB");
             dev = aml_device("PC%.02X", bus_num);
-            aml_append(dev, aml_name_decl("_UID", aml_int(bus_num)));
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
-            if (pci_bus_is_express(bus)) {
-                aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
-                aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
-                aml_append(dev, build_q35_osc_method());
-            } else {
-                aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
-            }
+            init_pci_acpi(dev, bus_num, pci_bus_is_express(bus) ? PCIE : PCI);
 
             if (numa_node != NUMA_NODE_UNASSIGNED) {
                 aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
Signature · 1 line
2.30.0
#29 lore raw

RFC PATCH v2 18/32 tests/acpi: remove stale allowed tables

In reply to Ben Widawsky
     Scope (_SB)
     {
         Device (PCI0)
         {
             Name (_HID, EisaId ("PNP0A03") /* PCI Bus */)  // _HID: Hardware ID
-            Name (_ADR, Zero)  // _ADR: Address
             Name (_UID, Zero)  // _UID: Unique ID
+            Name (_ADR, Zero)  // _ADR: Address
---
 tests/data/acpi/pc/DSDT                     | Bin 5065 -> 5065 bytes
 tests/data/acpi/pc/DSDT.acpihmat            | Bin 6390 -> 6390 bytes
 tests/data/acpi/pc/DSDT.bridge              | Bin 6924 -> 6924 bytes
 tests/data/acpi/pc/DSDT.cphp                | Bin 5529 -> 5529 bytes
 tests/data/acpi/pc/DSDT.dimmpxm             | Bin 6719 -> 6719 bytes
 tests/data/acpi/pc/DSDT.hpbridge            | Bin 5026 -> 5026 bytes
 tests/data/acpi/pc/DSDT.hpbrroot            | Bin 3084 -> 3084 bytes
 tests/data/acpi/pc/DSDT.ipmikcs             | Bin 5137 -> 5137 bytes
 tests/data/acpi/pc/DSDT.memhp               | Bin 6424 -> 6424 bytes
 tests/data/acpi/pc/DSDT.numamem             | Bin 5071 -> 5071 bytes
 tests/data/acpi/pc/DSDT.roothp              | Bin 5261 -> 5261 bytes
 tests/data/acpi/q35/DSDT                    | Bin 7801 -> 7801 bytes
 tests/data/acpi/q35/DSDT.acpihmat           | Bin 9126 -> 9126 bytes
 tests/data/acpi/q35/DSDT.bridge             | Bin 7819 -> 7819 bytes
 tests/data/acpi/q35/DSDT.cphp               | Bin 8265 -> 8265 bytes
 tests/data/acpi/q35/DSDT.dimmpxm            | Bin 9455 -> 9455 bytes
 tests/data/acpi/q35/DSDT.ipmibt             | Bin 7876 -> 7876 bytes
 tests/data/acpi/q35/DSDT.memhp              | Bin 9160 -> 9160 bytes
 tests/data/acpi/q35/DSDT.mmio64             | Bin 8932 -> 8932 bytes
 tests/data/acpi/q35/DSDT.numamem            | Bin 7807 -> 7807 bytes
 tests/qtest/bios-tables-test-allowed-diff.h |  21 --------------------
 21 files changed, 21 deletions(-)
tests/data/acpi/pc/DSDT binary +0 -0
index f6173df1d598767a79aa34ad7585ad7d45c5d4f3..b516745128e3f1a297b6327e9057026a2d16229c 100644
GIT binary patch
tests/data/acpi/pc/DSDT.acpihmat binary +0 -0
index 67f3f7249eaaa9404ebf0f2d0a324b8c8e3bd445..aeae285c6434ae6cf3c53660e34425727a497871 100644
GIT binary patch
tests/data/acpi/pc/DSDT.bridge binary +0 -0
index 643390f4c4138b37fc481656d3f555d0eeedcb02..4cd26a87dd11d96e10bf6de786b9d56ebfe0a4f9 100644
GIT binary patch
tests/data/acpi/pc/DSDT.cphp binary +0 -0
index 1ddcf7d8812f5d8d4d38fe7e7b35fd5885806046..fecb784812cbb2308ef58acf4a2c580f56d35c39 100644
GIT binary patch
tests/data/acpi/pc/DSDT.dimmpxm binary +0 -0
index c44385cc01879324738ffb7f997b8cdd762cbf97..f2c31e150ead16e4931367a6dab42704950a21e9 100644
GIT binary patch
tests/data/acpi/pc/DSDT.hpbridge binary +0 -0
index 4ecf1eb13bf49499f729b53a6d0114672a76e28d..7a8955cdbc52c025a2fd8f160cf8aff9442c985b 100644
GIT binary patch
tests/data/acpi/pc/DSDT.hpbrroot binary +0 -0
index a3046226ec1dcb234b726029b3790dfedb3b9221..88d23fca4743c2ee57493e7d77d6297a60964d3c 100644
GIT binary patch
tests/data/acpi/pc/DSDT.ipmikcs binary +0 -0
index f1638c5d079a9442c09390426a913010df6efd8d..d670ae793b5778c095a7f8c79ff1a046889d1a56 100644
GIT binary patch
tests/data/acpi/pc/DSDT.memhp binary +0 -0
index 4c19e45e66918c61674785c99e4474e58866f125..a7de3d9fd94e62e8fc357fe3093bf7f394a39219 100644
GIT binary patch
tests/data/acpi/pc/DSDT.numamem binary +0 -0
index 40cfd933259af05ac2aee07fca32f22122255211..57958b6cec216c1fb8731f4ed2da67f0fad7484a 100644
GIT binary patch
tests/data/acpi/pc/DSDT.roothp binary +0 -0
index 078fc8031b479cc77b6527a2b7b4bd576b6e6028..624d0e367693fe267a4237a5fc97295cee2ebd60 100644
GIT binary patch
tests/data/acpi/q35/DSDT binary +0 -0
index d25cd7072932886d6967f4023faac1e1fa6e836c..17e2aebde98e0a3161d93e9b2e200737b13699ac 100644
GIT binary patch
tests/data/acpi/q35/DSDT.acpihmat binary +0 -0
index 722e06af83abcde203a2b96a8ec81fd3bab9fc98..7b3d659352a0923822f6a5db1dbd0a6ad853c446 100644
GIT binary patch
tests/data/acpi/q35/DSDT.bridge binary +0 -0
index 06bac139d668ddfc7914e258b471a303c9dbd192..5961b55b1067c3090b2f1f4cd3386d71efee241d 100644
GIT binary patch
tests/data/acpi/q35/DSDT.cphp binary +0 -0
index 2b933ac482e6883efccbd7d6c96089602f2c0b4d..09c92d52f92bb346ed807945b9638cad958446f8 100644
GIT binary patch
tests/data/acpi/q35/DSDT.dimmpxm binary +0 -0
index bd8f8305b028ef20f9b6d1a0c69ac428d027e3d1..1da97afb32dddafefe7f27934acbcb7d56a67489 100644
GIT binary patch
tests/data/acpi/q35/DSDT.ipmibt binary +0 -0
index a8f868e23c25688ab1c0371016c071f23e9d732f..c7e68432b66e7b4d03284c882c65bbf3066825dc 100644
GIT binary patch
tests/data/acpi/q35/DSDT.memhp binary +0 -0
index 9a802e4c67022386442976d5cb997ea3fc57b58f..3af457dd550461b2d2ea85aa85d7740452913b34 100644
GIT binary patch
tests/data/acpi/q35/DSDT.mmio64 binary +0 -0
index 948c2dc7264c31932b490ca00691a7c4d9aefdb0..a4d20f676ac173e6846dcd4e076220d512215963 100644
GIT binary patch
tests/data/acpi/q35/DSDT.numamem binary +0 -0
index 44ec1b0af400da6d298284aa959aa38add7e6dd5..bbab0d10a2a064528519fa69e90c799430129b75 100644
GIT binary patch
tests/qtest/bios-tables-test-allowed-diff.h +0 -21
index 5c695cdf37..dfb8523c8b 100644
@@ -1,22 +1 @@
 /* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/pc/DSDT",
-"tests/data/acpi/pc/DSDT.acpihmat",
-"tests/data/acpi/pc/DSDT.bridge",
-"tests/data/acpi/pc/DSDT.cphp",
-"tests/data/acpi/pc/DSDT.dimmpxm",
-"tests/data/acpi/pc/DSDT.hpbridge",
-"tests/data/acpi/pc/DSDT.hpbrroot",
-"tests/data/acpi/pc/DSDT.ipmikcs",
-"tests/data/acpi/pc/DSDT.memhp",
-"tests/data/acpi/pc/DSDT.numamem",
-"tests/data/acpi/pc/DSDT.roothp",
-"tests/data/acpi/q35/DSDT",
-"tests/data/acpi/q35/DSDT.acpihmat",
-"tests/data/acpi/q35/DSDT.bridge",
-"tests/data/acpi/q35/DSDT.cphp",
-"tests/data/acpi/q35/DSDT.dimmpxm",
-"tests/data/acpi/q35/DSDT.ipmibt",
-"tests/data/acpi/q35/DSDT.memhp",
-"tests/data/acpi/q35/DSDT.mmio64",
-"tests/data/acpi/q35/DSDT.numamem",
-"tests/data/acpi/q35/DSDT.tis",
Signature · 1 line
2.30.0
#30 lore raw

RFC PATCH v2 19/32 hw/pci: Plumb _UID through host bridges

In reply to Ben Widawsky

Currently, QEMU makes _UID equivalent to the bus number (_BBN). While there is nothing wrong with doing it this way, CXL spec has a heavy reliance on _UID to identify host bridges and there is no link to the bus number. Having a distinct UID solves two problems. The first is it gets us around the limitation of 256 (current max bus number). The second is it allows us to replicate hardware configurations where bus number and uid aren't equivalent. The latter has benefits for our development and debugging using QEMU.

The other way to do this would be to implement the expanded bus numbering, but having an explicit uid makes more sense when trying to replicate real hardware configurations.

The QEMU commandline to utilize this would be:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1,uid=x

--

I'm guessing this patch will be somewhat controversial. For early CXL work, this can be dropped without too much heartache.

---
 hw/i386/acpi-build.c                |  3 ++-
 hw/pci-bridge/pci_expander_bridge.c | 19 +++++++++++++++++++
 hw/pci/pci.c                        | 11 +++++++++++
 include/hw/pci/pci.h                |  1 +
 include/hw/pci/pci_bus.h            |  1 +
 5 files changed, 34 insertions(+), 1 deletion(-)
hw/i386/acpi-build.c +2 -1
index 3c214bba67..26e4ddd025 100644
@@ -1343,6 +1343,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
         QLIST_FOREACH(bus, &bus->child, sibling) {
             uint8_t bus_num = pci_bus_num(bus);
             uint8_t numa_node = pci_bus_numa_node(bus);
+            int32_t uid = pci_bus_uid(bus);
 
             /* look only for expander root buses */
             if (!pci_bus_is_root(bus)) {
@@ -1356,7 +1357,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
             scope = aml_scope("\\_SB");
             dev = aml_device("PC%.02X", bus_num);
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
-            init_pci_acpi(dev, bus_num, pci_bus_is_express(bus) ? PCIE : PCI);
+            init_pci_acpi(dev, uid, pci_bus_is_express(bus) ? PCIE : PCI);
 
             if (numa_node != NUMA_NODE_UNASSIGNED) {
                 aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
hw/pci-bridge/pci_expander_bridge.c +19 -0
index b42592e1ff..5021b60435 100644
@@ -67,6 +67,7 @@ struct PXBDev {
 
     uint8_t bus_nr;
     uint16_t numa_node;
+    int32_t uid;
 };
 
 static PXBDev *convert_to_pxb(PCIDevice *dev)
@@ -98,12 +99,20 @@ static uint16_t pxb_bus_numa_node(PCIBus *bus)
     return pxb->numa_node;
 }
 
+static int32_t pxb_bus_uid(PCIBus *bus)
+{
+    PXBDev *pxb = convert_to_pxb(bus->parent_dev);
+
+    return pxb->uid;
+}
+
 static void pxb_bus_class_init(ObjectClass *class, void *data)
 {
     PCIBusClass *pbc = PCI_BUS_CLASS(class);
 
     pbc->bus_num = pxb_bus_num;
     pbc->numa_node = pxb_bus_numa_node;
+    pbc->uid = pxb_bus_uid;
 }
 
 static const TypeInfo pxb_bus_info = {
@@ -329,6 +338,7 @@ static Property pxb_dev_properties[] = {
     /* Note: 0 is not a legal PXB bus number. */
     DEFINE_PROP_UINT8("bus_nr", PXBDev, bus_nr, 0),
     DEFINE_PROP_UINT16("numa_node", PXBDev, numa_node, NUMA_NODE_UNASSIGNED),
+    DEFINE_PROP_INT32("uid", PXBDev, uid, -1),
     DEFINE_PROP_END_OF_LIST(),
 };
 
@@ -400,12 +410,21 @@ static const TypeInfo pxb_pcie_dev_info = {
 
 static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
 {
+    PXBDev *pxb = convert_to_pxb(dev);
+
     /* A CXL PXB's parent bus is still PCIe */
     if (!pci_bus_is_express(pci_get_bus(dev))) {
         error_setg(errp, "pxb-cxl devices cannot reside on a PCI bus");
         return;
     }
 
+    if (pxb->uid < 0) {
+        error_setg(errp, "pxb-cxl devices must have a valid uid (0-2147483647)");
+        return;
+    }
+
+    /* FIXME: Check that uid doesn't collide with UIDs of other host bridges */
+
     pxb_dev_realize_common(dev, CXL, errp);
 }
 
hw/pci/pci.c +11 -0
index fb454e2317..d2eae06e18 100644
@@ -170,6 +170,11 @@ static uint16_t pcibus_numa_node(PCIBus *bus)
     return NUMA_NODE_UNASSIGNED;
 }
 
+static int32_t pcibus_uid(PCIBus *bus)
+{
+    return -1;
+}
+
 static void pci_bus_class_init(ObjectClass *klass, void *data)
 {
     BusClass *k = BUS_CLASS(klass);
@@ -184,6 +189,7 @@ static void pci_bus_class_init(ObjectClass *klass, void *data)
 
     pbc->bus_num = pcibus_num;
     pbc->numa_node = pcibus_numa_node;
+    pbc->uid = pcibus_uid;
 }
 
 static const TypeInfo pci_bus_info = {
@@ -530,6 +536,11 @@ int pci_bus_numa_node(PCIBus *bus)
     return PCI_BUS_GET_CLASS(bus)->numa_node(bus);
 }
 
+int pci_bus_uid(PCIBus *bus)
+{
+    return PCI_BUS_GET_CLASS(bus)->uid(bus);
+}
+
 static int get_pci_config_device(QEMUFile *f, void *pv, size_t size,
                                  const VMStateField *field)
 {
include/hw/pci/pci.h +1 -0
index c464597e53..d65c2bceae 100644
@@ -462,6 +462,7 @@ static inline int pci_dev_bus_num(const PCIDevice *dev)
 }
 
 int pci_bus_numa_node(PCIBus *bus);
+int pci_bus_uid(PCIBus *bus);
 void pci_for_each_device(PCIBus *bus, int bus_num,
                          void (*fn)(PCIBus *bus, PCIDevice *d, void *opaque),
                          void *opaque);
include/hw/pci/pci_bus.h +1 -0
index eb94e7e85c..3c9fbc55bb 100644
@@ -17,6 +17,7 @@ struct PCIBusClass {
 
     int (*bus_num)(PCIBus *bus);
     uint16_t (*numa_node)(PCIBus *bus);
+    int32_t (*uid)(PCIBus *bus);
 };
 
 enum PCIBusFlags {
Signature · 1 line
2.30.0
#31 lore raw

RFC PATCH v2 20/32 hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)

In reply to Ben Widawsky

CXL host bridges themselves may have MMIO. Since host bridges don't have a BAR they are treated as special for MMIO.

--

It's arbitrarily chosen here to pick 0xD0000000 as the base for the host bridge MMIO. I'm not sure what the right way to find free space for platform hardcoded things like this is.

---
 hw/pci-bridge/pci_expander_bridge.c | 53 ++++++++++++++++++++++++++++-
 include/hw/cxl/cxl.h                |  2 ++
 2 files changed, 54 insertions(+), 1 deletion(-)
hw/pci-bridge/pci_expander_bridge.c +52 -1
index 5021b60435..226a8a5fff 100644
@@ -17,6 +17,7 @@
 #include "hw/pci/pci_host.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci/pci_bridge.h"
+#include "hw/cxl/cxl.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
@@ -70,6 +71,12 @@ struct PXBDev {
     int32_t uid;
 };
 
+typedef struct CXLHost {
+    PCIHostState parent_obj;
+
+    CXLComponentState cxl_cstate;
+} CXLHost;
+
 static PXBDev *convert_to_pxb(PCIDevice *dev)
 {
     /* A CXL PXB's parent bus is PCIe, so the normal check won't work */
@@ -85,6 +92,9 @@ static GList *pxb_dev_list;
 
 #define TYPE_PXB_HOST "pxb-host"
 
+#define TYPE_PXB_CXL_HOST "pxb-cxl-host"
+#define PXB_CXL_HOST(obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST)
+
 static int pxb_bus_num(PCIBus *bus)
 {
     PXBDev *pxb = convert_to_pxb(bus->parent_dev);
@@ -198,6 +208,46 @@ static const TypeInfo pxb_host_info = {
     .class_init    = pxb_host_class_init,
 };
 
+static void pxb_cxl_realize(DeviceState *dev, Error **errp)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(dev);
+    PCIHostState *phb = PCI_HOST_BRIDGE(dev);
+    CXLHost *cxl = PXB_CXL_HOST(dev);
+    CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
+    struct MemoryRegion *mr = &cxl_cstate->crb.component_registers;
+
+    cxl_component_register_block_init(OBJECT(dev), cxl_cstate,
+                                      TYPE_PXB_CXL_HOST);
+    sysbus_init_mmio(sbd, mr);
+
+    /* FIXME: support multiple host bridges. */
+    sysbus_mmio_map(sbd, 0, CXL_HOST_BASE +
+                            memory_region_size(mr) * pci_bus_uid(phb->bus));
+}
+
+static void pxb_cxl_host_class_init(ObjectClass *class, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(class);
+    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(class);
+
+    hc->root_bus_path = pxb_host_root_bus_path;
+    dc->fw_name = "cxl";
+    dc->realize = pxb_cxl_realize;
+    /* Reason: Internal part of the pxb/pxb-pcie device, not usable by itself */
+    dc->user_creatable = false;
+}
+
+/*
+ * This is a device to handle the MMIO for a CXL host bridge. It does nothing
+ * else.
+ */
+static const TypeInfo cxl_host_info = {
+    .name          = TYPE_PXB_CXL_HOST,
+    .parent        = TYPE_PCI_HOST_BRIDGE,
+    .instance_size = sizeof(CXLHost),
+    .class_init    = pxb_cxl_host_class_init,
+};
+
 /*
  * Registers the PXB bus as a child of pci host root bus.
  */
@@ -272,7 +322,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
         dev_name = dev->qdev.id;
     }
 
-    ds = qdev_new(TYPE_PXB_HOST);
+    ds = qdev_new(type == CXL ? TYPE_PXB_CXL_HOST : TYPE_PXB_HOST);
     if (type == PCIE) {
         bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
     } else if (type == CXL) {
@@ -466,6 +516,7 @@ static void pxb_register_types(void)
     type_register_static(&pxb_pcie_bus_info);
     type_register_static(&pxb_cxl_bus_info);
     type_register_static(&pxb_host_info);
+    type_register_static(&cxl_host_info);
     type_register_static(&pxb_dev_info);
     type_register_static(&pxb_pcie_dev_info);
     type_register_static(&pxb_cxl_dev_info);
include/hw/cxl/cxl.h +2 -0
index 362cda40de..6bc344f205 100644
@@ -17,5 +17,7 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define CXL_HOST_BASE 0xD0000000
+
 #endif
 
Signature · 1 line
2.30.0
#32 lore raw

RFC PATCH v2 21/32 acpi/pxb/cxl: Reserve host bridge MMIO

In reply to Ben Widawsky

For all host bridges, reserve MMIO space with _CRS. The MMIO for the host bridge lives in a magically hard coded space in the system's physical address space. The standard mechanism to tell the OS about regions which can't be used for host bridges is _CRS.

---
 hw/i386/acpi-build.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)
hw/i386/acpi-build.c +19 -3
index 26e4ddd025..16cde677a0 100644
@@ -28,6 +28,7 @@
 #include "qemu/bitmap.h"
 #include "qemu/error-report.h"
 #include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
 #include "hw/core/cpu.h"
 #include "target/i386/cpu.h"
 #include "hw/misc/pvpanic.h"
@@ -1194,7 +1195,7 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)
     aml_append(table, scope);
 }
 
-enum { PCI, PCIE };
+enum { PCI, PCIE, CXL };
 static void init_pci_acpi(Aml *dev, int uid, int type)
 {
     if (type == PCI) {
@@ -1344,20 +1345,28 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
             uint8_t bus_num = pci_bus_num(bus);
             uint8_t numa_node = pci_bus_numa_node(bus);
             int32_t uid = pci_bus_uid(bus);
+            int type;
 
             /* look only for expander root buses */
             if (!pci_bus_is_root(bus)) {
                 continue;
             }
 
+            type = pci_bus_is_cxl(bus) ? CXL :
+                                         pci_bus_is_express(bus) ? PCIE : PCI;
+
             if (bus_num < root_bus_limit) {
                 root_bus_limit = bus_num - 1;
             }
 
             scope = aml_scope("\\_SB");
-            dev = aml_device("PC%.02X", bus_num);
+            if (type == CXL) {
+                dev = aml_device("CXL%.01X", pci_bus_uid(bus));
+            } else {
+                dev = aml_device("PC%.02X", bus_num);
+            }
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
-            init_pci_acpi(dev, uid, pci_bus_is_express(bus) ? PCIE : PCI);
+            init_pci_acpi(dev, uid, type);
 
             if (numa_node != NUMA_NODE_UNASSIGNED) {
                 aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
@@ -1368,6 +1377,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
             aml_append(dev, aml_name_decl("_CRS", crs));
             aml_append(scope, dev);
             aml_append(dsdt, scope);
+
+            /* Handle the ranges for the PXB expanders */
+            if (type == CXL) {
+                uint64_t base = CXL_HOST_BASE + uid * 0x10000;
+                crs_range_insert(crs_range_set.mem_ranges, base,
+                                 base + 0x10000 - 1);
+            }
         }
     }
 
Signature · 1 line
2.30.0
#33 lore raw

RFC PATCH v2 22/32 hw/pxb/cxl: Add "windows" for host bridges

In reply to Ben Widawsky

In a bare metal CXL capable system, system firmware will program physical address ranges on the host. This is done by programming internal registers that aren't typically known to OS. These address ranges might be contiguous or interleaved across host bridges.

For a QEMU guest a new construct is introduced allowing passing a memory backend to the host bridge for this same purpose. Each memory backend needs to be passed to the host bridge as well as any device that will be emulating that memory (not implemented here).

I'm hopeful the interleaving work in the link can be re-purposed here (see Link).

An example to create a host bridges with a 512M window at 0x4c0000000
-object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-memory-base=1,memory-base\[0\]=0x4c0000000,memory\[0\]=cxl-mem1

---
 hw/pci-bridge/pci_expander_bridge.c | 65 +++++++++++++++++++++++++++--
 include/hw/cxl/cxl.h                |  1 +
 2 files changed, 62 insertions(+), 4 deletions(-)
hw/pci-bridge/pci_expander_bridge.c +61 -4
index 226a8a5fff..af1450c69d 100644
@@ -69,12 +69,19 @@ struct PXBDev {
     uint8_t bus_nr;
     uint16_t numa_node;
     int32_t uid;
+    struct cxl_dev {
+        HostMemoryBackend *memory_window[CXL_WINDOW_MAX];
+
+        uint32_t num_windows;
+        hwaddr *window_base[CXL_WINDOW_MAX];
+    } cxl;
 };
 
 typedef struct CXLHost {
     PCIHostState parent_obj;
 
     CXLComponentState cxl_cstate;
+    PXBDev *dev;
 } CXLHost;
 
 static PXBDev *convert_to_pxb(PCIDevice *dev)
@@ -213,16 +220,31 @@ static void pxb_cxl_realize(DeviceState *dev, Error **errp)
     SysBusDevice *sbd = SYS_BUS_DEVICE(dev);
     PCIHostState *phb = PCI_HOST_BRIDGE(dev);
     CXLHost *cxl = PXB_CXL_HOST(dev);
+    struct cxl_dev *cxl_dev = &cxl->dev->cxl;
     CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
     struct MemoryRegion *mr = &cxl_cstate->crb.component_registers;
+    int uid = pci_bus_uid(phb->bus);
 
     cxl_component_register_block_init(OBJECT(dev), cxl_cstate,
                                       TYPE_PXB_CXL_HOST);
     sysbus_init_mmio(sbd, mr);
 
-    /* FIXME: support multiple host bridges. */
-    sysbus_mmio_map(sbd, 0, CXL_HOST_BASE +
-                            memory_region_size(mr) * pci_bus_uid(phb->bus));
+    sysbus_mmio_map(sbd, 0, CXL_HOST_BASE + memory_region_size(mr) * uid);
+
+    /*
+     * A CXL host bridge can exist without a fixed memory window, but it would
+     * only operate in legacy PCIe mode.
+     */
+    if (!cxl_dev->memory_window[uid]) {
+        warn_report(
+            "CXL expander bridge created without window. Consider using %s",
+            "memdev[0]=<memory_backend>");
+        return;
+    }
+
+    mr = host_memory_backend_get_memory(cxl_dev->memory_window[uid]);
+    sysbus_init_mmio(sbd, mr);
+    sysbus_mmio_map(sbd, 1 + uid, *cxl_dev->window_base[uid]);
 }
 
 static void pxb_cxl_host_class_init(ObjectClass *class, void *data)
@@ -328,6 +350,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
     } else if (type == CXL) {
         bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_CXL_BUS);
         bus->flags |= PCI_BUS_CXL;
+        PXB_CXL_HOST(ds)->dev = PXB_CXL_DEV(dev);
     } else {
         bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
         bds = qdev_new("pci-bridge");
@@ -389,6 +412,8 @@ static Property pxb_dev_properties[] = {
     DEFINE_PROP_UINT8("bus_nr", PXBDev, bus_nr, 0),
     DEFINE_PROP_UINT16("numa_node", PXBDev, numa_node, NUMA_NODE_UNASSIGNED),
     DEFINE_PROP_INT32("uid", PXBDev, uid, -1),
+    DEFINE_PROP_ARRAY("window-base", PXBDev, cxl.num_windows, cxl.window_base,
+                      qdev_prop_uint64, hwaddr),
     DEFINE_PROP_END_OF_LIST(),
 };
 
@@ -460,7 +485,9 @@ static const TypeInfo pxb_pcie_dev_info = {
 
 static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
 {
-    PXBDev *pxb = convert_to_pxb(dev);
+    PXBDev *pxb = PXB_CXL_DEV(dev);
+    struct cxl_dev *cxl = &pxb->cxl;
+    int count = 0;
 
     /* A CXL PXB's parent bus is still PCIe */
     if (!pci_bus_is_express(pci_get_bus(dev))) {
@@ -476,6 +503,23 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
     /* FIXME: Check that uid doesn't collide with UIDs of other host bridges */
 
     pxb_dev_realize_common(dev, CXL, errp);
+
+    for (unsigned i = 0; i < CXL_WINDOW_MAX; i++) {
+        if (!cxl->memory_window[i]) {
+            continue;
+        }
+
+        count++;
+    }
+
+    if (!count) {
+        warn_report("memory-windows should be set when creating CXL host bridges");
+    }
+
+    if (count != cxl->num_windows) {
+        error_setg(errp, "window bases count (%d) must match window count (%d)",
+                   cxl->num_windows, count);
+    }
 }
 
 static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)
@@ -496,6 +540,19 @@ static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)
 
     /* Host bridges aren't hotpluggable. FIXME: spec reference */
     dc->hotpluggable = false;
+
+    /*
+     * Below is moral equivalent of:
+     *   DEFINE_PROP_ARRAY("memdev", PXBDev, window_count, windows,
+     *                     qdev_prop_memory_device, HostMemoryBackend)
+     */
+    for (unsigned i = 0; i < CXL_WINDOW_MAX; i++) {
+        g_autofree char *name = g_strdup_printf("memdev[%u]", i);
+        object_class_property_add_link(klass, name, TYPE_MEMORY_BACKEND,
+                offsetof(PXBDev, cxl.memory_window[i]),
+                qdev_prop_allow_set_link_before_realize,
+                OBJ_PROP_LINK_STRONG);
+    }
 }
 
 static const TypeInfo pxb_cxl_dev_info = {
include/hw/cxl/cxl.h +1 -0
index 6bc344f205..b1e5f4a8fa 100644
@@ -18,6 +18,7 @@
 #define DEVICE_REG_BAR_IDX 2
 
 #define CXL_HOST_BASE 0xD0000000
+#define CXL_WINDOW_MAX 10
 
 #endif
 
Signature · 1 line
2.30.0
#34 lore raw

RFC PATCH v2 23/32 hw/cxl/rp: Add a root port

In reply to Ben Widawsky

This adds just enough of a root port implementation to be able to enumerate root ports (creating the required DVSEC entries). What's not here yet is the MMIO nor the ability to write some of the DVSEC entries.

This can be added with the qemu commandline by adding a rootport to a specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4

Like the host bridge patch, the ACPI tables aren't generated at this point and so system software cannot use it.

---
 hw/pci-bridge/Kconfig          |   5 +
 hw/pci-bridge/cxl_root_port.c  | 231 +++++++++++++++++++++++++++++++++
 hw/pci-bridge/meson.build      |   1 +
 hw/pci-bridge/pcie_root_port.c |   6 +-
 hw/pci/pci.c                   |   4 +-
 5 files changed, 245 insertions(+), 2 deletions(-)
 create mode 100644 hw/pci-bridge/cxl_root_port.c
hw/pci-bridge/Kconfig +5 -0
index a51ec716f5..a821b531da 100644
@@ -27,3 +27,8 @@ config DEC_PCI
 
 config SIMBA
     bool
+
+config CXL
+    bool
+    default y if PCI_EXPRESS && PXB
+    depends on PCI_EXPRESS && MSI_NONBROKEN && PXB
hw/pci-bridge/cxl_root_port.c new file +231 -0

231 lines not on this page · raw message

hw/pci-bridge/meson.build +1 -0
index daab8acf2a..b6d26a03d5 100644
@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c'))
 pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c'))
 pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c'))
 pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c'))
+pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c'))
 
 # NewWorld PowerMac
 pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))
hw/pci-bridge/pcie_root_port.c +5 -1
index f1cfe9d14a..460e48269d 100644
@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)
     int rc;
 
     pci_config_set_interrupt_pin(d->config, 1);
-    pci_bridge_initfn(d, TYPE_PCIE_BUS);
+    if (d->cap_present & QEMU_PCIE_CAP_CXL) {
+        pci_bridge_initfn(d, TYPE_CXL_BUS);
+    } else {
+        pci_bridge_initfn(d, TYPE_PCIE_BUS);
+    }
     pcie_port_init_reg(d);
 
     rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,
hw/pci/pci.c +3 -1
index d2eae06e18..ecc98938f9 100644
@@ -2692,7 +2692,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
             object_class_dynamic_cast(klass, INTERFACE_CONVENTIONAL_PCI_DEVICE);
         ObjectClass *pcie =
             object_class_dynamic_cast(klass, INTERFACE_PCIE_DEVICE);
-        assert(conventional || pcie);
+        ObjectClass *cxl =
+            object_class_dynamic_cast(klass, INTERFACE_CXL_DEVICE);
+        assert(conventional || pcie || cxl);
     }
 }
 
Signature · 1 line
2.30.0
#35 lore raw

RFC PATCH v2 24/32 hw/cxl/device: Add a memory device (8.2.8.5)

In reply to Ben Widawsky

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"

---
 hw/core/numa.c             |   3 +
 hw/cxl/cxl-mailbox-utils.c |  41 ++++++
 hw/i386/pc.c               |   1 +
 hw/mem/Kconfig             |   5 +
 hw/mem/cxl_type3.c         | 262 +++++++++++++++++++++++++++++++++++++
 hw/mem/meson.build         |   1 +
 hw/pci/pcie.c              |  30 +++++
 include/hw/cxl/cxl.h       |   2 +
 include/hw/cxl/cxl_pci.h   |  22 ++++
 include/hw/pci/pci_ids.h   |   1 +
 monitor/hmp-cmds.c         |  15 +++
 qapi/machine.json          |   1 +
 12 files changed, 384 insertions(+)
 create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
index 68cee65f61..cd7df371e6 100644
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[])
                 node_mem[pcdimm_info->node].node_plugged_mem +=
                     pcdimm_info->size;
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                /* FINISHME */
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 /* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0

65 lines not on this page · raw message

hw/i386/pc.c +1 -0
index 5458f61d10..5d41809b37 100644
@@ -79,6 +79,7 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
index a0ef2cf648..7d9d1ced3e 100644
@@ -10,3 +10,8 @@ config NVDIMM
     default y
     depends on (PC || PSERIES || ARM_VIRT)
     select MEM_DEVICE
+
+config CXL_MEM_DEVICE
+    bool
+    default y if CXL
+    select MEM_DEVICE
hw/mem/cxl_type3.c new file +262 -0

262 lines not on this page · raw message

hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +30 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
+     *  }
+     */
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }
Signature · 1 line
2.30.0
#36 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Review

Quoted 9 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Quoted 235 lines
---
 hw/core/numa.c             |   3 +
 hw/cxl/cxl-mailbox-utils.c |  41 ++++++
 hw/i386/pc.c               |   1 +
 hw/mem/Kconfig             |   5 +
 hw/mem/cxl_type3.c         | 262 +++++++++++++++++++++++++++++++++++++
 hw/mem/meson.build         |   1 +
 hw/pci/pcie.c              |  30 +++++
 include/hw/cxl/cxl.h       |   2 +
 include/hw/cxl/cxl_pci.h   |  22 ++++
 include/hw/pci/pci_ids.h   |   1 +
 monitor/hmp-cmds.c         |  15 +++
 qapi/machine.json          |   1 +
 12 files changed, 384 insertions(+)
 create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
index 68cee65f61..cd7df371e6 100644
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[])
                 node_mem[pcdimm_info->node].node_plugged_mem +=
                     pcdimm_info->size;
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                /* FINISHME */
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 /* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0
index f68ec5b5b9..eeb10b8943 100644
@@ -49,6 +49,8 @@ enum {
     LOGS        = 0x04,
         #define GET_SUPPORTED 0x0
         #define GET_LOG       0x1
+    IDENTIFY    = 0x40,
+        #define MEMORY_DEVICE 0x0
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);
 declare_mailbox_handler(TIMESTAMP_SET);
 declare_mailbox_handler(LOGS_GET_SUPPORTED);
 declare_mailbox_handler(LOGS_GET_LOG);
+declare_mailbox_handler(IDENTIFY_MEMORY_DEVICE);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
 #define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
     CXL_CMD(LOGS, GET_SUPPORTED, 0, 0),
     CXL_CMD(LOGS, GET_LOG, 0x18, 0),
+    CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0),
 };
 
 #undef CXL_CMD
@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)
     return CXL_MBOX_SUCCESS;
 }
 
+/* 8.2.9.5.1.1 */
+define_mailbox_handler(IDENTIFY_MEMORY_DEVICE)
+{
+    struct {
+        char fw_revision[0x10];
+        uint64_t total_capacity;
+        uint64_t volatile_capacity;
+        uint64_t persistent_capacity;
+        uint64_t partition_align;
+        uint16_t info_event_log_size;
+        uint16_t warning_event_log_size;
+        uint16_t failure_event_log_size;
+        uint16_t fatal_event_log_size;
+        uint32_t lsa_size;
+        uint8_t poison_list_max_mer[3];
+        uint16_t inject_poison_limit;
+        uint8_t poison_caps;
+        uint8_t qos_telemetry_caps;
+    } __attribute__((packed)) *id;
+    _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+
+    if (memory_region_size(cxl_dstate->pmem) < (256 << 20)) {
+        return CXL_MBOX_INTERNAL_ERROR;
+    }
+
+    id = (void *)cmd->payload;
+    memset(id, 0, sizeof(*id));
+
+    /* PMEM only */
+    snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
+    id->total_capacity = memory_region_size(cxl_dstate->pmem);
+    id->persistent_capacity = memory_region_size(cxl_dstate->pmem);
+
+    *len = sizeof(*id);
+    return CXL_MBOX_SUCCESS;
+}
+
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
 {
     uint16_t ret = CXL_MBOX_SUCCESS;
hw/i386/pc.c +1 -0
index 5458f61d10..5d41809b37 100644
@@ -79,6 +79,7 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
index a0ef2cf648..7d9d1ced3e 100644
@@ -10,3 +10,8 @@ config NVDIMM
     default y
     depends on (PC || PSERIES || ARM_VIRT)
     select MEM_DEVICE
+
+config CXL_MEM_DEVICE
+    bool
+    default y if CXL
+    select MEM_DEVICE
hw/mem/cxl_type3.c new file +100 -0
new file mode 100644
index 0000000000..3985bb8d0b
@@ -0,0 +1,262 @@
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/error-report.h"
+#include "hw/mem/memory-device.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/pci/pci.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qemu/range.h"
+#include "qemu/rcu.h"
+#include "sysemu/hostmem.h"
+#include "hw/cxl/cxl.h"
+
+typedef struct cxl_type3_dev {
+    /* Private */
+    PCIDevice parent_obj;
+
+    /* Properties */
+    uint64_t size;
+    HostMemoryBackend *hostmem;
+
+    /* State */
+    CXLComponentState cxl_cstate;
+    CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+
+static void build_dvsecs(CXLType3Dev *ct3d)
+{
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    uint8_t *dvsec;
+
+    dvsec = (uint8_t *)&(struct dvsec_device){
+        .cap = 0x1e,
+        .ctrl = 0x6,
+        .status2 = 0x2,
+        .range1_size_hi = 0,
+        .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
+        .range1_base_hi = 0,
+        .range1_base_lo = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
+                               PCIE_CXL_DEVICE_DVSEC,
+                               PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
+
+    dvsec = (uint8_t *)&(struct dvsec_register_locator){
+        .rsvd         = 0,
+        .reg0_base_lo = RBI_COMPONENT_REG | COMPONENT_REG_BAR_IDX,
+        .reg0_base_hi = 0,
+        .reg1_base_lo = RBI_CXL_DEVICE_REG | DEVICE_REG_BAR_IDX,
+        .reg1_base_hi = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+                               REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void ct3_instance_init(Object *obj)
+{
+    /* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
+}
+
+static void ct3_finalize(Object *obj)
+{
+    CXLType3Dev *ct3d = CT3(obj);
+
+    g_free(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;

hw/mem/cxl_type3.c line 110

Quoted 10 lines
hw/mem/cxl_type3.c new file +10 -0
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

hw/mem/cxl_type3.c line 115

Quoted 5 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif

What this hunk is supposed to do, why it's ifdef-ed?

Quoted 193 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +13 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444

Quoted 10 lines
hw/pci/pcie.c +10 -0
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }
#37 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

Quoted 8 lines

Review

Quoted 4 lines

The following example will create a 256M device in a 512M window:

(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.

This isn't implemented in the code yet, but it is planned.

Quoted 578 lines
Quoted 237 lines
---
 hw/core/numa.c             |   3 +
 hw/cxl/cxl-mailbox-utils.c |  41 ++++++
 hw/i386/pc.c               |   1 +
 hw/mem/Kconfig             |   5 +
 hw/mem/cxl_type3.c         | 262 +++++++++++++++++++++++++++++++++++++
 hw/mem/meson.build         |   1 +
 hw/pci/pcie.c              |  30 +++++
 include/hw/cxl/cxl.h       |   2 +
 include/hw/cxl/cxl_pci.h   |  22 ++++
 include/hw/pci/pci_ids.h   |   1 +
 monitor/hmp-cmds.c         |  15 +++
 qapi/machine.json          |   1 +
 12 files changed, 384 insertions(+)
 create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
index 68cee65f61..cd7df371e6 100644
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[])
                 node_mem[pcdimm_info->node].node_plugged_mem +=
                     pcdimm_info->size;
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                /* FINISHME */
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 /* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0
index f68ec5b5b9..eeb10b8943 100644
@@ -49,6 +49,8 @@ enum {
     LOGS        = 0x04,
         #define GET_SUPPORTED 0x0
         #define GET_LOG       0x1
+    IDENTIFY    = 0x40,
+        #define MEMORY_DEVICE 0x0
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);
 declare_mailbox_handler(TIMESTAMP_SET);
 declare_mailbox_handler(LOGS_GET_SUPPORTED);
 declare_mailbox_handler(LOGS_GET_LOG);
+declare_mailbox_handler(IDENTIFY_MEMORY_DEVICE);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
 #define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
     CXL_CMD(LOGS, GET_SUPPORTED, 0, 0),
     CXL_CMD(LOGS, GET_LOG, 0x18, 0),
+    CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0),
 };
 
 #undef CXL_CMD
@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)
     return CXL_MBOX_SUCCESS;
 }
 
+/* 8.2.9.5.1.1 */
+define_mailbox_handler(IDENTIFY_MEMORY_DEVICE)
+{
+    struct {
+        char fw_revision[0x10];
+        uint64_t total_capacity;
+        uint64_t volatile_capacity;
+        uint64_t persistent_capacity;
+        uint64_t partition_align;
+        uint16_t info_event_log_size;
+        uint16_t warning_event_log_size;
+        uint16_t failure_event_log_size;
+        uint16_t fatal_event_log_size;
+        uint32_t lsa_size;
+        uint8_t poison_list_max_mer[3];
+        uint16_t inject_poison_limit;
+        uint8_t poison_caps;
+        uint8_t qos_telemetry_caps;
+    } __attribute__((packed)) *id;
+    _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+
+    if (memory_region_size(cxl_dstate->pmem) < (256 << 20)) {
+        return CXL_MBOX_INTERNAL_ERROR;
+    }
+
+    id = (void *)cmd->payload;
+    memset(id, 0, sizeof(*id));
+
+    /* PMEM only */
+    snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
+    id->total_capacity = memory_region_size(cxl_dstate->pmem);
+    id->persistent_capacity = memory_region_size(cxl_dstate->pmem);
+
+    *len = sizeof(*id);
+    return CXL_MBOX_SUCCESS;
+}
+
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
 {
     uint16_t ret = CXL_MBOX_SUCCESS;
hw/i386/pc.c +1 -0
index 5458f61d10..5d41809b37 100644
@@ -79,6 +79,7 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
index a0ef2cf648..7d9d1ced3e 100644
@@ -10,3 +10,8 @@ config NVDIMM
     default y
     depends on (PC || PSERIES || ARM_VIRT)
     select MEM_DEVICE
+
+config CXL_MEM_DEVICE
+    bool
+    default y if CXL
+    select MEM_DEVICE
hw/mem/cxl_type3.c new file +100 -0
new file mode 100644
index 0000000000..3985bb8d0b
@@ -0,0 +1,262 @@
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/error-report.h"
+#include "hw/mem/memory-device.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/pci/pci.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qemu/range.h"
+#include "qemu/rcu.h"
+#include "sysemu/hostmem.h"
+#include "hw/cxl/cxl.h"
+
+typedef struct cxl_type3_dev {
+    /* Private */
+    PCIDevice parent_obj;
+
+    /* Properties */
+    uint64_t size;
+    HostMemoryBackend *hostmem;
+
+    /* State */
+    CXLComponentState cxl_cstate;
+    CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+
+static void build_dvsecs(CXLType3Dev *ct3d)
+{
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    uint8_t *dvsec;
+
+    dvsec = (uint8_t *)&(struct dvsec_device){
+        .cap = 0x1e,
+        .ctrl = 0x6,
+        .status2 = 0x2,
+        .range1_size_hi = 0,
+        .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
+        .range1_base_hi = 0,
+        .range1_base_lo = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
+                               PCIE_CXL_DEVICE_DVSEC,
+                               PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
+
+    dvsec = (uint8_t *)&(struct dvsec_register_locator){
+        .rsvd         = 0,
+        .reg0_base_lo = RBI_COMPONENT_REG | COMPONENT_REG_BAR_IDX,
+        .reg0_base_hi = 0,
+        .reg1_base_lo = RBI_CXL_DEVICE_REG | DEVICE_REG_BAR_IDX,
+        .reg1_base_hi = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+                               REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void ct3_instance_init(Object *obj)
+{
+    /* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
+}
+
+static void ct3_finalize(Object *obj)
+{
+    CXLType3Dev *ct3d = CT3(obj);
+
+    g_free(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;

hw/mem/cxl_type3.c line 110

Quoted 10 lines
hw/mem/cxl_type3.c new file +10 -0
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

hw/mem/cxl_type3.c line 115

Quoted 5 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif

What this hunk is supposed to do, why it's ifdef-ed?

Quoted 193 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +13 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444

Quoted 10 lines
hw/pci/pcie.c +10 -0
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }
#38 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 18 lines · On Wed, 27 Jan 2021 13:11:16 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

Quoted 8 lines

Review

Quoted 4 lines

The following example will create a 256M device in a 512M window:

(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Quoted 4 lines

A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.

This isn't implemented in the code yet, but it is planned.

could you add here how it supposed to look like CLI interface wise?

also see other questions below.

Quoted 579 lines
Quoted 578 lines
Quoted 237 lines
---
 hw/core/numa.c             |   3 +
 hw/cxl/cxl-mailbox-utils.c |  41 ++++++
 hw/i386/pc.c               |   1 +
 hw/mem/Kconfig             |   5 +
 hw/mem/cxl_type3.c         | 262 +++++++++++++++++++++++++++++++++++++
 hw/mem/meson.build         |   1 +
 hw/pci/pcie.c              |  30 +++++
 include/hw/cxl/cxl.h       |   2 +
 include/hw/cxl/cxl_pci.h   |  22 ++++
 include/hw/pci/pci_ids.h   |   1 +
 monitor/hmp-cmds.c         |  15 +++
 qapi/machine.json          |   1 +
 12 files changed, 384 insertions(+)
 create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
index 68cee65f61..cd7df371e6 100644
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[])
                 node_mem[pcdimm_info->node].node_plugged_mem +=
                     pcdimm_info->size;
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                /* FINISHME */
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 /* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0
index f68ec5b5b9..eeb10b8943 100644
@@ -49,6 +49,8 @@ enum {
     LOGS        = 0x04,
         #define GET_SUPPORTED 0x0
         #define GET_LOG       0x1
+    IDENTIFY    = 0x40,
+        #define MEMORY_DEVICE 0x0
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);
 declare_mailbox_handler(TIMESTAMP_SET);
 declare_mailbox_handler(LOGS_GET_SUPPORTED);
 declare_mailbox_handler(LOGS_GET_LOG);
+declare_mailbox_handler(IDENTIFY_MEMORY_DEVICE);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
 #define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
     CXL_CMD(LOGS, GET_SUPPORTED, 0, 0),
     CXL_CMD(LOGS, GET_LOG, 0x18, 0),
+    CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0),
 };
 
 #undef CXL_CMD
@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)
     return CXL_MBOX_SUCCESS;
 }
 
+/* 8.2.9.5.1.1 */
+define_mailbox_handler(IDENTIFY_MEMORY_DEVICE)
+{
+    struct {
+        char fw_revision[0x10];
+        uint64_t total_capacity;
+        uint64_t volatile_capacity;
+        uint64_t persistent_capacity;
+        uint64_t partition_align;
+        uint16_t info_event_log_size;
+        uint16_t warning_event_log_size;
+        uint16_t failure_event_log_size;
+        uint16_t fatal_event_log_size;
+        uint32_t lsa_size;
+        uint8_t poison_list_max_mer[3];
+        uint16_t inject_poison_limit;
+        uint8_t poison_caps;
+        uint8_t qos_telemetry_caps;
+    } __attribute__((packed)) *id;
+    _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+
+    if (memory_region_size(cxl_dstate->pmem) < (256 << 20)) {
+        return CXL_MBOX_INTERNAL_ERROR;
+    }
+
+    id = (void *)cmd->payload;
+    memset(id, 0, sizeof(*id));
+
+    /* PMEM only */
+    snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
+    id->total_capacity = memory_region_size(cxl_dstate->pmem);
+    id->persistent_capacity = memory_region_size(cxl_dstate->pmem);
+
+    *len = sizeof(*id);
+    return CXL_MBOX_SUCCESS;
+}
+
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
 {
     uint16_t ret = CXL_MBOX_SUCCESS;
hw/i386/pc.c +1 -0
index 5458f61d10..5d41809b37 100644
@@ -79,6 +79,7 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
index a0ef2cf648..7d9d1ced3e 100644
@@ -10,3 +10,8 @@ config NVDIMM
     default y
     depends on (PC || PSERIES || ARM_VIRT)
     select MEM_DEVICE
+
+config CXL_MEM_DEVICE
+    bool
+    default y if CXL
+    select MEM_DEVICE
hw/mem/cxl_type3.c new file +100 -0
new file mode 100644
index 0000000000..3985bb8d0b
@@ -0,0 +1,262 @@
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/error-report.h"
+#include "hw/mem/memory-device.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/pci/pci.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qemu/range.h"
+#include "qemu/rcu.h"
+#include "sysemu/hostmem.h"
+#include "hw/cxl/cxl.h"
+
+typedef struct cxl_type3_dev {
+    /* Private */
+    PCIDevice parent_obj;
+
+    /* Properties */
+    uint64_t size;
+    HostMemoryBackend *hostmem;
+
+    /* State */
+    CXLComponentState cxl_cstate;
+    CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+
+static void build_dvsecs(CXLType3Dev *ct3d)
+{
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    uint8_t *dvsec;
+
+    dvsec = (uint8_t *)&(struct dvsec_device){
+        .cap = 0x1e,
+        .ctrl = 0x6,
+        .status2 = 0x2,
+        .range1_size_hi = 0,
+        .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
+        .range1_base_hi = 0,
+        .range1_base_lo = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
+                               PCIE_CXL_DEVICE_DVSEC,
+                               PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
+
+    dvsec = (uint8_t *)&(struct dvsec_register_locator){
+        .rsvd         = 0,
+        .reg0_base_lo = RBI_COMPONENT_REG | COMPONENT_REG_BAR_IDX,
+        .reg0_base_hi = 0,
+        .reg1_base_lo = RBI_CXL_DEVICE_REG | DEVICE_REG_BAR_IDX,
+        .reg1_base_hi = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+                               REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void ct3_instance_init(Object *obj)
+{
+    /* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
+}
+
+static void ct3_finalize(Object *obj)
+{
+    CXLType3Dev *ct3d = CT3(obj);
+
+    g_free(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;

hw/mem/cxl_type3.c line 110

Quoted 10 lines
hw/mem/cxl_type3.c new file +10 -0
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

hw/mem/cxl_type3.c line 115

Quoted 5 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);  
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif  

What this hunk is supposed to do, why it's ifdef-ed?

Quoted 193 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +13 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 444

Quoted 10 lines
hw/pci/pcie.c +10 -0
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }  

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }  
#39 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 21 lines · Igor Mammedov
Quoted 18 lines · On Wed, 27 Jan 2021 13:11:16 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 16 lines · Igor Mammedov

Review

Quoted 9 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Quoted 9 lines
Quoted 4 lines

A CXL memory device may participate in an interleave set. In such a case, it would be < the total size of the memory window.

This isn't implemented in the code yet, but it is planned.

could you add here how it supposed to look like CLI interface wise?

also see other questions below.

My mistake on the other questions. I forked another thread for those.

Interleave is still being fleshed out. But generally to set up a 512M address range interleaves across 2 devices, each 256M, and each connected to a root port on the host bridge:

# Memory backend
-object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M

# Host Bridge
-device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c0000000 memdev[0]=cxl-mem1

# 2 root ports
-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 -device cxl-rp,id=rp1,bus=cxl.0,addr=0.1,chassis=0,slot=1,memdev=cxl-mem1

# 2 PMEM devices
-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M

#40 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

In reply to Igor Mammedov
Quoted 8 lines · Igor Mammedov
Quoted 5 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

Quoted 8 lines

Review

Quoted 4 lines

The following example will create a 256M device in a 512M window:

(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

Quoted 242 lines
Quoted 242 lines
---
 hw/core/numa.c             |   3 +
 hw/cxl/cxl-mailbox-utils.c |  41 ++++++
 hw/i386/pc.c               |   1 +
 hw/mem/Kconfig             |   5 +
 hw/mem/cxl_type3.c         | 262 +++++++++++++++++++++++++++++++++++++
 hw/mem/meson.build         |   1 +
 hw/pci/pcie.c              |  30 +++++
 include/hw/cxl/cxl.h       |   2 +
 include/hw/cxl/cxl_pci.h   |  22 ++++
 include/hw/pci/pci_ids.h   |   1 +
 monitor/hmp-cmds.c         |  15 +++
 qapi/machine.json          |   1 +
 12 files changed, 384 insertions(+)
 create mode 100644 hw/mem/cxl_type3.c
hw/core/numa.c +3 -0
index 68cee65f61..cd7df371e6 100644
@@ -770,6 +770,9 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[])
                 node_mem[pcdimm_info->node].node_plugged_mem +=
                     pcdimm_info->size;
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                /* FINISHME */
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 /* TODO: once we support numa, assign to right node */
hw/cxl/cxl-mailbox-utils.c +41 -0
index f68ec5b5b9..eeb10b8943 100644
@@ -49,6 +49,8 @@ enum {
     LOGS        = 0x04,
         #define GET_SUPPORTED 0x0
         #define GET_LOG       0x1
+    IDENTIFY    = 0x40,
+        #define MEMORY_DEVICE 0x0
 };
 
 /* 8.2.8.4.5.1 Command Return Codes */
@@ -127,6 +129,7 @@ declare_mailbox_handler(TIMESTAMP_GET);
 declare_mailbox_handler(TIMESTAMP_SET);
 declare_mailbox_handler(LOGS_GET_SUPPORTED);
 declare_mailbox_handler(LOGS_GET_LOG);
+declare_mailbox_handler(IDENTIFY_MEMORY_DEVICE);
 
 #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
 #define IMMEDIATE_POLICY_CHANGE (1 << 3)
@@ -144,6 +147,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
     CXL_CMD(TIMESTAMP, SET, 8, IMMEDIATE_POLICY_CHANGE),
     CXL_CMD(LOGS, GET_SUPPORTED, 0, 0),
     CXL_CMD(LOGS, GET_LOG, 0x18, 0),
+    CXL_CMD(IDENTIFY, MEMORY_DEVICE, 0, 0),
 };
 
 #undef CXL_CMD
@@ -262,6 +266,43 @@ define_mailbox_handler(LOGS_GET_LOG)
     return CXL_MBOX_SUCCESS;
 }
 
+/* 8.2.9.5.1.1 */
+define_mailbox_handler(IDENTIFY_MEMORY_DEVICE)
+{
+    struct {
+        char fw_revision[0x10];
+        uint64_t total_capacity;
+        uint64_t volatile_capacity;
+        uint64_t persistent_capacity;
+        uint64_t partition_align;
+        uint16_t info_event_log_size;
+        uint16_t warning_event_log_size;
+        uint16_t failure_event_log_size;
+        uint16_t fatal_event_log_size;
+        uint32_t lsa_size;
+        uint8_t poison_list_max_mer[3];
+        uint16_t inject_poison_limit;
+        uint8_t poison_caps;
+        uint8_t qos_telemetry_caps;
+    } __attribute__((packed)) *id;
+    _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+
+    if (memory_region_size(cxl_dstate->pmem) < (256 << 20)) {
+        return CXL_MBOX_INTERNAL_ERROR;
+    }
+
+    id = (void *)cmd->payload;
+    memset(id, 0, sizeof(*id));
+
+    /* PMEM only */
+    snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
+    id->total_capacity = memory_region_size(cxl_dstate->pmem);
+    id->persistent_capacity = memory_region_size(cxl_dstate->pmem);
+
+    *len = sizeof(*id);
+    return CXL_MBOX_SUCCESS;
+}
+
 void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
 {
     uint16_t ret = CXL_MBOX_SUCCESS;
hw/i386/pc.c +1 -0
index 5458f61d10..5d41809b37 100644
@@ -79,6 +79,7 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
hw/mem/Kconfig +5 -0
index a0ef2cf648..7d9d1ced3e 100644
@@ -10,3 +10,8 @@ config NVDIMM
     default y
     depends on (PC || PSERIES || ARM_VIRT)
     select MEM_DEVICE
+
+config CXL_MEM_DEVICE
+    bool
+    default y if CXL
+    select MEM_DEVICE
hw/mem/cxl_type3.c new file +105 -0
new file mode 100644
index 0000000000..3985bb8d0b
@@ -0,0 +1,262 @@
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/error-report.h"
+#include "hw/mem/memory-device.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/pci/pci.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qemu/range.h"
+#include "qemu/rcu.h"
+#include "sysemu/hostmem.h"
+#include "hw/cxl/cxl.h"
+
+typedef struct cxl_type3_dev {
+    /* Private */
+    PCIDevice parent_obj;
+
+    /* Properties */
+    uint64_t size;
+    HostMemoryBackend *hostmem;
+
+    /* State */
+    CXLComponentState cxl_cstate;
+    CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+
+static void build_dvsecs(CXLType3Dev *ct3d)
+{
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    uint8_t *dvsec;
+
+    dvsec = (uint8_t *)&(struct dvsec_device){
+        .cap = 0x1e,
+        .ctrl = 0x6,
+        .status2 = 0x2,
+        .range1_size_hi = 0,
+        .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
+        .range1_base_hi = 0,
+        .range1_base_lo = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
+                               PCIE_CXL_DEVICE_DVSEC,
+                               PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
+
+    dvsec = (uint8_t *)&(struct dvsec_register_locator){
+        .rsvd         = 0,
+        .reg0_base_lo = RBI_COMPONENT_REG | COMPONENT_REG_BAR_IDX,
+        .reg0_base_hi = 0,
+        .reg1_base_lo = RBI_CXL_DEVICE_REG | DEVICE_REG_BAR_IDX,
+        .reg1_base_hi = 0,
+    };
+    cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+                               REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void ct3_instance_init(Object *obj)
+{
+    /* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
+}
+
+static void ct3_finalize(Object *obj)
+{
+    CXLType3Dev *ct3d = CT3(obj);
+
+    g_free(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

hw/mem/cxl_type3.c line 110

Quoted 5 lines
hw/mem/cxl_type3.c new file +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Quoted 7 lines

hw/mem/cxl_type3.c line 115

Quoted 6 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif

What this hunk is supposed to do, why it's ifdef-ed?

In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.

Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +17 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
Quoted 9 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445

Quoted 6 lines
hw/pci/pcie.c +6 -0
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.

Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }
#41 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

hw/mem/cxl_type3.c line 110

Quoted 5 lines
hw/mem/cxl_type3.c new file +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Quoted 340 lines
Quoted 9 lines

hw/mem/cxl_type3.c line 115

Quoted 8 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);  
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif  

What this hunk is supposed to do, why it's ifdef-ed?

In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.

Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +17 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
Quoted 9 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445

Quoted 6 lines
hw/pci/pcie.c +6 -0
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }  

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.

Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }  
#42 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

hw/mem/cxl_type3.c line 110

Quoted 5 lines
hw/mem/cxl_type3.c new file +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

Quoted 8 lines
			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.

range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.

I was digging into this yesterday, but opted to start a new thread on the matter.

Quoted 342 lines
Quoted 340 lines
Quoted 9 lines

hw/mem/cxl_type3.c line 115

Quoted 8 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);  
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif  

What this hunk is supposed to do, why it's ifdef-ed?

In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.

Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +17 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
Quoted 9 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445

Quoted 6 lines
hw/pci/pcie.c +6 -0
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }  

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.

Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }  
#43 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 85 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

hw/mem/cxl_type3.c line 110

Quoted 5 lines
hw/mem/cxl_type3.c new file +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

Quoted 8 lines
			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Quoted 8 lines

Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.

range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.

I was digging into this yesterday, but opted to start a new thread on the matter.

Hmm. I think I need to figure out the right add_subregion after this and it might work. I'll keep digging, but if you have ideas, let me know.

Quoted 343 lines
Quoted 342 lines
Quoted 340 lines
Quoted 9 lines

hw/mem/cxl_type3.c line 115

Quoted 8 lines
hw/mem/cxl_type3.c new file +4 -0
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);  
 
+#ifdef SET_PMEM_PADDR
+    memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
+#endif  

What this hunk is supposed to do, why it's ifdef-ed?

In general, the BIOS or OS should program the physical address of the device via a piece of hardware called the HDM (host defined memory) decoder. That is hooked up in the code in a later patch. For debug, I set the address internally in QEMU, mimicking what might be done by BIOS.

Quoted 199 lines
Quoted 199 lines
hw/mem/cxl_type3.c new file +147 -0
+}
+
+static MemoryRegion *cxl_md_get_memory_region(MemoryDeviceState *md,
+                                              Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    return ct3d->cxl_dstate.pmem;
+}
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+    CXLType3Dev *ct3d = CT3(pci_dev);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
+    MemoryRegion *mr = &regs->component_registers;
+    uint8_t *pci_conf = pci_dev->config;
+
+    if (!ct3d->cxl_dstate.pmem) {
+        cxl_setup_memory(ct3d, errp);
+    }
+
+    pci_config_set_prog_interface(pci_conf, 0x10);
+    pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+    pcie_endpoint_cap_init(pci_dev, 0x80);
+    cxl_cstate->dvsec_offset = 0x100;
+
+    ct3d->cxl_cstate.pdev = pci_dev;
+    build_dvsecs(ct3d);
+
+    cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+                                      TYPE_CXL_TYPE3_DEV);
+
+    pci_register_bar(
+        pci_dev, COMPONENT_REG_BAR_IDX,
+        PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+    cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+    pci_register_bar(pci_dev, DEVICE_REG_BAR_IDX,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                         PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &ct3d->cxl_dstate.device_registers);
+}
+
+static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
+{
+    CXLType3Dev *ct3d = CT3(md);
+
+    return memory_region_get_ram_addr(ct3d->cxl_dstate.pmem);
+}
+
+static void cxl_md_set_addr(MemoryDeviceState *md, uint64_t addr, Error **errp)
+{
+    object_property_set_uint(OBJECT(md), "paddr", addr, errp);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CT3(dev);
+    uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+    cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+    cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+    DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+    DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+                     HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pc_dimm_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    PCDIMMDeviceInfo *di = g_new0(PCDIMMDeviceInfo, 1);
+    const DeviceClass *dc = DEVICE_GET_CLASS(md);
+    const DeviceState *dev = DEVICE(md);
+    CXLType3Dev *ct3d = CT3(md);
+
+    if (dev->id) {
+        di->has_id = true;
+        di->id = g_strdup(dev->id);
+    }
+    di->hotplugged = dev->hotplugged;
+    di->hotpluggable = dc->hotpluggable;
+    di->addr = cxl_md_get_addr(md);
+    di->slot = 0;
+    di->node = 0;
+    di->size = memory_device_get_region_size(md, NULL);
+    di->memdev = object_get_canonical_path(OBJECT(ct3d->hostmem));
+
+
+    info->u.cxl.data = di;
+    info->type = MEMORY_DEVICE_INFO_KIND_CXL;
+}
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    pc->realize = ct3_realize;
+    pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd93; /* LVF for now */
+    pc->revision = 1;
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    dc->desc = "CXL PMEM Device (Type 3)";
+    dc->reset = ct3d_reset;
+    device_class_set_props(dc, ct3_props);
+
+    mdc->get_memory_region = cxl_md_get_memory_region;
+    mdc->get_addr = cxl_md_get_addr;
+    mdc->fill_device_info = pc_dimm_md_fill_device_info;
+    mdc->get_plugged_size = memory_device_get_region_size;
+    mdc->set_addr = cxl_md_set_addr;
+}
+
+static const TypeInfo ct3d_info = {
+    .name = TYPE_CXL_TYPE3_DEV,
+    .parent = TYPE_PCI_DEVICE,
+    .class_init = ct3_class_init,
+    .instance_size = sizeof(CXLType3Dev),
+    .instance_init = ct3_instance_init,
+    .instance_finalize = ct3_finalize,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { INTERFACE_CXL_DEVICE },
+        { INTERFACE_PCIE_DEVICE },
+        {}
+    },
+};
+
+static void ct3d_registers(void)
+{
+    type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
hw/mem/meson.build +1 -0
index 0d22f2b572..d13c3ed117 100644
@@ -3,5 +3,6 @@ mem_ss.add(files('memory-device.c'))
 mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
 mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
 mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
 
 softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
hw/pci/pcie.c +17 -0
index d4010cf8f3..1ecf6f6a55 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "hw/mem/memory-device.h"
 #include "hw/pci/pci_bridge.h"
 #include "hw/pci/pcie.h"
 #include "hw/pci/msix.h"
@@ -27,6 +28,8 @@
 #include "hw/pci/pci_bus.h"
 #include "hw/pci/pcie_regs.h"
 #include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl.h"
+#include "hw/boards.h"
 #include "qemu/range.h"
 
 //#define DEBUG_PCIE
@@ -419,6 +422,28 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
+
+#ifdef CXL_MEM_DEVICE
+    /*
+     * FIXME:
+     * if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+     *    HotplugHandler *hotplug_ctrl;
+     *   Error *local_err = NULL;
+     *  hotplug_ctrl = qdev_get_hotplug_handler(dev);
+     *  if (hotplug_ctrl) {
+     *      hotplug_handler_pre_plug(hotplug_ctrl, dev, &local_err);
+     *      if (local_err) {
+     *          error_propagate(errp, local_err);
+     *          return;
+     *      }
Quoted 9 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 445

Quoted 6 lines
hw/pci/pcie.c +6 -0
+     *  }
+     */
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV)) {
+        memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()),
+                               NULL, errp);
+    }  

why use MEMORY_DEVICE interface instead of exposing memory as PCI BAR?

A CXL memory device adds memory to a system that is physically addressable in the same way an NVDIMM device would be.

Quoted 113 lines
Quoted 112 lines
hw/pci/pcie.c +6 -0
+#endif
 }
 
 void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
@@ -455,6 +480,11 @@ void pcie_cap_slot_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         pcie_cap_slot_event(hotplug_pdev,
                             PCI_EXP_HP_EV_PDC | PCI_EXP_HP_EV_ABP);
     }
+
+#ifdef CXL_MEM_DEVICE
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CXL_TYPE3_DEV))
+        memory_device_plug(MEMORY_DEVICE(dev), MACHINE(qdev_get_machine()));
+#endif
 }
 
 void pcie_cap_slot_unplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
include/hw/cxl/cxl.h +2 -0
index b1e5f4a8fa..809ed7de60 100644
@@ -17,6 +17,8 @@
 #define COMPONENT_REG_BAR_IDX 0
 #define DEVICE_REG_BAR_IDX 2
 
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
include/hw/cxl/cxl_pci.h +22 -0
index a53c2e5ae7..9ec28c9feb 100644
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
  * CXL 2.0 Downstream Port: 3, 4, 7, 8
  */
 
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct dvsec_device {
+    struct dvsec_header hdr;
+    uint16_t cap;
+    uint16_t ctrl;
+    uint16_t status;
+    uint16_t ctrl2;
+    uint16_t status2;
+    uint16_t lock;
+    uint16_t cap2;
+    uint32_t range1_size_hi;
+    uint32_t range1_size_lo;
+    uint32_t range1_base_hi;
+    uint32_t range1_base_lo;
+    uint32_t range2_size_hi;
+    uint32_t range2_size_lo;
+    uint32_t range2_base_hi;
+    uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct dvsec_device) == 0x38,
+               "dvsec device size incorrect");
+
 /* CXL 2.0 - 8.1.5 (ID 0003) */
 struct extensions_dvsec_port {
     struct dvsec_header hdr;
include/hw/pci/pci_ids.h +1 -0
index 11f8ab7149..76bf3ed590 100644
@@ -53,6 +53,7 @@
 #define PCI_BASE_CLASS_MEMORY            0x05
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
+#define PCI_CLASS_MEMORY_CXL             0x0502
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06
monitor/hmp-cmds.c +15 -0
index 0dd594f92b..0f67bc61ce 100644
@@ -1887,6 +1887,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                 monitor_printf(mon, "  hotpluggable: %s\n",
                                di->hotpluggable ? "true" : "false");
                 break;
+            case MEMORY_DEVICE_INFO_KIND_CXL:
+                di = value->u.cxl.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               di->id ? di->id : "");
+                monitor_printf(mon, "  addr: 0x%" PRIx64 "\n", di->addr);
+                monitor_printf(mon, "  slot: %" PRId64 "\n", di->slot);
+                monitor_printf(mon, "  node: %" PRId64 "\n", di->node);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", di->size);
+                monitor_printf(mon, "  memdev: %s\n", di->memdev);
+                monitor_printf(mon, "  hotplugged: %s\n",
+                               di->hotplugged ? "true" : "false");
+                monitor_printf(mon, "  hotpluggable: %s\n",
+                               di->hotpluggable ? "true" : "false");
+                break;
             case MEMORY_DEVICE_INFO_KIND_VIRTIO_PMEM:
                 vpi = value->u.virtio_pmem.data;
                 monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
qapi/machine.json +1 -0
index 330189efe3..aa96d662bd 100644
@@ -1394,6 +1394,7 @@
 { 'union': 'MemoryDeviceInfo',
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
+            'cxl': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
             'virtio-mem': 'VirtioMEMDeviceInfo'
           }  
#44 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
hw/mem/cxl_type3.c new file +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

hw/mem/cxl_type3.c line 110

Quoted 5 lines
hw/mem/cxl_type3.c new file +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

Quoted 8 lines
			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.

Quoted 8 lines
Quoted 5 lines

range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.

I was digging into this yesterday, but opted to start a new thread on the matter.

Hmm. I think I need to figure out the right add_subregion after this and it might work. I'll keep digging, but if you have ideas, let me know.

[snip]

I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.

     memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
             "cxl_type3-memory", mr, mr->addr + offset, ct3d->size);
     memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
     memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);

-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments

#45 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 104 lines · Ben Widawsky
Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"  

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
(quoted diff) +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);  

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

Quoted 8 lines
			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.

Quoted 8 lines
Quoted 5 lines

range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.

I was digging into this yesterday, but opted to start a new thread on the matter.

Hmm. I think I need to figure out the right add_subregion after this and it might work. I'll keep digging, but if you have ideas, let me know.

[snip]

I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.

Quoted 7 lines
     memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
             "cxl_type3-memory", mr, mr->addr + offset, ct3d->size);
     memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
     memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);

-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments

Ignore this. It was a problem with my commandline.

I think I have something limping along now.

#46 lore raw

Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5)

Quoted 110 lines · On Thu, 28 Jan 2021 08:58:01 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 109 lines · Ben Widawsky
Quoted 89 lines · Ben Widawsky
Quoted 88 lines · Ben Widawsky
Quoted 75 lines · Jonathan Cameron
Quoted 11 lines · On Wed, 27 Jan 2021 13:26:45 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 5 Jan 2021 08:53:15 -0800 Ben Widawsky <[email protected]> wrote:

A CXL memory device (AKA Type 3) is a CXL component that contains some combination of volatile and persistent memory. It also implements the previously defined mailbox interface as well as the memory device firmware interface.

The following example will create a 256M device in a 512M window:

Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +0 -2
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"    

I'd expect whole backend used by frontend, so one would not need "size" property on frontend (like we do with memory devices).
So question is why it partially uses memdev?

Answered in a separate thread...

One possible suggestion inline.

Quoted 44 lines
Quoted 35 lines
Quoted 35 lines
(quoted diff) +35 -0
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+    MemoryRegionSection mrs;
+    MemoryRegion *mr;
+    uint64_t offset = 0;
+    size_t remaining_size;
+
+    if (!ct3d->hostmem) {
+        error_setg(errp, "memdev property must be set");
+        return;
+    }
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
+
+    /* Create our new subregion */
+    ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+
+    /* Find the first free space in the window */
+    WITH_RCU_READ_LOCK_GUARD()
+    {
+        mrs = memory_region_find(mr, offset, 1);
+        while (mrs.mr && mrs.mr != mr) {
+            offset += memory_region_size(mrs.mr);
+            mrs = memory_region_find(mr, offset, 1);
+        }
+    }
+
+    remaining_size = memory_region_size(mr) - offset;
+    if (remaining_size < ct3d->size) {
+        g_free(ct3d->cxl_dstate.pmem);
+        error_setg(errp,
+                   "Not enough free space (%zd) required for device (%" PRId64  ")",
+                   remaining_size, ct3d->size);
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+    }
+
+    /* Register our subregion as non-volatile */
+    memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
+                           "cxl_type3-memory", ct3d->size, errp);    

this allocates ct3d->size of anon RAM, was this an intention? If yes, can you clarify why extra RAM is used instead of using what backend provides?

Quoted 3 lines

It sounds like I'm doing the wrong thing then. There should be one chunk of memory which is a subset of the full memory backend object. Could you please advise on what I should be doing instead? Is add_subregion() sufficient?

Taking inspiration from nvdimm I'm carrying a patch that uses memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q,

Quoted 8 lines
			 "cxl_type3-memory", mr, offset, ct3d->size);

I 'think' that's doing the right thing, but haven't fully tested it yet so may be completely wrong :)

Then for the pmem addr, call memory_region_set_address() to put it in a particular location.

Yes - this is what I'd like to do and what I initially tried, and I also believe it's right, but it doesn't work.

Quoted 8 lines
Quoted 5 lines

range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed.

I was digging into this yesterday, but opted to start a new thread on the matter.

Hmm. I think I need to figure out the right add_subregion after this and it might work. I'll keep digging, but if you have ideas, let me know.

[snip]

I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later.

     memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
             "cxl_type3-memory", mr, mr->addr + offset, ct3d->size);
     memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
     memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
Quoted 6 lines
Quoted 2 lines

-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments

Ignore this. It was a problem with my commandline.

I think I have something limping along now.

Great. It's going to get more interesting to do this 'right' though.

We'll need to define a memory window similar to device_mem which is used for normal memory hotplug then export that window via the relevant ACPI tables (once defined somewhere public). Finally we'll then want to actually do the memory_region_add_subregion() only on the OS / firmware having configured the type3 device by setting it's PA base.

Going to be fiddly - but should be doable. With it basically working should be a series of sensible (ish) steps to get there.

Jonathan

#47 lore raw

RFC PATCH v2 25/32 hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)

In reply to Ben Widawsky

A device's volatile and persistent memory are known Host Defined Memory (HDM) regions. The mechanism by which the device is programmed to claim the addresses associated with those regions is through dedicated logic known as the HDM decoder. In order to allow the OS to properly program the HDMs, the HDM decoders must be modeled.

There are two ways the HDM decoders can be implemented, the legacy mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8), and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not implemented.

Much of CXL device logic is implemented in cxl-utils. The HDM decoder however is implemented directly by the device implementation. The generic cxl-utils probably should be the correct place to put this since HDM decoders aren't unique to a type3 device. It is however easier at the moment, and requires less design consideration to simply implement it in the device, and figure out how to consolidate it later.

---
 hw/mem/cxl_type3.c | 82 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 77 insertions(+), 5 deletions(-)
hw/mem/cxl_type3.c +77 -5
index 3985bb8d0b..dee5a8884b 100644
@@ -57,6 +57,71 @@ static void build_dvsecs(CXLType3Dev *ct3d)
                                REG_LOC_DVSEC_REVID, dvsec);
 }
 
+static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
+{
+    MemoryRegion *pmem = ct3d->cxl_dstate.pmem;
+    MemoryRegion *mr = host_memory_backend_get_memory(ct3d->hostmem);
+    Range window, device;
+    ComponentRegisters *cregs = &ct3d->cxl_cstate.crb;
+    uint32_t *cache_mem = cregs->cache_mem_registers;
+    uint64_t offset, size;
+    Error *err = NULL;
+
+    assert(which == 0);
+
+    ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMIT, 0);
+    ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, ERROR, 0);
+
+    offset = ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_BASE_HI] << 32) |
+             cache_mem[R_CXL_HDM_DECODER0_BASE_LO];
+    size = ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_SIZE_HI] << 32) |
+           cache_mem[R_CXL_HDM_DECODER0_SIZE_LO];
+
+    range_init_nofail(&window, mr->addr, memory_region_size(mr));
+    range_init_nofail(&device, offset, size);
+
+    if (!range_contains_range(&window, &device)) {
+        ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, ERROR, 1);
+        return;
+    }
+
+    memory_region_ram_resize(pmem, size, &err);
+    if (err) {
+        ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, ERROR, 1);
+        return;
+    }
+
+    offset -= mr->addr;
+    memory_region_add_subregion(mr, offset, pmem);
+
+    ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
+}
+
+static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, unsigned size)
+{
+    CXLComponentState *cxl_cstate = opaque;
+    ComponentRegisters *cregs = &cxl_cstate->crb;
+    CXLType3Dev *ct3d = container_of(cxl_cstate, CXLType3Dev, cxl_cstate);
+    uint32_t *cache_mem = cregs->cache_mem_registers;
+    bool should_commit = false;
+    int which_hdm = -1;
+
+    assert(size == 4);
+
+    switch (offset) {
+    case A_CXL_HDM_DECODER0_CTRL:
+        should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
+        which_hdm = 0;
+        break;
+    default:
+        break;
+    }
+
+    stl_le_p((uint8_t *)cache_mem + offset, value);
+    if (should_commit)
+        hdm_decoder_commit(ct3d, which_hdm);
+}
+
 static void ct3_instance_init(Object *obj)
 {
     /* MemoryDeviceClass *mdc = MEMORY_DEVICE_GET_CLASS(obj); */
@@ -65,7 +130,10 @@ static void ct3_instance_init(Object *obj)
 static void ct3_finalize(Object *obj)
 {
     CXLType3Dev *ct3d = CT3(obj);
+    CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+    ComponentRegisters *regs = &cxl_cstate->crb;
 
+    g_free((void *)regs->special_ops);
     g_free(ct3d->cxl_dstate.pmem);
 }
 
@@ -81,11 +149,12 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
         return;
     }
 
-    /* FIXME: need to check mr is the host bridge's MR */
-    mr = host_memory_backend_get_memory(ct3d->hostmem);
-
     /* Create our new subregion */
     ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1);
+    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
+
+    /* FIXME: need to check mr is the host bridge's MR */
+    mr = host_memory_backend_get_memory(ct3d->hostmem);
 
     /* Find the first free space in the window */
     WITH_RCU_READ_LOCK_GUARD()
@@ -108,8 +177,6 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
     /* Register our subregion as non-volatile */
     memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d),
                            "cxl_type3-memory", ct3d->size, errp);
-    memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true);
-
 #ifdef SET_PMEM_PADDR
     memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem);
 #endif
@@ -148,6 +215,11 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
     ct3d->cxl_cstate.pdev = pci_dev;
     build_dvsecs(ct3d);
 
+#ifndef SET_PMEM_PADDR
+    regs->special_ops = g_new0(MemoryRegionOps, 1);
+    regs->special_ops->write = ct3d_reg_write;
+#endif
+
     cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
                                       TYPE_CXL_TYPE3_DEV);
 
Signature · 1 line
2.30.0
#48 lore raw

RFC PATCH v2 26/32 acpi/cxl: Add _OSC implementation (9.14.2)

In reply to Ben Widawsky

CXL 2.0 specification adds 2 new dwords to the existing _OSC definition from PCIe. The new dwords are accessed with a new uuid. This implementation supports what is in the specification.

We are currently in the process of trying to define a new definition for _OSC. See later work for an explanation.

---
 hw/acpi/Kconfig       |   5 ++
 hw/acpi/cxl.c         | 104 ++++++++++++++++++++++++++++++++++++++++++
 hw/acpi/meson.build   |   1 +
 hw/i386/acpi-build.c  |  12 ++++-
 include/hw/acpi/cxl.h |  23 ++++++++++
 5 files changed, 144 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 include/hw/acpi/cxl.h
hw/acpi/Kconfig +5 -0
index 1932f66af8..b27907953e 100644
@@ -5,6 +5,7 @@ config ACPI_X86
     bool
     select ACPI
     select ACPI_NVDIMM
+    select ACPI_CXL
     select ACPI_CPU_HOTPLUG
     select ACPI_MEMORY_HOTPLUG
     select ACPI_HMAT
@@ -42,3 +43,7 @@ config ACPI_VMGENID
     depends on PC
 
 config ACPI_HW_REDUCED
+
+config ACPI_CXL
+    bool
+    depends on ACPI
hw/acpi/cxl.c new file +104 -0

104 lines not on this page · raw message

hw/acpi/meson.build +1 -0
index dd69577212..9f5c5ced28 100644
@@ -10,6 +10,7 @@ acpi_ss.add(when: 'CONFIG_ACPI_CPU_HOTPLUG', if_true: files('cpu_hotplug.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_MEMORY_HOTPLUG', if_true: files('memory_hotplug.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_true: files('nvdimm.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_PCI', if_true: files('pci.c'))
+acpi_ss.add(when: 'CONFIG_ACPI_CXL', if_true: files('cxl.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_VMGENID', if_true: files('vmgenid.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device.c'))
 acpi_ss.add(when: 'CONFIG_ACPI_HMAT', if_true: files('hmat.c'))
hw/i386/acpi-build.c +11 -1
index 16cde677a0..3eb07b9741 100644
@@ -66,6 +66,7 @@
 #include "hw/acpi/aml-build.h"
 #include "hw/acpi/utils.h"
 #include "hw/acpi/pci.h"
+#include "hw/acpi/cxl.h"
 
 #include "qom/qom-qobject.h"
 #include "hw/i386/amd_iommu.h"
@@ -1201,11 +1202,20 @@ static void init_pci_acpi(Aml *dev, int uid, int type)
     if (type == PCI) {
         aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
         aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
-    } else {
+    } else if (type == PCIE) {
         aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
         aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
         aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
         aml_append(dev, build_q35_osc_method());
+    } else /* CXL */ {
+        struct Aml *pkg = aml_package(2);
+
+        aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0016")));
+        aml_append(pkg, aml_eisaid("PNP0A08"));
+        aml_append(pkg, aml_eisaid("PNP0A03"));
+        aml_append(dev, aml_name_decl("_CID", pkg));
+        aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+        build_cxl_osc_method(dev);
     }
 }
 
include/hw/acpi/cxl.h new file +23 -0
new file mode 100644
index 0000000000..7b8f3b8a2e
@@ -0,0 +1,23 @@
+/*
+ * Copyright (C) 2020 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef HW_ACPI_CXL_H
+#define HW_ACPI_CXL_H
+
+void build_cxl_osc_method(Aml *dev);
+
+#endif
Signature · 1 line
2.30.0
#49 lore raw

RFC PATCH v2 27/32 tests/acpi: allow CEDT table addition

In reply to Ben Widawsky
---
 tests/data/acpi/pc/CEDT                     | 0
 tests/data/acpi/q35/CEDT                    | 0
 tests/qtest/bios-tables-test-allowed-diff.h | 2 ++
 3 files changed, 2 insertions(+)
 create mode 100644 tests/data/acpi/pc/CEDT
 create mode 100644 tests/data/acpi/q35/CEDT
tests/data/acpi/pc/CEDT new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/data/acpi/q35/CEDT new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/qtest/bios-tables-test-allowed-diff.h +2 -0
index dfb8523c8b..9b07f1e1ff 100644
@@ -1 +1,3 @@
 /* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/pc/CEDT",
+"tests/data/acpi/q35/CEDT",
Signature · 1 line
2.30.0
#50 lore raw

RFC PATCH v2 28/32 acpi/cxl: Create the CEDT (9.14.1)

In reply to Ben Widawsky

The CXL Early Discovery Table is defined in the CXL 2.0 specification as a way for the OS to get CXL specific information from the system firmware.

As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge Structure (CHBS) which is primarily useful for telling the OS exactly where the MMIO for the host bridge is.

v2: Update CHBS to spec released definition

---
 hw/acpi/cxl.c                       | 69 +++++++++++++++++++++++++++++
 hw/i386/acpi-build.c                |  6 ++-
 hw/pci-bridge/pci_expander_bridge.c | 21 +--------
 include/hw/acpi/cxl.h               |  4 ++
 include/hw/pci/pci_bridge.h         | 25 +++++++++++
 5 files changed, 104 insertions(+), 21 deletions(-)
hw/acpi/cxl.c +69 -0
index 7124d5a1a3..68db0fe3a8 100644
@@ -18,14 +18,83 @@
  */
 
 #include "qemu/osdep.h"
+#include "hw/sysbus.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_host.h"
 #include "hw/cxl/cxl.h"
+#include "hw/mem/memory-device.h"
 #include "hw/acpi/acpi.h"
 #include "hw/acpi/aml-build.h"
 #include "hw/acpi/bios-linker-loader.h"
 #include "hw/acpi/cxl.h"
+#include "hw/acpi/cxl.h"
 #include "qapi/error.h"
 #include "qemu/uuid.h"
 
+static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(cxl->cxl.cxl_host_bridge);
+    struct MemoryRegion *mr = sbd->mmio[0].memory;
+
+    /* Type */
+    build_append_int_noprefix(table_data, 0, 1);
+
+    /* Reserved */
+    build_append_int_noprefix(table_data, 0, 1);
+
+    /* Record Length */
+    build_append_int_noprefix(table_data, 32, 2);
+
+    /* UID */
+    build_append_int_noprefix(table_data, cxl->uid, 4);
+
+    /* Version */
+    build_append_int_noprefix(table_data, 1, 4);
+
+    /* Reserved */
+    build_append_int_noprefix(table_data, 0, 4);
+
+    /* Base */
+    build_append_int_noprefix(table_data, mr->addr, 8);
+
+    /* Length */
+    build_append_int_noprefix(table_data, memory_region_size(mr), 8);
+}
+
+static int cxl_foreach_pxb_hb(Object *obj, void *opaque)
+{
+    Aml *cedt = opaque;
+
+    if (object_dynamic_cast(obj, TYPE_PXB_CXL_DEVICE)) {
+        PXBDev *pxb = PXB_CXL_DEV(obj);
+
+        cedt_build_chbs(cedt->buf, pxb);
+    }
+
+    return 0;
+}
+
+void cxl_build_cedt(GArray *table_offsets, GArray *table_data,
+                    BIOSLinker *linker)
+{
+    const int cedt_start = table_data->len;
+    Aml *cedt;
+
+    cedt = init_aml_allocator();
+
+    /* reserve space for CEDT header */
+    acpi_add_table(table_offsets, table_data);
+    acpi_data_push(cedt->buf, sizeof(AcpiTableHeader));
+
+    object_child_foreach_recursive(object_get_root(), cxl_foreach_pxb_hb, cedt);
+
+    /* copy AML table into ACPI tables blob and patch header there */
+    g_array_append_vals(table_data, cedt->buf->data, cedt->buf->len);
+    build_header(linker, table_data, (void *)(table_data->data + cedt_start),
+                 "CEDT", table_data->len - cedt_start, 1, NULL, NULL);
+    free_aml_allocator();
+}
+
 static Aml *__build_cxl_osc_method(void)
 {
     Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
hw/i386/acpi-build.c +5 -1
index 3eb07b9741..49242eb8f3 100644
@@ -75,6 +75,8 @@
 #include "hw/acpi/ipmi.h"
 #include "hw/acpi/hmat.h"
 
+#include "hw/acpi/cxl.h"
+
 /* These are used to size the ACPI tables for -M pc-i440fx-1.7 and
  * -M pc-i440fx-2.0.  Even if the actual amount of AML generated grows
  * a little bit, there should be plenty of free space since the DSDT
@@ -1371,7 +1373,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
 
             scope = aml_scope("\\_SB");
             if (type == CXL) {
-                dev = aml_device("CXL%.01X", pci_bus_uid(bus));
+                dev = aml_device("CXL%.01X", uid);
             } else {
                 dev = aml_device("PC%.02X", bus_num);
             }
@@ -2277,6 +2279,8 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)
                           machine->nvdimms_state, machine->ram_slots);
     }
 
+    cxl_build_cedt(table_offsets, tables_blob, tables->linker);
+
     acpi_add_table(table_offsets, tables_blob);
     build_waet(tables_blob, tables->linker);
 
hw/pci-bridge/pci_expander_bridge.c +1 -20
index af1450c69d..6458d5b76e 100644
@@ -57,26 +57,6 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,
 DECLARE_INSTANCE_CHECKER(PXBDev, PXB_PCIE_DEV,
                          TYPE_PXB_PCIE_DEVICE)
 
-#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
-DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
-                         TYPE_PXB_CXL_DEVICE)
-
-struct PXBDev {
-    /*< private >*/
-    PCIDevice parent_obj;
-    /*< public >*/
-
-    uint8_t bus_nr;
-    uint16_t numa_node;
-    int32_t uid;
-    struct cxl_dev {
-        HostMemoryBackend *memory_window[CXL_WINDOW_MAX];
-
-        uint32_t num_windows;
-        hwaddr *window_base[CXL_WINDOW_MAX];
-    } cxl;
-};
-
 typedef struct CXLHost {
     PCIHostState parent_obj;
 
@@ -351,6 +331,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
         bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_CXL_BUS);
         bus->flags |= PCI_BUS_CXL;
         PXB_CXL_HOST(ds)->dev = PXB_CXL_DEV(dev);
+        PXB_CXL_DEV(dev)->cxl.cxl_host_bridge = ds;
     } else {
         bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
         bds = qdev_new("pci-bridge");
include/hw/acpi/cxl.h +4 -0
index 7b8f3b8a2e..db2063f8c9 100644
@@ -18,6 +18,10 @@
 #ifndef HW_ACPI_CXL_H
 #define HW_ACPI_CXL_H
 
+#include "hw/acpi/bios-linker-loader.h"
+
+void cxl_build_cedt(GArray *table_offsets, GArray *table_data,
+                    BIOSLinker *linker);
 void build_cxl_osc_method(Aml *dev);
 
 #endif
include/hw/pci/pci_bridge.h +25 -0
index a94d350034..50dd7fdf33 100644
@@ -28,6 +28,7 @@
 
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bus.h"
+#include "hw/cxl/cxl.h"
 #include "qom/object.h"
 
 typedef struct PCIBridgeWindows PCIBridgeWindows;
@@ -81,6 +82,30 @@ struct PCIBridge {
 #define PCI_BRIDGE_DEV_PROP_MSI        "msi"
 #define PCI_BRIDGE_DEV_PROP_SHPC       "shpc"
 
+struct PXBDev {
+    /*< private >*/
+    PCIDevice parent_obj;
+    /*< public >*/
+
+    uint8_t bus_nr;
+    uint16_t numa_node;
+    int32_t uid;
+
+    struct cxl_dev {
+        HostMemoryBackend *memory_window[CXL_WINDOW_MAX];
+
+        uint32_t num_windows;
+        hwaddr *window_base[CXL_WINDOW_MAX];
+
+        void *cxl_host_bridge; /* Pointer to a CXLHost */
+    } cxl;
+};
+
+typedef struct PXBDev PXBDev;
+#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
+DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
+                         TYPE_PXB_CXL_DEVICE)
+
 int pci_bridge_ssvid_init(PCIDevice *dev, uint8_t offset,
                           uint16_t svid, uint16_t ssid,
                           Error **errp);
Signature · 1 line
2.30.0
#51 lore raw

RFC PATCH v2 29/32 Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)

In reply to Ben Widawsky

This represents Intel's proposal for how the system firmware can notify Linux that the CEDT exists and provides a driver attach point. It is not in the CXL 2.0 specification as of now.

CXL 2.0 specification adds an _HID, ACPI0016, for CXL capable host bridges, with a _CID of PNP0A08 (PCIe host bridge). CXL aware software is able to use this initiate the proper _OSC method, and get the _UID which is referenced by the CEDT. Therefore the existence of an ACPI0016 device allows a CXL aware driver perform the necessary actions. For a CXL capable OS, this works. For a CXL unaware OS, this works.

The motivation for ACPI0017 is to provide the possibility of having a Linux CXL module that can work on a legacy Linux kernel. Linux core PCI/ACPI which won't be built as a module, will see the _CID of PNP0A08 and bind a driver to it. If we later loaded a driver for ACPI0016, Linux won't be able to bind it to the hardware because it has already bound the PNP0A08 driver. The ACPI0017 device is an opportunity to have an object to bind a driver will be used by a Linux driver to walk the CXL topology and do everything that we would have preferred to do with ACPI0016.

There is another motivation for an ACPI0017 device which isn't implemented here. An operating system needs an attach point for a non-volatile region provider that understands cross-hostbridge interleaving. Since QEMU emulation doesn't support interleaving yet, this is more important on the OS side, for now.

---
 hw/i386/acpi-build.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
hw/i386/acpi-build.c +19 -0
index 49242eb8f3..470dd7ddf4 100644
@@ -1221,6 +1221,19 @@ static void init_pci_acpi(Aml *dev, int uid, int type)
     }
 }
 
+static void build_acpi0017(Aml *table)
+{
+    Aml *dev;
+    Aml *scope;
+
+    scope =  aml_scope("_SB");
+    dev = aml_device("CXLM");
+    aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+    aml_append(scope, dev);
+    aml_append(table, scope);
+}
+
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker,
            AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1237,6 +1250,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
     int root_bus_limit = 0xFF;
     PCIBus *bus = NULL;
     TPMIf *tpm = tpm_find();
+    bool cxl_present = false;
     int i;
     VMBusBridge *vmbus_bridge = vmbus_bridge_find();
 
@@ -1392,6 +1406,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
 
             /* Handle the ranges for the PXB expanders */
             if (type == CXL) {
+                cxl_present = true;
                 uint64_t base = CXL_HOST_BASE + uid * 0x10000;
                 crs_range_insert(crs_range_set.mem_ranges, base,
                                  base + 0x10000 - 1);
@@ -1399,6 +1414,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
         }
     }
 
+    if (cxl_present) {
+        build_acpi0017(dsdt);
+    }
+
     /*
      * At this point crs_range_set has all the ranges used by pci
      * busses *other* than PCI0.  These ranges will be excluded from
Signature · 1 line
2.30.0
#52 lore raw

RFC PATCH v2 30/32 tests/acpi: Add new CEDT files

In reply to Ben Widawsky
---
 tests/data/acpi/pc/CEDT                     | Bin 0 -> 36 bytes
 tests/data/acpi/q35/CEDT                    | Bin 0 -> 36 bytes
 tests/qtest/bios-tables-test-allowed-diff.h |   2 --
 3 files changed, 2 deletions(-)
tests/data/acpi/pc/CEDT binary +0 -0
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ebf9b54b0b27d9efca53359c3c2e560511f0e165 100644
GIT binary patch
tests/data/acpi/q35/CEDT binary +0 -0
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..ebf9b54b0b27d9efca53359c3c2e560511f0e165 100644
GIT binary patch
tests/qtest/bios-tables-test-allowed-diff.h +0 -2
index 9b07f1e1ff..dfb8523c8b 100644
@@ -1,3 +1 @@
 /* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/pc/CEDT",
-"tests/data/acpi/q35/CEDT",
Signature · 1 line
2.30.0
#53 lore raw

RFC PATCH v2 31/32 WIP: i386/cxl: Initialize a host bridge

In reply to Ben Widawsky

This patch allows initializing the primary host bridge as a CXL capable hostbridge.

--
This patch is WIP.

---
 hw/arm/virt.c        |  1 +
 hw/core/machine.c    | 26 ++++++++++++++++++++++++++
 hw/i386/acpi-build.c |  8 +++++++-
 hw/i386/microvm.c    |  1 +
 hw/i386/pc.c         |  1 +
 hw/ppc/spapr.c       |  2 ++
 include/hw/boards.h  |  2 ++
 include/hw/cxl/cxl.h |  4 ++++
 8 files changed, 44 insertions(+), 1 deletion(-)
hw/arm/virt.c +1 -0
index 96985917d3..c9d6636efa 100644
@@ -2481,6 +2481,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     hc->unplug_request = virt_machine_device_unplug_request_cb;
     hc->unplug = virt_machine_device_unplug_cb;
     mc->nvdimm_supported = true;
+    mc->cxl_supported = false;
     mc->auto_enable_numa_with_memhp = true;
     mc->auto_enable_numa_with_memdev = true;
     mc->default_ram_id = "mach-virt.ram";
hw/core/machine.c +26 -0
index de3b8f1b31..c739803854 100644
@@ -30,6 +30,7 @@
 #include "sysemu/qtest.h"
 #include "hw/pci/pci.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
 #include "migration/global_state.h"
 #include "migration/vmstate.h"
 
@@ -502,6 +503,20 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value,
     nvdimms_state->persistence_string = g_strdup(value);
 }
 
+static bool machine_get_cxl(Object *obj, Error **errp)
+{
+    MachineState *ms = MACHINE(obj);
+
+    return ms->cxl_devices_state->is_enabled;
+}
+
+static void machine_set_cxl(Object *obj, bool value, Error **errp)
+{
+    MachineState *ms = MACHINE(obj);
+
+    ms->cxl_devices_state->is_enabled = value;
+}
+
 void machine_class_allow_dynamic_sysbus_dev(MachineClass *mc, const char *type)
 {
     QAPI_LIST_PREPEND(mc->allowed_dynamic_sysbus_devices, g_strdup(type));
@@ -903,6 +918,16 @@ static void machine_initfn(Object *obj)
                                         "Valid values are cpu, mem-ctrl");
     }
 
+    if (mc->cxl_supported) {
+        Object *obj = OBJECT(ms);
+
+        ms->cxl_devices_state = g_new0(CXLState, 1);
+        object_property_add_bool(obj, "cxl", machine_get_cxl, machine_set_cxl);
+        object_property_set_description(obj, "cxl",
+                                        "Set on/off to enable/disable "
+                                        "CXL instantiation");
+    }
+
     if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) {
         ms->numa_state = g_new0(NumaState, 1);
         object_property_add_bool(obj, "hmat",
@@ -939,6 +964,7 @@ static void machine_finalize(Object *obj)
     g_free(ms->device_memory);
     g_free(ms->nvdimms_state);
     g_free(ms->numa_state);
+    g_free(ms->cxl_devices_state);
 }
 
 bool machine_usb(MachineState *machine)
hw/i386/acpi-build.c +7 -1
index 470dd7ddf4..90d52e7774 100644
@@ -53,6 +53,7 @@
 #include "sysemu/numa.h"
 #include "sysemu/reset.h"
 #include "hw/hyperv/vmbus-bridge.h"
+#include "hw/cxl/cxl.h"
 
 /* Supported chipsets: */
 #include "hw/southbridge/piix.h"
@@ -1277,8 +1278,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
         build_piix4_pci0_int(dsdt);
     } else {
         sb_scope = aml_scope("_SB");
+        /*
+         * XXX: CXL spec calls this "CXL0", but that would require lots of
+         * changes throughout and so even for CXL enabled, we call it "PCI0"
+         */
         dev = aml_device("PCI0");
-        init_pci_acpi(dev, 0, PCIE);
+        init_pci_acpi(dev, 0,
+                machine->cxl_devices_state->is_enabled ? CXL : PCIE);
         aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
         aml_append(sb_scope, dev);
 
hw/i386/microvm.c +1 -0
index edf2b0f061..970b299a69 100644
@@ -688,6 +688,7 @@ static void microvm_class_init(ObjectClass *oc, void *data)
     mc->auto_enable_numa_with_memdev = false;
     mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE;
     mc->nvdimm_supported = false;
+    mc->cxl_supported = false;
     mc->default_ram_id = "microvm.ram";
 
     /* Avoid relying too much on kernel components */
hw/i386/pc.c +1 -0
index 5d41809b37..7350eeea9c 100644
@@ -1725,6 +1725,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data)
     hc->unplug = pc_machine_device_unplug_cb;
     mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE;
     mc->nvdimm_supported = true;
+    mc->cxl_supported = true;
     mc->default_ram_id = "pc.ram";
 
     object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
hw/ppc/spapr.c +2 -0
index 489cefcb81..b5ea497403 100644
@@ -4442,6 +4442,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     mc->default_cpu_type = POWERPC_CPU_TYPE_NAME("power9_v2.0");
     mc->has_hotpluggable_cpus = true;
     mc->nvdimm_supported = true;
+    mc->cxl_supported = false;
     smc->resize_hpt_default = SPAPR_RESIZE_HPT_ENABLED;
     fwc->get_dev_path = spapr_get_fw_dev_path;
     nc->nmi_monitor_handler = spapr_nmi;
@@ -4602,6 +4603,7 @@ static void spapr_machine_4_2_class_options(MachineClass *mc)
     smc->default_caps.caps[SPAPR_CAP_FWNMI] = SPAPR_CAP_OFF;
     smc->rma_limit = 16 * GiB;
     mc->nvdimm_supported = false;
+    mc->cxl_supported = false;
 }
 
 DEFINE_SPAPR_MACHINE(4_2, "4.2", false);
include/hw/boards.h +2 -0
index 17b1f3f0b9..808f73e134 100644
@@ -206,6 +206,7 @@ struct MachineClass {
     bool ignore_boot_device_suffixes;
     bool smbus_no_migration_support;
     bool nvdimm_supported;
+    bool cxl_supported;
     bool numa_mem_supported;
     bool auto_enable_numa;
     const char *default_ram_id;
@@ -292,6 +293,7 @@ struct MachineState {
     CPUArchIdList *possible_cpus;
     CpuTopology smp;
     struct NVDIMMState *nvdimms_state;
+    struct CXLState *cxl_devices_state;
     struct NumaState *numa_state;
 };
 
include/hw/cxl/cxl.h +4 -0
index 809ed7de60..6961e47076 100644
@@ -22,5 +22,9 @@
 #define CXL_HOST_BASE 0xD0000000
 #define CXL_WINDOW_MAX 10
 
+typedef struct CXLState {
+    bool is_enabled;
+} CXLState;
+
 #endif
 
Signature · 1 line
2.30.0
#54 lore raw

RFC PATCH v2 32/32 qtest/cxl: Add very basic sanity tests

In reply to Ben Widawsky
---
 tests/qtest/cxl-test.c  | 93 +++++++++++++++++++++++++++++++++++++++++
 tests/qtest/meson.build |  4 ++
 2 files changed, 97 insertions(+)
 create mode 100644 tests/qtest/cxl-test.c
tests/qtest/cxl-test.c new file +93 -0
new file mode 100644
index 0000000000..00eca14faa
@@ -0,0 +1,93 @@
+/*
+ * QTest testcase for CXL
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest-single.h"
+
+#define QEMU_PXB_CMD "-machine q35 -object memory-backend-file,id=cxl-mem1," \
+                     "share,mem-path=%s,size=512M "                          \
+                     "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,"  \
+                     "len-window-base=1,window-base[0]=0x4c0000000,memdev[0]=cxl-mem1"
+#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0"
+
+#define QEMU_T3D "-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M"
+
+static void cxl_basic_hb(void)
+{
+    qtest_start("-machine q35,cxl");
+    qtest_end();
+}
+
+static void cxl_basic_pxb(void)
+{
+    qtest_start("-machine q35 -device pxb-cxl,bus=pcie.0,uid=0");
+    qtest_end();
+}
+
+static void cxl_pxb_with_window(void)
+{
+    GString *cmdline;
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    cmdline = g_string_new(NULL);
+    g_string_printf(cmdline, QEMU_PXB_CMD, tmpfs);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+
+    g_string_free(cmdline, TRUE);
+}
+
+static void cxl_root_port(void)
+{
+    GString *cmdline;
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    cmdline = g_string_new(NULL);
+    g_string_printf(cmdline, QEMU_PXB_CMD " %s", tmpfs, QEMU_RP);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+
+    g_string_free(cmdline, TRUE);
+}
+
+static void cxl_t3d(void)
+{
+    GString *cmdline;
+    char template[] = "/tmp/cxl-test-XXXXXX";
+    const char *tmpfs;
+
+    tmpfs = mkdtemp(template);
+
+    cmdline = g_string_new(NULL);
+    g_string_printf(cmdline, QEMU_PXB_CMD " %s %s", tmpfs, QEMU_RP, QEMU_T3D);
+
+    qtest_start(cmdline->str);
+    qtest_end();
+
+    g_string_free(cmdline, TRUE);
+}
+
+int main(int argc, char **argv)
+{
+    g_test_init(&argc, &argv, NULL);
+
+    qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+    qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+    qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+    qtest_add_func("/pci/cxl/root_port", cxl_root_port);
+    qtest_add_func("/pci/cxl/type3_device", cxl_t3d);
+
+    return g_test_run();
+}
tests/qtest/meson.build +4 -0
index 6a67c538be..6ab14fc79a 100644
@@ -22,6 +22,9 @@ qtests_pci = \
   (config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : []) +                  \
   (config_all_devices.has_key('CONFIG_IVSHMEM_DEVICE') ? ['ivshmem-test'] : [])
 
+qtests_cxl = \
+  (config_all_devices.has_key('CONFIG_CXL') ? ['cxl-test'] : [])
+
 qtests_i386 = \
   (slirp.found() ? ['pxe-test', 'test-netfilter'] : []) +             \
   (config_host.has_key('CONFIG_POSIX') ? ['test-filter-mirror'] : []) +                     \
@@ -47,6 +50,7 @@ qtests_i386 = \
   (config_all_devices.has_key('CONFIG_TPM_TIS_ISA') ? ['tpm-tis-swtpm-test'] : []) +        \
   (config_all_devices.has_key('CONFIG_RTL8139_PCI') ? ['rtl8139-test'] : []) +              \
   qtests_pci +                                                                              \
+  qtests_cxl +                                                                              \
   ['fdc-test',
    'ide-test',
    'hd-geo-test',
Signature · 1 line
2.30.0
#55 lore raw
In reply to Ben Widawsky
Quoted 20 lines · On Tue, 5 Jan 2021 08:52:51 -0800 Ben Widawsky <[email protected]> wrote:

Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbed device_reg_init_common() (Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)

New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support

Quoted 4 lines

(There are some new patches that I just named 'v2' for simplicity's sake)

Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).

Hi Ben,

In interests of avoid duplication, I thought I'd give a quick update of where I am. I have what is here working on arm64 (enabling is fairly simple). I'll have a bit more feedback next week and hopefully have time for an indepth review.

I did run into some issues around alignment for the persistent memory that (I think) are down to the fact the memory-backend isn't quite tied up to the device.

Flows wise, I initially started hacking in NFIT building changes in qemu but I'm not yet sure how the hotplug flows are supposed to work and who is responsible for setting up a hotplugged device (OS or firmware).

Plans wise, I was thinking I could look at DOE emulation as seems that we'll be needing that fairly soon to get any useful memory usecases up and running. Let me know if you already have that underway.

Jonathan

Quoted 193 lines

The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.

Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).

CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).

We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs

As mentioned above there are three benefits to carrying this enabling in upstream QEMU:

1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.

2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (nfit_test). In retrospect a QEMU emulation of nfit_test capabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.

3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?

There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity

The flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.

The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case

[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/

Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb _UID through host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests

Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.

 MAINTAINERS                               |   6 +
 hw/Kconfig                                |   1 +
 hw/acpi/Kconfig                           |   5 +
 hw/acpi/cxl.c                             | 173 ++++++++++
 hw/acpi/meson.build                       |   1 +
 hw/arm/virt.c                             |   1 +
 hw/core/machine.c                         |  26 ++
 hw/core/numa.c                            |   3 +
 hw/cxl/Kconfig                            |   3 +
 hw/cxl/cxl-component-utils.c              | 193 +++++++++++
 hw/cxl/cxl-device-utils.c                 | 278 ++++++++++++++++
 hw/cxl/cxl-mailbox-utils.c                | 386 ++++++++++++++++++++++
 hw/cxl/meson.build                        |   5 +
 hw/i386/acpi-build.c                      |  87 ++++-
 hw/i386/microvm.c                         |   1 +
 hw/i386/pc.c                              |   2 +
 hw/mem/Kconfig                            |   5 +
 hw/mem/cxl_type3.c                        | 334 +++++++++++++++++++
 hw/mem/meson.build                        |   1 +
 hw/meson.build                            |   1 +
 hw/pci-bridge/Kconfig                     |   5 +
 hw/pci-bridge/cxl_root_port.c             | 231 +++++++++++++
 hw/pci-bridge/meson.build                 |   1 +
 hw/pci-bridge/pci_expander_bridge.c       | 209 +++++++++++-
 hw/pci-bridge/pcie_root_port.c            |   6 +-
 hw/pci/pci.c                              |  32 +-
 hw/pci/pcie.c                             |  30 ++
 hw/ppc/spapr.c                            |   2 +
 include/hw/acpi/cxl.h                     |  27 ++
 include/hw/boards.h                       |   2 +
 include/hw/cxl/cxl.h                      |  30 ++
 include/hw/cxl/cxl_component.h            | 187 +++++++++++
 include/hw/cxl/cxl_device.h               | 210 ++++++++++++
 include/hw/cxl/cxl_pci.h                  | 160 +++++++++
 include/hw/pci/pci.h                      |  15 +
 include/hw/pci/pci_bridge.h               |  25 ++
 include/hw/pci/pci_bus.h                  |   8 +
 include/hw/pci/pci_ids.h                  |   1 +
 include/standard-headers/linux/pci_regs.h |   1 +
 monitor/hmp-cmds.c                        |  15 +
 qapi/machine.json                         |   1 +
 tests/data/acpi/pc/CEDT                   | Bin 0 -> 36 bytes
 tests/data/acpi/pc/DSDT                   | Bin 5065 -> 5065 bytes
 tests/data/acpi/pc/DSDT.acpihmat          | Bin 6390 -> 6390 bytes
 tests/data/acpi/pc/DSDT.bridge            | Bin 6924 -> 6924 bytes
 tests/data/acpi/pc/DSDT.cphp              | Bin 5529 -> 5529 bytes
 tests/data/acpi/pc/DSDT.dimmpxm           | Bin 6719 -> 6719 bytes
 tests/data/acpi/pc/DSDT.hpbridge          | Bin 5026 -> 5026 bytes
 tests/data/acpi/pc/DSDT.hpbrroot          | Bin 3084 -> 3084 bytes
 tests/data/acpi/pc/DSDT.ipmikcs           | Bin 5137 -> 5137 bytes
 tests/data/acpi/pc/DSDT.memhp             | Bin 6424 -> 6424 bytes
 tests/data/acpi/pc/DSDT.numamem           | Bin 5071 -> 5071 bytes
 tests/data/acpi/pc/DSDT.roothp            | Bin 5261 -> 5261 bytes
 tests/data/acpi/q35/CEDT                  | Bin 0 -> 36 bytes
 tests/data/acpi/q35/DSDT                  | Bin 7801 -> 7801 bytes
 tests/data/acpi/q35/DSDT.acpihmat         | Bin 9126 -> 9126 bytes
 tests/data/acpi/q35/DSDT.bridge           | Bin 7819 -> 7819 bytes
 tests/data/acpi/q35/DSDT.cphp             | Bin 8265 -> 8265 bytes
 tests/data/acpi/q35/DSDT.dimmpxm          | Bin 9455 -> 9455 bytes
 tests/data/acpi/q35/DSDT.ipmibt           | Bin 7876 -> 7876 bytes
 tests/data/acpi/q35/DSDT.memhp            | Bin 9160 -> 9160 bytes
 tests/data/acpi/q35/DSDT.mmio64           | Bin 8932 -> 8932 bytes
 tests/data/acpi/q35/DSDT.numamem          | Bin 7807 -> 7807 bytes
 tests/qtest/cxl-test.c                    |  93 ++++++
 tests/qtest/meson.build                   |   4 +
 65 files changed, 2777 insertions(+), 30 deletions(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/cxl-device-utils.c
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 hw/mem/cxl_type3.c
 create mode 100644 hw/pci-bridge/cxl_root_port.c
 create mode 100644 include/hw/acpi/cxl.h
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_device.h
 create mode 100644 include/hw/cxl/cxl_pci.h
 create mode 100644 tests/data/acpi/pc/CEDT
 create mode 100644 tests/data/acpi/q35/CEDT
 create mode 100644 tests/qtest/cxl-test.c
#56 lore raw
Quoted 25 lines · Jonathan Cameron
Quoted 22 lines · On Tue, 5 Jan 2021 08:52:51 -0800 Ben Widawsky <[email protected]> wrote:

Fixes since v1 [1]:
* Defer introducing some commands/registers not yet used (Ben)
* Add stubbed device_reg_init_common() (Ben)
* Improve assertions in DVSEC creation (Jonathan)
* Use 'n' for HDM register offsets (Jonathan)
* Correct revision ID for extensions (Jonathan)
* Minor cleanups and clarifications (Jonathan)
* Remove error codes not yet used (Jonathan)
* Fix interrupt enable bit width (Jonathan)
* Add comment for weird register size (Jonathan)
* Break out register alignment checks (Jonathan)
* Use the reg alignment helper (Jonathan)
* Rename error codes to match spec
* Fix cap count mid series (Jonathan)

New since v1 [1]:
* Entirely reworked framework for firmware handling
* Implemented more device commands
* CEL support

(There are some new patches that I just named 'v2' for simplicity's sake)

Quoted 8 lines
Quoted 2 lines

Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/).

Hi Ben,

In interests of avoid duplication, I thought I'd give a quick update of where I am. I have what is here working on arm64 (enabling is fairly simple). I'll have a bit more feedback next week and hopefully have time for an indepth review.

Thanks for this. Dan recommended I send out an email with plans for what's needed, what we intend to work on and so on. I'll try to get that out next week also.

Quoted 4 lines

I did run into some issues around alignment for the persistent memory that (I think) are down to the fact the memory-backend isn't quite tied up to the device.

I did have plans to change this to support interleaving.

Quoted 4 lines

Flows wise, I initially started hacking in NFIT building changes in qemu but I'm not yet sure how the hotplug flows are supposed to work and who is responsible for setting up a hotplugged device (OS or firmware).

It won't be NFIT :-). I'd redirect this question to the linux-cxl mailing list, as it's a good one. The spec is somewhat handwavy around this.

Quoted 7 lines

Plans wise, I was thinking I could look at DOE emulation as seems that we'll be needing that fairly soon to get any useful memory usecases up and running. Let me know if you already have that underway.

Jonathan

Chris in the Cc had some plans to enable DOE for their purposes. It'd be good to sync with him. AFAIK though, nobody has plans to enable DOE in Linux, which might be a good target next.

Quoted 194 lines
Quoted 193 lines

The emulation has been critical to get the Linux enabling started [2], it would be an ideal place to land regression tests for different topology handling, and there may be applications for this emulation as a way for a guest to manipulate its address space relative to different performance memories.

Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports aren't implemented (the two components needed to make up a switch).

CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the implementation utilizes existing PCI paradigms. To implement the host bridge, I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural fit even though it doesn't directly map to how hardware will work. For persistent capacity of the memory device, I utilized the memory subsystem (hw/mem).

We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs

As mentioned above there are three benefits to carrying this enabling in upstream QEMU:

1. Linux driver feature development benefits from emulation both due to a lack of initial hardware availability, but also, as is seen with NVDIMM/PMEM emulation, there is value in being able to share topologies with system-software developers even after hardware is available.

2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake resources via custom modules (nfit_test). In retrospect a QEMU emulation of nfit_test capabilities would have made the test environment more portable, and allowed for easier community contributions of example configurations.

3. This is still being fleshed out, but in short it provides a standardized mechanism for the guest to provide feedback to the host about size and placement needs of the memory. After the host gives the guest a physical window mapping to the CXL device, the emulated HDM decoders allow the guest a way to tell the host how much it wants and where. There are likely simpler ways to do this, but they'd require inventing a new interface and you'd need to have diverging driver code in the guest programming of the HDM decoder vs. the host. Since we've already done this work, why not use it?

There is quite a long list of work to do for full spec compliance, but I don't believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity

The flow of the patches in general is to define all the data structures and registers associated with the various components in a top down manner. Host bridge, component, ports, devices. Then, the actual implementation is done in the same order.

The summary is:
1-8: Put infrastructure in place for emulation of the components. 9-12: Implement device mailboxes
13-15: Create the concept of a CXL bus and plumb into PXB 16-22: Implement host bridges
23: Implement a root port
24: Implement a memory device
25: Implement HDM decoders
26-30: ACPI bits
31: Start working on enabling the main host bridge
32: Basic test case

[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/

Ben Widawsky (31):
hw/pci/cxl: Add a CXL component type (interface) hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) hw/cxl/device: Introduce a CXL device (8.2.8) hw/cxl/device: Implement the CAP array (8.2.8.1-2) hw/cxl/device: Add device status (8.2.8.3) hw/cxl/device: Implement basic mailbox (8.2.8.4) hw/cxl/device: Add memory devices (8.2.8.5) hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) hw/cxl/device: Placeholder for firmware commands (8.2.9.2) hw/cxl/device: Timestamp implementation (8.2.9.3) hw/cxl/device: Add log commands (8.2.9.4) + CEL hw/pxb: Use a type for realizing expanders hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
qtest: allow DSDT acpi table changes
acpi/pci: Consolidate host bridge setup
tests/acpi: remove stale allowed tables
hw/pci: Plumb _UID through host bridges hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5) hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) acpi/cxl: Add _OSC implementation (9.14.2) tests/acpi: allow CEDT table addition
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) tests/acpi: Add new CEDT files
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests

Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.

 MAINTAINERS                               |   6 +
 hw/Kconfig                                |   1 +
 hw/acpi/Kconfig                           |   5 +
 hw/acpi/cxl.c                             | 173 ++++++++++
 hw/acpi/meson.build                       |   1 +
 hw/arm/virt.c                             |   1 +
 hw/core/machine.c                         |  26 ++
 hw/core/numa.c                            |   3 +
 hw/cxl/Kconfig                            |   3 +
 hw/cxl/cxl-component-utils.c              | 193 +++++++++++
 hw/cxl/cxl-device-utils.c                 | 278 ++++++++++++++++
 hw/cxl/cxl-mailbox-utils.c                | 386 ++++++++++++++++++++++
 hw/cxl/meson.build                        |   5 +
 hw/i386/acpi-build.c                      |  87 ++++-
 hw/i386/microvm.c                         |   1 +
 hw/i386/pc.c                              |   2 +
 hw/mem/Kconfig                            |   5 +
 hw/mem/cxl_type3.c                        | 334 +++++++++++++++++++
 hw/mem/meson.build                        |   1 +
 hw/meson.build                            |   1 +
 hw/pci-bridge/Kconfig                     |   5 +
 hw/pci-bridge/cxl_root_port.c             | 231 +++++++++++++
 hw/pci-bridge/meson.build                 |   1 +
 hw/pci-bridge/pci_expander_bridge.c       | 209 +++++++++++-
 hw/pci-bridge/pcie_root_port.c            |   6 +-
 hw/pci/pci.c                              |  32 +-
 hw/pci/pcie.c                             |  30 ++
 hw/ppc/spapr.c                            |   2 +
 include/hw/acpi/cxl.h                     |  27 ++
 include/hw/boards.h                       |   2 +
 include/hw/cxl/cxl.h                      |  30 ++
 include/hw/cxl/cxl_component.h            | 187 +++++++++++
 include/hw/cxl/cxl_device.h               | 210 ++++++++++++
 include/hw/cxl/cxl_pci.h                  | 160 +++++++++
 include/hw/pci/pci.h                      |  15 +
 include/hw/pci/pci_bridge.h               |  25 ++
 include/hw/pci/pci_bus.h                  |   8 +
 include/hw/pci/pci_ids.h                  |   1 +
 include/standard-headers/linux/pci_regs.h |   1 +
 monitor/hmp-cmds.c                        |  15 +
 qapi/machine.json                         |   1 +
 tests/data/acpi/pc/CEDT                   | Bin 0 -> 36 bytes
 tests/data/acpi/pc/DSDT                   | Bin 5065 -> 5065 bytes
 tests/data/acpi/pc/DSDT.acpihmat          | Bin 6390 -> 6390 bytes
 tests/data/acpi/pc/DSDT.bridge            | Bin 6924 -> 6924 bytes
 tests/data/acpi/pc/DSDT.cphp              | Bin 5529 -> 5529 bytes
 tests/data/acpi/pc/DSDT.dimmpxm           | Bin 6719 -> 6719 bytes
 tests/data/acpi/pc/DSDT.hpbridge          | Bin 5026 -> 5026 bytes
 tests/data/acpi/pc/DSDT.hpbrroot          | Bin 3084 -> 3084 bytes
 tests/data/acpi/pc/DSDT.ipmikcs           | Bin 5137 -> 5137 bytes
 tests/data/acpi/pc/DSDT.memhp             | Bin 6424 -> 6424 bytes
 tests/data/acpi/pc/DSDT.numamem           | Bin 5071 -> 5071 bytes
 tests/data/acpi/pc/DSDT.roothp            | Bin 5261 -> 5261 bytes
 tests/data/acpi/q35/CEDT                  | Bin 0 -> 36 bytes
 tests/data/acpi/q35/DSDT                  | Bin 7801 -> 7801 bytes
 tests/data/acpi/q35/DSDT.acpihmat         | Bin 9126 -> 9126 bytes
 tests/data/acpi/q35/DSDT.bridge           | Bin 7819 -> 7819 bytes
 tests/data/acpi/q35/DSDT.cphp             | Bin 8265 -> 8265 bytes
 tests/data/acpi/q35/DSDT.dimmpxm          | Bin 9455 -> 9455 bytes
 tests/data/acpi/q35/DSDT.ipmibt           | Bin 7876 -> 7876 bytes
 tests/data/acpi/q35/DSDT.memhp            | Bin 9160 -> 9160 bytes
 tests/data/acpi/q35/DSDT.mmio64           | Bin 8932 -> 8932 bytes
 tests/data/acpi/q35/DSDT.numamem          | Bin 7807 -> 7807 bytes
 tests/qtest/cxl-test.c                    |  93 ++++++
 tests/qtest/meson.build                   |   4 +
 65 files changed, 2777 insertions(+), 30 deletions(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/cxl-device-utils.c
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 hw/mem/cxl_type3.c
 create mode 100644 hw/pci-bridge/cxl_root_port.c
 create mode 100644 include/hw/acpi/cxl.h
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_device.h
 create mode 100644 include/hw/cxl/cxl_pci.h
 create mode 100644 tests/data/acpi/pc/CEDT
 create mode 100644 tests/data/acpi/q35/CEDT
 create mode 100644 tests/qtest/cxl-test.c