Skip to contentSkip to conversation

hw/cxl/mctp/i2c/usb: MCTP for OoB control of CXL devices.

RFC PATCH In review Jonathan Cameron 2 participants 8 messages Jun 9, 2025 – Jun 24, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover hw/cxl/mctp/i2c/usb: MCTP for OoB control of CXL devices.

This posting is primarily about sharing the USB device emulation to get some early feedback.

RFC reasons:
- Known 'inaccuracies' in emulation (not obeying MTU in the to host direction for example)./
- Not sure what to do wrt to Klaus' I2C MCTP support given that has been stalled for some time. For now only the headers are really shared between the two implementations.
- This is more of an FYI / request for testing than a formal suggestion that this might be ready for upstream.

Why add a CXL FM-API over MCTP over USB device?
- Can be emulated on pretty much any host system as USB is discoverable and expandable. If you want a giggle, see the hacks on i386/pc and arm/virt on we've been using until now given only I2C controller that works is the aspeed one. e.g. https://gitlab.com/jic23/qemu/-/commit/134c2e3952b
- Being able to talk to both the fabric management out of band interfaces and the in band devices on the same host makes testing much simpler.

Background:

Back in 2022 I posted some support for controlling the CXL fabric via the spec defined out of band interfaces (CXL Fabric Management API - FM-API) over MCTP on I2C

https://lore.kernel.org/qemu-devel/[email protected]/
I reworked that on top of the NVME-MI work from Klauss.

To that end I hacked the aspeed-i2c controller onto both i386/pc and arm/virt and posted kernel patches to enabled ACPI support for that device (more or less). It worked and has been useful in the meantime, but adding that i2c controller to those boards was obviously not going to be upstreamable - and to build reliable tests against it I don't want to carry this out of tree for ever. I messed around with a PCI hosted aspeed controller and might come back to that at some point.

In the meantime, DMTF published a transport binding for MCTP over USB
https://www.dmtf.org/sites/default/files/standards/documents/DSP0283_1.0.1.pdf

Kernel support duly followed early this year: drivers/net/mctp/mctp-usb.c
https://codeconstruct.com.au/docs/mctp-over-usb/

Given the ease of adding a suitable USB controller on a PCI bus, emulating a suitable endpoint provides what I think is an upstreamable solution.

To use this:
-device usb-ehci,id=ehci
-device usb-cxl-mctp,bus=ehci.0,id=fred,target=us0

where target is either a CXL switch upstream port, or a type 3 device.

Then install the mctp userspace tools in your guest and configure it with

mctp addr add 8 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb0 up

systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0

I've been testing the CXL commands with

https://gitlab.com/jic23/cxl-fmapi-tests
(mostly because I still had them in my image from the i2c work) but libcxlmi is probably a better bet

https://github.com/computexpresslink/libcxlmi

I'll post a tree with this on at gitlab.com/jic23/qemu shortly (cxl-<latest date>).

Jonathan Cameron (3):
hw/cxl/i2c_mctp_cxl: Initial device emulation
docs: cxl: Add example commandline for MCTP CXL CCIs usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.

Klaus Jensen (2):
hw/i2c: add smbus pec utility function
hw/i2c: add mctp core

 MAINTAINERS                               |   7 +
 docs/system/devices/cxl.rst               |  27 +
 include/hw/cxl/cxl_device.h               |   8 +
 include/hw/i2c/mctp.h                     | 125 +++++
 include/hw/i2c/smbus_master.h             |   2 +
 include/hw/pci-bridge/cxl_upstream_port.h |   1 +
 include/hw/usb.h                          |   1 +
 include/net/mctp.h                        | 100 ++++
 hw/cxl/cxl-mailbox-utils.c                |  49 ++
 hw/cxl/i2c_mctp_cxl.c                     | 289 ++++++++++
 hw/i2c/mctp.c                             | 414 ++++++++++++++
 hw/i2c/smbus_master.c                     |  26 +
 hw/usb/dev-mctp.c                         | 639 ++++++++++++++++++++++
 hw/arm/Kconfig                            |   1 +
 hw/cxl/Kconfig                            |   4 +
 hw/cxl/meson.build                        |   4 +
 hw/i2c/Kconfig                            |   4 +
 hw/i2c/meson.build                        |   1 +
 hw/i2c/trace-events                       |  14 +
 hw/usb/Kconfig                            |   5 +
 hw/usb/meson.build                        |   1 +
 21 files changed, 1722 insertions(+)
 create mode 100644 include/hw/i2c/mctp.h
 create mode 100644 include/net/mctp.h
 create mode 100644 hw/cxl/i2c_mctp_cxl.c
 create mode 100644 hw/i2c/mctp.c
 create mode 100644 hw/usb/dev-mctp.c
Signature · 1 line
2.48.1
#1 lore raw

RFC PATCH 1/5 hw/i2c: add smbus pec utility function

From: Klaus Jensen <[email protected]>

Add i2c_smbus_pec() to calculate the SMBus Packet Error Code for a message.

---
 include/hw/i2c/smbus_master.h |  2 ++
 hw/i2c/smbus_master.c         | 26 ++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
include/hw/i2c/smbus_master.h +2 -0
index bb13bc423c..d90f81767d 100644
@@ -27,6 +27,8 @@
 
 #include "hw/i2c/i2c.h"
 
+uint8_t i2c_smbus_pec(uint8_t crc, uint8_t *buf, size_t len);
+
 /* Master device commands.  */
 int smbus_quick_command(I2CBus *bus, uint8_t addr, int read);
 int smbus_receive_byte(I2CBus *bus, uint8_t addr);
hw/i2c/smbus_master.c +26 -0
index 6a53c34e70..01a8e47002 100644
@@ -15,6 +15,32 @@
 #include "hw/i2c/i2c.h"
 #include "hw/i2c/smbus_master.h"
 
+static uint8_t crc8(uint16_t data)
+{
+    int i;
+
+    for (i = 0; i < 8; i++) {
+        if (data & 0x8000) {
+            data ^= 0x1070U << 3;
+        }
+
+        data <<= 1;
+    }
+
+    return (uint8_t)(data >> 8);
+}
+
+uint8_t i2c_smbus_pec(uint8_t crc, uint8_t *buf, size_t len)
+{
+    int i;
+
+    for (i = 0; i < len; i++) {
+        crc = crc8((crc ^ buf[i]) << 8);
+    }
+
+    return crc;
+}
+
 /* Master device commands.  */
 int smbus_quick_command(I2CBus *bus, uint8_t addr, int read)
 {
Signature · 1 line
2.48.1
#2 lore raw

RFC PATCH 2/5 hw/i2c: add mctp core

In reply to Jonathan Cameron

From: Klaus Jensen <[email protected]>

Add an abstract MCTP over I2C endpoint model. This implements MCTP control message handling as well as handling the actual I2C transport (packetization).

Devices are intended to derive from this and implement the class methods.

Parts of this implementation is inspired by code[1] previously posted by Jonathan Cameron.

Squashed a fix[2] from Matt Johnston.

[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/qemu-devel/[email protected]/

---
 MAINTAINERS           |   7 +
 include/hw/i2c/mctp.h | 125 ++++++++++++
 include/net/mctp.h    |  35 ++++
 hw/i2c/mctp.c         | 431 ++++++++++++++++++++++++++++++++++++++++++
 hw/arm/Kconfig        |   1 +
 hw/i2c/Kconfig        |   4 +
 hw/i2c/meson.build    |   1 +
 hw/i2c/trace-events   |  14 ++
 8 files changed, 618 insertions(+)
MAINTAINERS +7 -0
index aa6763077e..4ad35a1aa3 100644
@@ -3782,6 +3782,13 @@ F: include/hw/fsi/*
 F: docs/specs/fsi.rst
 F: tests/qtest/aspeed_fsi-test.c
 
+MCTP I2C Transport
+M: Klaus Jensen <[email protected]>
+S: Maintained
+F: hw/i2c/mctp.c
+F: include/hw/i2c/mctp.h
+F: include/net/mctp.h
+
 Firmware schema specifications
 M: Philippe Mathieu-Daudé <[email protected]>
 R: Daniel P. Berrange <[email protected]>
include/hw/i2c/mctp.h new file +125 -0

125 lines not on this page · raw message

include/net/mctp.h new file +35 -0
new file mode 100644
index 0000000000..5d26d855db
@@ -0,0 +1,35 @@
+#ifndef QEMU_MCTP_H
+#define QEMU_MCTP_H
+
+#include "hw/registerfields.h"
+
+/* DSP0236 1.3.0, Section 8.3.1 */
+#define MCTP_BASELINE_MTU 64
+
+/* DSP0236 1.3.0, Table 1, Message body */
+FIELD(MCTP_MESSAGE_H, TYPE, 0, 7)
+FIELD(MCTP_MESSAGE_H, IC,   7, 1)
+
+/* DSP0236 1.3.0, Table 1, MCTP transport header */
+FIELD(MCTP_H_FLAGS, TAG,    0, 3);
+FIELD(MCTP_H_FLAGS, TO,     3, 1);
+FIELD(MCTP_H_FLAGS, PKTSEQ, 4, 2);
+FIELD(MCTP_H_FLAGS, EOM,    6, 1);
+FIELD(MCTP_H_FLAGS, SOM,    7, 1);
+
+/* DSP0236 1.3.0, Figure 4 */
+typedef struct MCTPPacketHeader {
+    uint8_t version;
+    struct {
+        uint8_t dest;
+        uint8_t source;
+    } eid;
+    uint8_t flags;
+} MCTPPacketHeader;
+
+typedef struct MCTPPacket {
+    MCTPPacketHeader hdr;
+    uint8_t          payload[];
+} MCTPPacket;
+
+#endif /* QEMU_MCTP_H */
hw/i2c/mctp.c new file +431 -0

431 lines not on this page · raw message

hw/arm/Kconfig +1 -0
index f543d944c3..d12f575c8d 100644
@@ -530,6 +530,7 @@ config ASPEED_SOC
     select DS1338
     select FTGMAC100
     select I2C
+    select I2C_MCTP
     select DPS310
     select PCA9552
     select SERIAL_MM
hw/i2c/Kconfig +4 -0
index 596a7a3165..794cccc053 100644
@@ -6,6 +6,10 @@ config I2C_DEVICES
     # to any board's i2c bus
     bool
 
+config I2C_MCTP
+    bool
+    select I2C
+
 config SMBUS
     bool
     select I2C
hw/i2c/meson.build +1 -0
index c459adcb59..a86457e610 100644
@@ -1,5 +1,6 @@
 i2c_ss = ss.source_set()
 i2c_ss.add(when: 'CONFIG_I2C', if_true: files('core.c'))
+i2c_ss.add(when: 'CONFIG_I2C_MCTP', if_true: files('mctp.c'))
 i2c_ss.add(when: 'CONFIG_SMBUS', if_true: files('smbus_slave.c', 'smbus_master.c'))
 i2c_ss.add(when: 'CONFIG_ACPI_SMBUS', if_true: files('pm_smbus.c'))
 i2c_ss.add(when: 'CONFIG_ACPI_ICH9', if_true: files('smbus_ich9.c'))
hw/i2c/trace-events +14 -0
index 1ad0e95c0e..da78fa327f 100644
@@ -61,3 +61,17 @@ pca954x_read_data(uint8_t value) "PCA954X read data: 0x%02x"
 
 imx_i2c_read(const char *id, const char *reg, uint64_t ofs, uint64_t value) "%s:[%s (0x%" PRIx64 ")] -> 0x%02" PRIx64
 imx_i2c_write(const char *id, const char *reg, uint64_t ofs, uint64_t value) "%s:[%s (0x%" PRIx64 ")] <- 0x%02" PRIx64
+
+# mctp.c
+
+i2c_mctp_tx_start_send(size_t len) "len %zu"
+i2c_mctp_tx_send_byte(size_t pos, uint8_t byte) "pos %zu byte 0x%"PRIx8""
+i2c_mctp_tx_done(void) "packet sent"
+i2c_mctp_handle_control(uint8_t command) "command 0x%"PRIx8""
+i2c_mctp_unhandled_control(uint8_t command) "command 0x%"PRIx8""
+i2c_mctp_drop_invalid_length(unsigned byte_count, size_t expected) "byte_count %u expected %zu"
+i2c_mctp_drop_invalid_pec(uint8_t pec, uint8_t expected) "pec 0x%"PRIx8" expected 0x%"PRIx8""
+i2c_mctp_drop_invalid_eid(uint8_t eid, uint8_t expected) "eid 0x%"PRIx8" expected 0x%"PRIx8""
+i2c_mctp_drop_invalid_pktseq(uint8_t pktseq, uint8_t expected) "pktseq 0x%"PRIx8" expected 0x%"PRIx8""
+i2c_mctp_drop_short_packet(size_t len) "len %zu"
+i2c_mctp_drop_expected_som(void) ""
Signature · 1 line
2.48.1
#3 lore raw

RFC PATCH 3/5 hw/cxl/i2c_mctp_cxl: Initial device emulation

In reply to Jonathan Cameron

The CCI and Fabric Manager APIs are used to configure CXL switches and devices. DMTF has defined an MCTP binding specification to carry these messages. The end goal of this work is to hook this up to emulated CXL switches and devices to allow control of the configuration.

Since this relies on i2c target mode, this can currently only be used with an SoC that includes the Aspeed I2C controller.

Note, only get timestamp added for now.

---
 include/hw/cxl/cxl_device.h               |   8 +
 include/hw/pci-bridge/cxl_upstream_port.h |   1 +
 hw/cxl/cxl-mailbox-utils.c                |  49 ++++
 hw/cxl/i2c_mctp_cxl.c                     | 289 ++++++++++++++++++++++
 hw/cxl/Kconfig                            |   4 +
 hw/cxl/meson.build                        |   4 +
 6 files changed, 355 insertions(+)
include/hw/cxl/cxl_device.h +8 -0
index 6086d4c85b..8d87c7151e 100644
@@ -360,6 +360,10 @@ int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd,
                             size_t len_in, uint8_t *pl_in,
                             size_t *len_out, uint8_t *pl_out,
                             bool *bg_started);
+
+void cxl_initialize_t3_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
+                               size_t payload_max);
+
 void cxl_initialize_t3_fm_owned_ld_mctpcci(CXLCCI *cci, DeviceState *d,
                                            DeviceState *intf,
                                            size_t payload_max);
@@ -367,6 +371,9 @@ void cxl_initialize_t3_fm_owned_ld_mctpcci(CXLCCI *cci, DeviceState *d,
 void cxl_initialize_t3_ld_cci(CXLCCI *cci, DeviceState *d,
                               DeviceState *intf, size_t payload_max);
 
+void cxl_initialize_usp_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
+                                size_t payload_max);
+
 #define cxl_device_cap_init(dstate, reg, cap_id, ver)                      \
     do {                                                                   \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                     \
@@ -606,6 +613,7 @@ struct CXLType3Dev {
     CXLComponentState cxl_cstate;
     CXLDeviceState cxl_dstate;
     CXLCCI cci; /* Primary PCI mailbox CCI */
+    CXLCCI oob_mctp_cci; /* Initialized only if targetted */
     /* Always initialized as no way to know if a VDM might show up */
     CXLCCI vdm_fm_owned_ld_mctp_cci;
     CXLCCI ld0_cci;
include/hw/pci-bridge/cxl_upstream_port.h +1 -0
index f208397ffe..7e529e0b5a 100644
@@ -12,6 +12,7 @@ typedef struct CXLUpstreamPort {
     /*< public >*/
     CXLComponentState cxl_cstate;
     CXLCCI swcci;
+    CXLCCI mctpcci;
 
     PCIExpLinkSpeed speed;
     PCIExpLinkWidth width;
hw/cxl/cxl-mailbox-utils.c +49 -0

58 lines not on this page · raw message

hw/cxl/i2c_mctp_cxl.c new file +289 -0

289 lines not on this page · raw message

hw/cxl/Kconfig +4 -0
index 8e67519b16..bc259fdf67 100644
@@ -1,3 +1,7 @@
 config CXL
     bool
     default y if PCI_EXPRESS
+
+config I2C_MCTP_CXL
+    bool
+    default y if CXL && I2C_MCTP
hw/cxl/meson.build +4 -0
index e3abb49d27..90fd83f680 100644
@@ -12,3 +12,7 @@ system_ss.add(when: 'CONFIG_CXL',
                if_false: files(
                    'cxl-host-stubs.c',
                ))
+system_ss.add(when: 'CONFIG_I2C_MCTP_CXL', if_true: files('i2c_mctp_cxl.c'))
+
+system_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))
+
Signature · 1 line
2.48.1
#4 lore raw

Re: [RFC PATCH qemu 3/5] hw/cxl/i2c_mctp_cxl: Initial device emulation

Quoted 11 lines · On Mon, 9 Jun 2025 17:33:31 +0100 Jonathan Cameron via <[email protected]>…

The CCI and Fabric Manager APIs are used to configure CXL switches and devices. DMTF has defined an MCTP binding specification to carry these messages. The end goal of this work is to hook this up to emulated CXL switches and devices to allow control of the configuration.

Since this relies on i2c target mode, this can currently only be used with an SoC that includes the Aspeed I2C controller.

Note, only get timestamp added for now.

Not very important given this is an RFC, but a chunk of code in here isn't used so I'll drop it from future version...
(I noticed whilst wondering why Anisa didn't update it in they
FMAPI DCD series).

Quoted 43 lines
---
 include/hw/cxl/cxl_device.h               |   8 +
 include/hw/pci-bridge/cxl_upstream_port.h |   1 +
 hw/cxl/cxl-mailbox-utils.c                |  49 ++++
 hw/cxl/i2c_mctp_cxl.c                     | 289 ++++++++++++++++++++++
 hw/cxl/Kconfig                            |   4 +
 hw/cxl/meson.build                        |   4 +
 6 files changed, 355 insertions(+)
include/hw/cxl/cxl_device.h +4 -0
index 6086d4c85b..8d87c7151e 100644
@@ -360,6 +360,10 @@ int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd,
                             size_t len_in, uint8_t *pl_in,
                             size_t *len_out, uint8_t *pl_out,
                             bool *bg_started);
+
+void cxl_initialize_t3_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
+                               size_t payload_max);
+
 
hw/cxl/cxl-mailbox-utils.c +13 -0
index 475547f212..4c9852642e 100644
@@ -3690,6 +3690,29 @@ void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max)
     cxl_init_cci(cci, payload_max);
 }
 
+static const struct cxl_cmd cxl_cmd_set_t3_mctp[256][256] = {
+    [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 },
+    [INFOSTAT][GET_RESPONSE_MSG_LIMIT] = { "GET_RESPONSE_MSG_LIMIT",
+                                           cmd_get_response_msg_limit, 0, 0 },
+    [INFOSTAT][SET_RESPONSE_MSG_LIMIT] = { "SET_RESPONSE_MSG_LIMIT",
+                                           cmd_set_response_msg_limit, 1, 0 },
+    [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 },
+    [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0,
+                              0 },
+    [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
+    [TUNNEL][MANAGEMENT_COMMAND] = { "TUNNEL_MANAGEMENT_COMMAND",
+                                     cmd_tunnel_management_cmd, ~0, 0 },
+};

hw/cxl/cxl-mailbox-utils.c void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) · line 3715

Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10 -0
+
+void cxl_initialize_t3_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
+                               size_t payload_max)
+{
+    cxl_copy_cci_commands(cci, cxl_cmd_set_t3_mctp);
+    cci->d = d;
+    cci->intf = intf;
+    cxl_init_cci(cci, payload_max);
+}
+

At some point a while back, when adding tunneling we decided to pretend we have an MLD (because it's more general) and as such introduced the fm owned mctpcci. That's the one one that is used for the landing point of mctp links to a type 3 device. So this code is unused.

#5 lore raw

RFC PATCH 4/5 docs: cxl: Add example commandline for MCTP CXL CCIs

In reply to Jonathan Cameron

Add initial documentation for the MCTP over I2C management device. At current time this can only be used with the Aspeed I2C controller which is only available in aspeed SoCs, though can be added to other emulated boards.

---
 docs/system/devices/cxl.rst | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
docs/system/devices/cxl.rst +27 -0
index e307caf3f8..523f99f205 100644
@@ -406,6 +406,33 @@ OS management of CXL memory devices as described here.
 * CONFIG_CXL_PORT
 * CONFIG_CXL_REGION
 
+
+CCI access via MCTP over I2C
+----------------------------
+
+In order to make use of this device, an I2C controller that supports MCTP
+is required.  The aspeed-i2c controller is an example of such a controller.
+
+Both CXL switches and CXL Type 3 devices support configuration via
+MCTP access to Component Command Interfaces (CCIs) on the devices.
+
+Example configuration:
+
+ -device cxl-upstream,port=33,bus=root_port0,id=us0,multifunction=on,addr=0.0,sn=12345678 \
+ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
+ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
+ -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \
+ -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem1,id=cxl-pmem0,lsa=cxl-lsa1,sn=3 \
+ -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem2,id=cxl-pmem1,lsa=cxl-lsa2,sn=4 \
+ -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem3,id=cxl-pmem2,lsa=cxl-lsa3,sn=5 \
+ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \
+ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=5,target=cxl-pmem0 \
+ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-pmem1 \
+ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=7,target=cxl-pmem2
+
+Guest OS communication with the MCTP CCI can then be established using standard
+MCTP configuration tools.
+
 References
 ----------
 
Signature · 1 line
2.48.1
#6 lore raw

RFC PATCH 5/5 usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.

In reply to Jonathan Cameron

Given the challenges in testing systems with the previously posted i2c based mctp support for x86 / arm64 machines (as only emulated controller that is suitably capable is the aspeed one) providing a USB interface greatly simplifies things.

This is a PoC of such an interface.

RFC reasons:
- Not all error conditions are fully checked / handled.
- MTU in the 'to host' direction is ignored. Seems the Linux stack doesn't care about this today but that should be brought into compliance with the specification anyway.

---
 include/hw/usb.h   |   1 +
 include/net/mctp.h |  77 +++++-
 hw/i2c/mctp.c      |  17 --
 hw/usb/dev-mctp.c  | 639 +++++++++++++++++++++++++++++++++++++++++++++
 hw/usb/Kconfig     |   5 +
 hw/usb/meson.build |   1 +
 6 files changed, 717 insertions(+), 23 deletions(-)
include/hw/usb.h +1 -0
index 26a9f3ecde..ae5f3f77cb 100644
@@ -81,6 +81,7 @@
 #define USB_CLASS_CDC_DATA              0x0a
 #define USB_CLASS_CSCID                 0x0b
 #define USB_CLASS_CONTENT_SEC           0x0d
+#define USB_CLASS_MCTP                  0x14
 #define USB_CLASS_APP_SPEC              0xfe
 #define USB_CLASS_VENDOR_SPEC           0xff
 
include/net/mctp.h +71 -6

103 lines not on this page · raw message

hw/i2c/mctp.c +0 -17
index cf45a46706..ac9bebb3c4 100644
@@ -33,23 +33,6 @@ typedef struct MCTPI2CPacket {
 #define i2c_mctp_payload_offset offsetof(MCTPI2CPacket, mctp.payload)
 #define i2c_mctp_payload(buf) (buf + i2c_mctp_payload_offset)
 
-/* DSP0236 1.3.0, Figure 20 */
-typedef struct MCTPControlMessage {
-#define MCTP_MESSAGE_TYPE_CONTROL 0x0
-    uint8_t type;
-#define MCTP_CONTROL_FLAGS_RQ               (1 << 7)
-#define MCTP_CONTROL_FLAGS_D                (1 << 6)
-    uint8_t flags;
-    uint8_t command_code;
-    uint8_t data[];
-} MCTPControlMessage;
-
-enum MCTPControlCommandCodes {
-    MCTP_CONTROL_SET_EID                    = 0x01,
-    MCTP_CONTROL_GET_EID                    = 0x02,
-    MCTP_CONTROL_GET_VERSION                = 0x04,
-    MCTP_CONTROL_GET_MESSAGE_TYPE_SUPPORT   = 0x05,
-};
 
 #define MCTP_CONTROL_ERROR_UNSUPPORTED_CMD 0x5
 
hw/usb/dev-mctp.c new file +639 -0

639 lines not on this page · raw message

hw/usb/Kconfig +5 -0
index 69c663be52..4340475708 100644
@@ -105,6 +105,11 @@ config USB_SERIAL
     default y
     depends on USB
 
+config USB_MCTP
+    bool
+    default y if CXL
+    depends on USB
+
 config USB_NETWORK
     bool
     default y
hw/usb/meson.build +1 -0
index 17360a5b5a..c2910f9818 100644
@@ -43,6 +43,7 @@ system_ss.add(when: 'CONFIG_USB_STORAGE_UAS', if_true: files('dev-uas.c'))
 system_ss.add(when: 'CONFIG_USB_AUDIO', if_true: files('dev-audio.c'))
 system_ss.add(when: 'CONFIG_USB_SERIAL', if_true: files('dev-serial.c'))
 system_ss.add(when: 'CONFIG_USB_NETWORK', if_true: files('dev-network.c'))
+system_ss.add(when: 'CONFIG_USB_MCTP', if_true: files('dev-mctp.c'))
 if host_os != 'windows'
   system_ss.add(when: 'CONFIG_USB_STORAGE_MTP', if_true: files('dev-mtp.c'))
 endif
Signature · 1 line
2.48.1
#7 lore raw
In reply to Jonathan Cameron
Quoted 58 lines · Jonathan Cameron

This posting is primarily about sharing the USB device emulation to get some early feedback.

RFC reasons:
- Known 'inaccuracies' in emulation (not obeying MTU in the to host direction for example)./
- Not sure what to do wrt to Klaus' I2C MCTP support given that has been stalled for some time. For now only the headers are really shared between the two implementations.
- This is more of an FYI / request for testing than a formal suggestion that this might be ready for upstream.

Why add a CXL FM-API over MCTP over USB device?
- Can be emulated on pretty much any host system as USB is discoverable and expandable. If you want a giggle, see the hacks on i386/pc and arm/virt on we've been using until now given only I2C controller that works is the aspeed one. e.g. https://gitlab.com/jic23/qemu/-/commit/134c2e3952b
- Being able to talk to both the fabric management out of band interfaces and the in band devices on the same host makes testing much simpler.

Background:

Back in 2022 I posted some support for controlling the CXL fabric via the spec defined out of band interfaces (CXL Fabric Management API - FM-API) over MCTP on I2C

https://lore.kernel.org/qemu-devel/[email protected]/
I reworked that on top of the NVME-MI work from Klauss.

To that end I hacked the aspeed-i2c controller onto both i386/pc and arm/virt and posted kernel patches to enabled ACPI support for that device (more or less). It worked and has been useful in the meantime, but adding that i2c controller to those boards was obviously not going to be upstreamable - and to build reliable tests against it I don't want to carry this out of tree for ever. I messed around with a PCI hosted aspeed controller and might come back to that at some point.

In the meantime, DMTF published a transport binding for MCTP over USB
https://www.dmtf.org/sites/default/files/standards/documents/DSP0283_1.0.1.pdf

Kernel support duly followed early this year: drivers/net/mctp/mctp-usb.c
https://codeconstruct.com.au/docs/mctp-over-usb/

Given the ease of adding a suitable USB controller on a PCI bus, emulating a suitable endpoint provides what I think is an upstreamable solution.

To use this:
-device usb-ehci,id=ehci
-device usb-cxl-mctp,bus=ehci.0,id=fred,target=us0

where target is either a CXL switch upstream port, or a type 3 device.

Then install the mctp userspace tools in your guest and configure it with

mctp addr add 8 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb0 up

Quoted 9 lines

systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0

I've been testing the CXL commands with

https://gitlab.com/jic23/cxl-fmapi-tests
(mostly because I still had them in my image from the i2c work) but libcxlmi is probably a better bet

I tested this patchset using libcxlmi with the following configuration:
- kernel version: https://github.com/weiny2/linux-kernel/tree/dcd-v6-2025-04-13
- QEMU: upstream ToT with MCTP USB CXL and FMAPI DCD support:
- MCTP USB CXL (https://lore.kernel.org/linux-cxl/[email protected]/T/#m21b9e0dfc689cb1890bb4d961710c23379e04902)

	- note: I modified the condition on line hw/usb/dev-mctp.c:509 to allow
	  the FMAPI DCD Management command set (0x56) to be processed

- FMAPI DCD Management patches (https://lore.kernel.org/linux-cxl/[email protected]/)

- Topology (1 DCD attached to downstream port of CXL Switch)

	'-device usb-ehci,id=ehci \
     -object memory-backend-file,id=cxl-mem1,mem-path=/tmp/t3_cxl1.raw,size=4G \
     -object memory-backend-file,id=cxl-lsa1,mem-path=/tmp/t3_lsa1.raw,size=1M \
     -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
     -device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
     -device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
     -device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
     -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
     -device cxl-type3,bus=swport0,volatile-dc-memdev=cxl-mem1,id=cxl-dcd0,lsa=cxl-lsa1,num-dc-regions=2,sn=99 \
     -device usb-cxl-mctp,bus=ehci.0,id=usb0,target=us0 \
     -device usb-cxl-mctp,bus=ehci.0,id=usb1,target=cxl-dcd0\
     -machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k'

After creating the VM with the above topology, to configure the 2 MCTP EPs (mctpusb0 and mctpusb1):

mctp link set mctpusb0 up
mctp addr add 50 dev mctpusb0
mctp link set mctpusb0 net 11

mctp link set mctpusb1 up
mctp addr add 51 dev mctpusb1
mctp link set mctpusb0 net 12

systemctl stop mctpd.service
systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0 busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb1 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0

To test this patchset, I used this program from libcxlmi:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/cxl-mctp.c

To summarize, this program scans the D-bus for MCTP endpoints or takes the NID:EID of the endpoint you want to open as arguments, then sends a variety of commands (identify, supported logs, etc.) through the endpoint(s) and prints the results, which I verified. Both methods of opening endpoints are successful and shows the expected output.

To ensure this works end-to-end with an FM workflow I also tested with this program, which allows users to interactively send the FMAPI Add/Release commands from the command line:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/fmapi-mctp.c

First I created a DC Region using this ndctl command:
cxl create-region -m mem0 -d decoder0.0 -s 1G -t dynamic_ram_a

Then I ran the program to add an extent of 512MB and was able to successfully create a DAX Device using the following commands:

daxctl create-device -r region0
daxctl reconfigure-device dax0.1 -m system-ram

The output of 'lsmem' shows 512M added:
RANGE SIZE STATE REMOVABLE BLOCK

0x0000000000000000-0x000000007fffffff    2G online       yes    0-15
0x0000000100000000-0x000000027fffffff    6G online       yes   32-79
0x0000001290000000-0x00000012afffffff  512M online       yes 594-597
Quoted 45 lines

https://github.com/computexpresslink/libcxlmi

I'll post a tree with this on at gitlab.com/jic23/qemu shortly (cxl-<latest date>).

Jonathan Cameron (3):
hw/cxl/i2c_mctp_cxl: Initial device emulation
docs: cxl: Add example commandline for MCTP CXL CCIs usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.

Klaus Jensen (2):
hw/i2c: add smbus pec utility function
hw/i2c: add mctp core

 MAINTAINERS                               |   7 +
 docs/system/devices/cxl.rst               |  27 +
 include/hw/cxl/cxl_device.h               |   8 +
 include/hw/i2c/mctp.h                     | 125 +++++
 include/hw/i2c/smbus_master.h             |   2 +
 include/hw/pci-bridge/cxl_upstream_port.h |   1 +
 include/hw/usb.h                          |   1 +
 include/net/mctp.h                        | 100 ++++
 hw/cxl/cxl-mailbox-utils.c                |  49 ++
 hw/cxl/i2c_mctp_cxl.c                     | 289 ++++++++++
 hw/i2c/mctp.c                             | 414 ++++++++++++++
 hw/i2c/smbus_master.c                     |  26 +
 hw/usb/dev-mctp.c                         | 639 ++++++++++++++++++++++
 hw/arm/Kconfig                            |   1 +
 hw/cxl/Kconfig                            |   4 +
 hw/cxl/meson.build                        |   4 +
 hw/i2c/Kconfig                            |   4 +
 hw/i2c/meson.build                        |   1 +
 hw/i2c/trace-events                       |  14 +
 hw/usb/Kconfig                            |   5 +
 hw/usb/meson.build                        |   1 +
 21 files changed, 1722 insertions(+)
 create mode 100644 include/hw/i2c/mctp.h
 create mode 100644 include/net/mctp.h
 create mode 100644 hw/cxl/i2c_mctp_cxl.c
 create mode 100644 hw/i2c/mctp.c
 create mode 100644 hw/usb/dev-mctp.c
Signature · 1 line
2.48.1