hw/cxl/mctp/i2c/usb: MCTP for OoB control of CXL devices.
Subsystems (3)
cover hw/cxl/mctp/i2c/usb: MCTP for OoB control of CXL devices.
This posting is primarily about sharing the USB device emulation to get some early feedback.
RFC reasons:
- Known 'inaccuracies' in emulation (not obeying MTU in the to host direction for example)./
- Not sure what to do wrt to Klaus' I2C MCTP support given that has been stalled for some time. For now only the headers are really shared between the two implementations.
- This is more of an FYI / request for testing than a formal suggestion that this might be ready for upstream.
Why add a CXL FM-API over MCTP over USB device?
- Can be emulated on pretty much any host system as USB is discoverable and expandable. If you want a giggle, see the hacks on i386/pc and arm/virt on we've been using until now given only I2C controller that works is the aspeed one. e.g. https://gitlab.com/jic23/qemu/-/commit/134c2e3952b
- Being able to talk to both the fabric management out of band interfaces and the in band devices on the same host makes testing much simpler.
Background:
Back in 2022 I posted some support for controlling the CXL fabric via the spec defined out of band interfaces (CXL Fabric Management API - FM-API) over MCTP on I2C
https://lore.kernel.org/qemu-devel/[email protected]/
I reworked that on top of the NVME-MI work from Klauss.
To that end I hacked the aspeed-i2c controller onto both i386/pc and arm/virt and posted kernel patches to enabled ACPI support for that device (more or less). It worked and has been useful in the meantime, but adding that i2c controller to those boards was obviously not going to be upstreamable - and to build reliable tests against it I don't want to carry this out of tree for ever. I messed around with a PCI hosted aspeed controller and might come back to that at some point.
In the meantime, DMTF published a transport binding for MCTP over USB
https://www.dmtf.org/sites/default/files/standards/documents/DSP0283_1.0.1.pdf
Kernel support duly followed early this year: drivers/net/mctp/mctp-usb.c
https://codeconstruct.com.au/docs/mctp-over-usb/
Given the ease of adding a suitable USB controller on a PCI bus, emulating a suitable endpoint provides what I think is an upstreamable solution.
To use this:
-device usb-ehci,id=ehci
-device usb-cxl-mctp,bus=ehci.0,id=fred,target=us0
where target is either a CXL switch upstream port, or a type 3 device.
Then install the mctp userspace tools in your guest and configure it with
mctp addr add 8 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb0 up
systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0
I've been testing the CXL commands with
https://gitlab.com/jic23/cxl-fmapi-tests
(mostly because I still had them in my image from the i2c work) but libcxlmi is probably a better bet
https://github.com/computexpresslink/libcxlmi
I'll post a tree with this on at gitlab.com/jic23/qemu shortly (cxl-<latest date>).
Jonathan Cameron (3):
hw/cxl/i2c_mctp_cxl: Initial device emulation
docs: cxl: Add example commandline for MCTP CXL CCIs usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.
Klaus Jensen (2):
hw/i2c: add smbus pec utility function
hw/i2c: add mctp core
MAINTAINERS | 7 + docs/system/devices/cxl.rst | 27 + include/hw/cxl/cxl_device.h | 8 + include/hw/i2c/mctp.h | 125 +++++ include/hw/i2c/smbus_master.h | 2 + include/hw/pci-bridge/cxl_upstream_port.h | 1 + include/hw/usb.h | 1 + include/net/mctp.h | 100 ++++ hw/cxl/cxl-mailbox-utils.c | 49 ++ hw/cxl/i2c_mctp_cxl.c | 289 ++++++++++ hw/i2c/mctp.c | 414 ++++++++++++++ hw/i2c/smbus_master.c | 26 + hw/usb/dev-mctp.c | 639 ++++++++++++++++++++++ hw/arm/Kconfig | 1 + hw/cxl/Kconfig | 4 + hw/cxl/meson.build | 4 + hw/i2c/Kconfig | 4 + hw/i2c/meson.build | 1 + hw/i2c/trace-events | 14 + hw/usb/Kconfig | 5 + hw/usb/meson.build | 1 + 21 files changed, 1722 insertions(+) create mode 100644 include/hw/i2c/mctp.h create mode 100644 include/net/mctp.h create mode 100644 hw/cxl/i2c_mctp_cxl.c create mode 100644 hw/i2c/mctp.c create mode 100644 hw/usb/dev-mctp.c
Signature · 1 line
2.48.1
RFC PATCH 1/5 hw/i2c: add smbus pec utility function
From: Klaus Jensen <[email protected]>
Add i2c_smbus_pec() to calculate the SMBus Packet Error Code for a message.
--- include/hw/i2c/smbus_master.h | 2 ++ hw/i2c/smbus_master.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
include/hw/i2c/smbus_master.h +2 -0
@@ -27,6 +27,8 @@#include "hw/i2c/i2c.h" +uint8_t i2c_smbus_pec (uint8_t crc,uint8_t *buf,size_t len); +/* Master device commands. */ int smbus_quick_command (I2CBus *bus,uint8_t addr,int read);int smbus_receive_byte (I2CBus *bus,uint8_t addr);
hw/i2c/smbus_master.c +26 -0
@@ -15,6 +15,32 @@#include "hw/i2c/i2c.h" #include "hw/i2c/smbus_master.h" +static uint8_t crc8 (uint16_t data) +{ +int i; + +for (i =0 ; i <8 ; i++) { +if (data &0x8000 ) { + data ^=0x1070U <<3 ; + } + + data <<=1 ; + } + +return (uint8_t )(data >>8 ); +} + +uint8_t i2c_smbus_pec (uint8_t crc,uint8_t *buf,size_t len) +{ +int i; + +for (i =0 ; i < len; i++) { + crc =crc8 ((crc ^ buf[i]) <<8 ); + } + +return crc; +} +/* Master device commands. */ int smbus_quick_command (I2CBus *bus,uint8_t addr,int read) {
Signature · 1 line
2.48.1
RFC PATCH 2/5 hw/i2c: add mctp core
In reply to Jonathan CameronFrom: Klaus Jensen <[email protected]>
Add an abstract MCTP over I2C endpoint model. This implements MCTP control message handling as well as handling the actual I2C transport (packetization).
Devices are intended to derive from this and implement the class methods.
Parts of this implementation is inspired by code[1] previously posted by Jonathan Cameron.
Squashed a fix[2] from Matt Johnston.
[1]: https://lore.kernel.org/qemu-devel/[email protected]/
[2]: https://lore.kernel.org/qemu-devel/[email protected]/
--- MAINTAINERS | 7 + include/hw/i2c/mctp.h | 125 ++++++++++++ include/net/mctp.h | 35 ++++ hw/i2c/mctp.c | 431 ++++++++++++++++++++++++++++++++++++++++++ hw/arm/Kconfig | 1 + hw/i2c/Kconfig | 4 + hw/i2c/meson.build | 1 + hw/i2c/trace-events | 14 ++ 8 files changed, 618 insertions(+)
MAINTAINERS +7 -0
@@ -3782,6 +3782,13 @@ F: include/hw/fsi/* F: docs/specs/fsi.rst F: tests/qtest/aspeed_fsi-test.c +MCTP I2C Transport +M: Klaus Jensen <[email protected]> +S: Maintained +F: hw/i2c/mctp.c +F: include/hw/i2c/mctp.h +F: include/net/mctp.h + Firmware schema specifications M: Philippe Mathieu-Daudé <[email protected]> R: Daniel P. Berrange <[email protected]>
include/hw/i2c/mctp.h new file +125 -0
125 lines not on this page · raw message
include/net/mctp.h new file +35 -0
@@ -0,0 +1,35 @@ +#ifndef QEMU_MCTP_H +#define QEMU_MCTP_H + +#include "hw/registerfields.h" + +/* DSP0236 1.3.0, Section 8.3.1 */ +#define MCTP_BASELINE_MTU 64 + +/* DSP0236 1.3.0, Table 1, Message body */ +FIELD(MCTP_MESSAGE_H, TYPE, 0, 7) +FIELD (MCTP_MESSAGE_H ,IC ,7 ,1 ) + +/* DSP0236 1.3.0, Table 1, MCTP transport header */ +FIELD (MCTP_H_FLAGS ,TAG ,0 ,3 ); +FIELD (MCTP_H_FLAGS ,TO ,3 ,1 ); +FIELD (MCTP_H_FLAGS ,PKTSEQ ,4 ,2 ); +FIELD (MCTP_H_FLAGS ,EOM ,6 ,1 ); +FIELD (MCTP_H_FLAGS ,SOM ,7 ,1 ); + +/* DSP0236 1.3.0, Figure 4 */ +typedef struct MCTPPacketHeader { +uint8_t version; +struct { +uint8_t dest; +uint8_t source; + } eid; +uint8_t flags; +}MCTPPacketHeader ; + +typedef struct MCTPPacket { +MCTPPacketHeader hdr; +uint8_t payload[]; +}MCTPPacket ; + +#endif /* QEMU_MCTP_H */
hw/i2c/mctp.c new file +431 -0
431 lines not on this page · raw message
hw/arm/Kconfig +1 -0
@@ -530,6 +530,7 @@ config ASPEED_SOC select DS1338 select FTGMAC100 select I2C + select I2C_MCTP select DPS310 select PCA9552 select SERIAL_MM
hw/i2c/Kconfig +4 -0
@@ -6,6 +6,10 @@ config I2C_DEVICES # to any board's i2c bus bool +config I2C_MCTP + bool + select I2C + config SMBUS bool select I2C
hw/i2c/meson.build +1 -0
@@ -1,5 +1,6 @@ i2c_ss = ss.source_set() i2c_ss.add(when: 'CONFIG_I2C', if_true: files('core.c')) +i2c_ss.add(when: 'CONFIG_I2C_MCTP', if_true: files('mctp.c')) i2c_ss.add(when: 'CONFIG_SMBUS', if_true: files('smbus_slave.c', 'smbus_master.c')) i2c_ss.add(when: 'CONFIG_ACPI_SMBUS', if_true: files('pm_smbus.c')) i2c_ss.add(when: 'CONFIG_ACPI_ICH9', if_true: files('smbus_ich9.c'))
hw/i2c/trace-events +14 -0
@@ -61,3 +61,17 @@ pca954x_read_data(uint8_t value) "PCA954X read data: 0x%02x" imx_i2c_read(const char *id, const char *reg, uint64_t ofs, uint64_t value) "%s:[%s (0x%" PRIx64 ")] -> 0x%02" PRIx64 imx_i2c_write(const char *id, const char *reg, uint64_t ofs, uint64_t value) "%s:[%s (0x%" PRIx64 ")] <- 0x%02" PRIx64 + +# mctp.c + +i2c_mctp_tx_start_send(size_t len) "len %zu" +i2c_mctp_tx_send_byte(size_t pos, uint8_t byte) "pos %zu byte 0x%"PRIx8"" +i2c_mctp_tx_done(void) "packet sent" +i2c_mctp_handle_control(uint8_t command) "command 0x%"PRIx8"" +i2c_mctp_unhandled_control(uint8_t command) "command 0x%"PRIx8"" +i2c_mctp_drop_invalid_length(unsigned byte_count, size_t expected) "byte_count %u expected %zu" +i2c_mctp_drop_invalid_pec(uint8_t pec, uint8_t expected) "pec 0x%"PRIx8" expected 0x%"PRIx8"" +i2c_mctp_drop_invalid_eid(uint8_t eid, uint8_t expected) "eid 0x%"PRIx8" expected 0x%"PRIx8"" +i2c_mctp_drop_invalid_pktseq(uint8_t pktseq, uint8_t expected) "pktseq 0x%"PRIx8" expected 0x%"PRIx8"" +i2c_mctp_drop_short_packet(size_t len) "len %zu" +i2c_mctp_drop_expected_som(void) ""
Signature · 1 line
2.48.1
RFC PATCH 3/5 hw/cxl/i2c_mctp_cxl: Initial device emulation
In reply to Jonathan CameronThe CCI and Fabric Manager APIs are used to configure CXL switches and devices. DMTF has defined an MCTP binding specification to carry these messages. The end goal of this work is to hook this up to emulated CXL switches and devices to allow control of the configuration.
Since this relies on i2c target mode, this can currently only be used with an SoC that includes the Aspeed I2C controller.
Note, only get timestamp added for now.
--- include/hw/cxl/cxl_device.h | 8 + include/hw/pci-bridge/cxl_upstream_port.h | 1 + hw/cxl/cxl-mailbox-utils.c | 49 ++++ hw/cxl/i2c_mctp_cxl.c | 289 ++++++++++++++++++++++ hw/cxl/Kconfig | 4 + hw/cxl/meson.build | 4 + 6 files changed, 355 insertions(+)
include/hw/cxl/cxl_device.h +8 -0
@@ -360,6 +360,10 @@ int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd, size_t len_in, uint8_t *pl_in, size_t *len_out, uint8_t *pl_out, bool *bg_started); + +void cxl_initialize_t3_mctpcci (CXLCCI *cci,DeviceState *d,DeviceState *intf, +size_t payload_max); +void cxl_initialize_t3_fm_owned_ld_mctpcci (CXLCCI *cci,DeviceState *d,DeviceState *intf,size_t payload_max);
@@ -367,6 +371,9 @@ void cxl_initialize_t3_fm_owned_ld_mctpcci(CXLCCI *cci, DeviceState *d,void cxl_initialize_t3_ld_cci (CXLCCI *cci,DeviceState *d,DeviceState *intf,size_t payload_max); +void cxl_initialize_usp_mctpcci (CXLCCI *cci,DeviceState *d,DeviceState *intf, +size_t payload_max); +#define cxl_device_cap_init (dstate, reg, cap_id, ver) \ do { \ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
@@ -606,6 +613,7 @@ struct CXLType3Dev {CXLComponentState cxl_cstate;CXLDeviceState cxl_dstate;CXLCCI cci;/* Primary PCI mailbox CCI */ +CXLCCI oob_mctp_cci;/* Initialized only if targetted */ /* Always initialized as no way to know if a VDM might show up */ CXLCCI vdm_fm_owned_ld_mctp_cci;CXLCCI ld0_cci;
include/hw/pci-bridge/cxl_upstream_port.h +1 -0
@@ -12,6 +12,7 @@ typedef struct CXLUpstreamPort {/*< public >*/ CXLComponentState cxl_cstate;CXLCCI swcci; +CXLCCI mctpcci;PCIExpLinkSpeed speed;PCIExpLinkWidth width;
hw/cxl/cxl-mailbox-utils.c +49 -0
58 lines not on this page · raw message
hw/cxl/i2c_mctp_cxl.c new file +289 -0
289 lines not on this page · raw message
hw/cxl/Kconfig +4 -0
@@ -1,3 +1,7 @@ config CXL bool default y if PCI_EXPRESS + +config I2C_MCTP_CXL + bool + default y if CXL && I2C_MCTP
hw/cxl/meson.build +4 -0
@@ -12,3 +12,7 @@ system_ss.add(when: 'CONFIG_CXL', if_false: files( 'cxl-host-stubs.c', )) +system_ss.add(when: 'CONFIG_I2C_MCTP_CXL', if_true: files('i2c_mctp_cxl.c')) + +system_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c')) +
Signature · 1 line
2.48.1
Re: [RFC PATCH qemu 3/5] hw/cxl/i2c_mctp_cxl: Initial device emulation
Quoted 11 lines · On Mon, 9 Jun 2025 17:33:31 +0100 Jonathan Cameron via <[email protected]>…
The CCI and Fabric Manager APIs are used to configure CXL switches and devices. DMTF has defined an MCTP binding specification to carry these messages. The end goal of this work is to hook this up to emulated CXL switches and devices to allow control of the configuration.
Since this relies on i2c target mode, this can currently only be used with an SoC that includes the Aspeed I2C controller.
Note, only get timestamp added for now.
Signed-off-by Jonathan Cameron <[email protected]>
Not very important given this is an RFC, but a chunk of code in here isn't used so I'll drop it from future version...
(I noticed whilst wondering why Anisa didn't update it in they
FMAPI DCD series).
Quoted 43 lines
--- include/hw/cxl/cxl_device.h | 8 + include/hw/pci-bridge/cxl_upstream_port.h | 1 + hw/cxl/cxl-mailbox-utils.c | 49 ++++ hw/cxl/i2c_mctp_cxl.c | 289 ++++++++++++++++++++++ hw/cxl/Kconfig | 4 + hw/cxl/meson.build | 4 + 6 files changed, 355 insertions(+)include/hw/cxl/cxl_device.h +4
-0@@ -360,6 +360,10 @@ int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd, size_t len_in, uint8_t *pl_in, size_t *len_out, uint8_t *pl_out, bool *bg_started); + +void cxl_initialize_t3_mctpcci (CXLCCI *cci,DeviceState *d,DeviceState *intf, +size_t payload_max); +hw/cxl/cxl-mailbox-utils.c +13
-0@@ -3690,6 +3690,29 @@ void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max)cxl_init_cci (cci, payload_max); } +static const struct cxl_cmd cxl_cmd_set_t3_mctp[256 ][256 ] = { + [INFOSTAT ][IS_IDENTIFY ] = {"IDENTIFY" , cmd_infostat_identify,0 ,0 }, + [INFOSTAT ][GET_RESPONSE_MSG_LIMIT ] = {"GET_RESPONSE_MSG_LIMIT" , + cmd_get_response_msg_limit,0 ,0 }, + [INFOSTAT ][SET_RESPONSE_MSG_LIMIT ] = {"SET_RESPONSE_MSG_LIMIT" , + cmd_set_response_msg_limit,1 ,0 }, + [TIMESTAMP ][GET ] = {"TIMESTAMP_GET" , cmd_timestamp_get,0 ,0 }, + [LOGS ][GET_SUPPORTED ] = {"LOGS_GET_SUPPORTED" , cmd_logs_get_supported,0 , +0 }, + [LOGS ][GET_LOG ] = {"LOGS_GET_LOG" , cmd_logs_get_log,0x18 ,0 }, + [TUNNEL ][MANAGEMENT_COMMAND ] = {"TUNNEL_MANAGEMENT_COMMAND" , + cmd_tunnel_management_cmd, ~0 ,0 }, +};
hw/cxl/cxl-mailbox-utils.c void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) · line 3715
Quoted 10 lines
hw/cxl/cxl-mailbox-utils.c +10
-0+ +void cxl_initialize_t3_mctpcci (CXLCCI *cci,DeviceState *d,DeviceState *intf, +size_t payload_max) +{ +cxl_copy_cci_commands (cci, cxl_cmd_set_t3_mctp); + cci->d = d; + cci->intf = intf; +cxl_init_cci (cci, payload_max); +} +
At some point a while back, when adding tunneling we decided to pretend we have an MLD (because it's more general) and as such introduced the fm owned mctpcci. That's the one one that is used for the landing point of mctp links to a type 3 device. So this code is unused.
RFC PATCH 4/5 docs: cxl: Add example commandline for MCTP CXL CCIs
In reply to Jonathan CameronAdd initial documentation for the MCTP over I2C management device. At current time this can only be used with the Aspeed I2C controller which is only available in aspeed SoCs, though can be added to other emulated boards.
--- docs/system/devices/cxl.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
docs/system/devices/cxl.rst +27 -0
@@ -406,6 +406,33 @@ OS management of CXL memory devices as described here. * CONFIG_CXL_PORT * CONFIG_CXL_REGION + +CCI access via MCTP over I2C +---------------------------- + +In order to make use of this device, an I2C controller that supports MCTP +is required. The aspeed-i2c controller is an example of such a controller. + +Both CXL switches and CXL Type 3 devices support configuration via +MCTP access to Component Command Interfaces (CCIs) on the devices. + +Example configuration: + + -device cxl-upstream,port=33,bus=root_port0,id=us0,multifunction=on,addr=0.0,sn=12345678 \ + -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ + -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ + -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ + -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem1,id=cxl-pmem0,lsa=cxl-lsa1,sn=3 \ + -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem2,id=cxl-pmem1,lsa=cxl-lsa2,sn=4 \ + -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem3,id=cxl-pmem2,lsa=cxl-lsa3,sn=5 \ + -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \ + -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=5,target=cxl-pmem0 \ + -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-pmem1 \ + -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=7,target=cxl-pmem2 + +Guest OS communication with the MCTP CCI can then be established using standard +MCTP configuration tools. + References ----------
Signature · 1 line
2.48.1
RFC PATCH 5/5 usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.
In reply to Jonathan CameronGiven the challenges in testing systems with the previously posted i2c based mctp support for x86 / arm64 machines (as only emulated controller that is suitably capable is the aspeed one) providing a USB interface greatly simplifies things.
This is a PoC of such an interface.
RFC reasons:
- Not all error conditions are fully checked / handled.
- MTU in the 'to host' direction is ignored. Seems the Linux stack doesn't care about this today but that should be brought into compliance with the specification anyway.
--- include/hw/usb.h | 1 + include/net/mctp.h | 77 +++++- hw/i2c/mctp.c | 17 -- hw/usb/dev-mctp.c | 639 +++++++++++++++++++++++++++++++++++++++++++++ hw/usb/Kconfig | 5 + hw/usb/meson.build | 1 + 6 files changed, 717 insertions(+), 23 deletions(-)
include/hw/usb.h +1 -0
@@ -81,6 +81,7 @@#define USB_CLASS_CDC_DATA 0x0a#define USB_CLASS_CSCID 0x0b#define USB_CLASS_CONTENT_SEC 0x0d +#define USB_CLASS_MCTP 0x14#define USB_CLASS_APP_SPEC 0xfe#define USB_CLASS_VENDOR_SPEC 0xff
include/net/mctp.h +71 -6
103 lines not on this page · raw message
hw/i2c/mctp.c +0 -17
@@ -33,23 +33,6 @@ typedef struct MCTPI2CPacket {#define i2c_mctp_payload_offset offsetof(MCTPI2CPacket, mctp.payload)#define i2c_mctp_payload (buf) (buf + i2c_mctp_payload_offset) -/* DSP0236 1.3.0, Figure 20 */ -typedef struct MCTPControlMessage { -#define MCTP_MESSAGE_TYPE_CONTROL 0x0 -uint8_t type; -#define MCTP_CONTROL_FLAGS_RQ (1 << 7) -#define MCTP_CONTROL_FLAGS_D (1 << 6) -uint8_t flags; -uint8_t command_code; -uint8_t data[]; -}MCTPControlMessage ; - -enum MCTPControlCommandCodes { -MCTP_CONTROL_SET_EID =0x01 , -MCTP_CONTROL_GET_EID =0x02 , -MCTP_CONTROL_GET_VERSION =0x04 , -MCTP_CONTROL_GET_MESSAGE_TYPE_SUPPORT =0x05 , -};#define MCTP_CONTROL_ERROR_UNSUPPORTED_CMD 0x5
hw/usb/dev-mctp.c new file +639 -0
639 lines not on this page · raw message
hw/usb/Kconfig +5 -0
@@ -105,6 +105,11 @@ config USB_SERIAL default y depends on USB +config USB_MCTP + bool + default y if CXL + depends on USB + config USB_NETWORK bool default y
hw/usb/meson.build +1 -0
@@ -43,6 +43,7 @@ system_ss.add(when: 'CONFIG_USB_STORAGE_UAS', if_true: files('dev-uas.c')) system_ss.add(when: 'CONFIG_USB_AUDIO', if_true: files('dev-audio.c')) system_ss.add(when: 'CONFIG_USB_SERIAL', if_true: files('dev-serial.c')) system_ss.add(when: 'CONFIG_USB_NETWORK', if_true: files('dev-network.c')) +system_ss.add(when: 'CONFIG_USB_MCTP', if_true: files('dev-mctp.c')) if host_os != 'windows' system_ss.add(when: 'CONFIG_USB_STORAGE_MTP', if_true: files('dev-mtp.c')) endif
Signature · 1 line
2.48.1
Quoted 58 lines · Jonathan Cameron
This posting is primarily about sharing the USB device emulation to get some early feedback.
RFC reasons:
- Known 'inaccuracies' in emulation (not obeying MTU in the to host direction for example)./
- Not sure what to do wrt to Klaus' I2C MCTP support given that has been stalled for some time. For now only the headers are really shared between the two implementations.
- This is more of an FYI / request for testing than a formal suggestion that this might be ready for upstream.Why add a CXL FM-API over MCTP over USB device?
- Can be emulated on pretty much any host system as USB is discoverable and expandable. If you want a giggle, see the hacks on i386/pc and arm/virt on we've been using until now given only I2C controller that works is the aspeed one. e.g. https://gitlab.com/jic23/qemu/-/commit/134c2e3952b
- Being able to talk to both the fabric management out of band interfaces and the in band devices on the same host makes testing much simpler.Background:
Back in 2022 I posted some support for controlling the CXL fabric via the spec defined out of band interfaces (CXL Fabric Management API - FM-API) over MCTP on I2C
https://lore.kernel.org/qemu-devel/[email protected]/
I reworked that on top of the NVME-MI work from Klauss.To that end I hacked the aspeed-i2c controller onto both i386/pc and arm/virt and posted kernel patches to enabled ACPI support for that device (more or less). It worked and has been useful in the meantime, but adding that i2c controller to those boards was obviously not going to be upstreamable - and to build reliable tests against it I don't want to carry this out of tree for ever. I messed around with a PCI hosted aspeed controller and might come back to that at some point.
In the meantime, DMTF published a transport binding for MCTP over USB
https://www.dmtf.org/sites/default/files/standards/documents/DSP0283_1.0.1.pdfKernel support duly followed early this year:
drivers/net/mctp/mctp-usb.c
https://codeconstruct.com.au/docs/mctp-over-usb/Given the ease of adding a suitable USB controller on a PCI bus, emulating a suitable endpoint provides what I think is an upstreamable solution.
To use this:
-device usb-ehci,id=ehci
-device usb-cxl-mctp,bus=ehci.0,id=fred,target=us0where target is either a CXL switch upstream port, or a type 3 device.
Then install the mctp userspace tools in your guest and configure it with
mctp addr add 8 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb0 up
Quoted 9 lines
systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0I've been testing the CXL commands with
https://gitlab.com/jic23/cxl-fmapi-tests
(mostly because I still had them in my image from the i2c work) but libcxlmi is probably a better bet
I tested this patchset using libcxlmi with the following configuration:
- kernel version: https://github.com/weiny2/linux-kernel/tree/dcd-v6-2025-04-13
- QEMU: upstream ToT with MCTP USB CXL and FMAPI DCD support:
- MCTP USB CXL (https://lore.kernel.org/linux-cxl/[email protected]/T/#m21b9e0dfc689cb1890bb4d961710c23379e04902)
- note: I modified the condition on line hw/usb/dev-mctp.c:509 to allow the FMAPI DCD Management command set (0x56) to be processed
- FMAPI DCD Management patches (https://lore.kernel.org/linux-cxl/[email protected]/)
- Topology (1 DCD attached to downstream port of CXL Switch)
'-device usb-ehci,id=ehci \
-object memory-backend-file,id=cxl-mem1,mem-path=/tmp/t3_cxl1.raw,size=4G \
-object memory-backend-file,id=cxl-lsa1,mem-path=/tmp/t3_lsa1.raw,size=1M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
-device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
-device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
-device cxl-type3,bus=swport0,volatile-dc-memdev=cxl-mem1,id=cxl-dcd0,lsa=cxl-lsa1,num-dc-regions=2,sn=99 \
-device usb-cxl-mctp,bus=ehci.0,id=usb0,target=us0 \
-device usb-cxl-mctp,bus=ehci.0,id=usb1,target=cxl-dcd0\
-machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k'After creating the VM with the above topology, to configure the 2 MCTP EPs (mctpusb0 and mctpusb1):
mctp link set mctpusb0 up
mctp addr add 50 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb1 up
mctp addr add 51 dev mctpusb1
mctp link set mctpusb0 net 12
systemctl stop mctpd.service
systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0 busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb1 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0
To test this patchset, I used this program from libcxlmi:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/cxl-mctp.c
To summarize, this program scans the D-bus for MCTP endpoints or takes the NID:EID of the endpoint you want to open as arguments, then sends a variety of commands (identify, supported logs, etc.) through the endpoint(s) and prints the results, which I verified. Both methods of opening endpoints are successful and shows the expected output.
To ensure this works end-to-end with an FM workflow I also tested with this program, which allows users to interactively send the FMAPI Add/Release commands from the command line:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/fmapi-mctp.c
First I created a DC Region using this ndctl command:
cxl create-region -m mem0 -d decoder0.0 -s 1G -t dynamic_ram_a
Then I ran the program to add an extent of 512MB and was able to successfully create a DAX Device using the following commands:
daxctl create-device -r region0
daxctl reconfigure-device dax0.1 -m system-ram
The output of 'lsmem' shows 512M added:
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online yes 0-15 0x0000000100000000-0x000000027fffffff 6G online yes 32-79 0x0000001290000000-0x00000012afffffff 512M online yes 594-597
Quoted 45 lines
https://github.com/computexpresslink/libcxlmi
I'll post a tree with this on at gitlab.com/jic23/qemu shortly (cxl-<latest date>).
Jonathan Cameron (3):
hw/cxl/i2c_mctp_cxl: Initial device emulation
docs: cxl: Add example commandline for MCTP CXL CCIs usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.Klaus Jensen (2):
hw/i2c: add smbus pec utility function
hw/i2c: add mctp coreMAINTAINERS | 7 + docs/system/devices/cxl.rst | 27 + include/hw/cxl/cxl_device.h | 8 + include/hw/i2c/mctp.h | 125 +++++ include/hw/i2c/smbus_master.h | 2 + include/hw/pci-bridge/cxl_upstream_port.h | 1 + include/hw/usb.h | 1 + include/net/mctp.h | 100 ++++ hw/cxl/cxl-mailbox-utils.c | 49 ++ hw/cxl/i2c_mctp_cxl.c | 289 ++++++++++ hw/i2c/mctp.c | 414 ++++++++++++++ hw/i2c/smbus_master.c | 26 + hw/usb/dev-mctp.c | 639 ++++++++++++++++++++++ hw/arm/Kconfig | 1 + hw/cxl/Kconfig | 4 + hw/cxl/meson.build | 4 + hw/i2c/Kconfig | 4 + hw/i2c/meson.build | 1 + hw/i2c/trace-events | 14 + hw/usb/Kconfig | 5 + hw/usb/meson.build | 1 + 21 files changed, 1722 insertions(+) create mode 100644 include/hw/i2c/mctp.h create mode 100644 include/net/mctp.h create mode 100644 hw/cxl/i2c_mctp_cxl.c create mode 100644 hw/i2c/mctp.c create mode 100644 hw/usb/dev-mctp.cSignature · 1 line
2.48.1