Question about NVMe MCTP in dbus-sensors
Hi,
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus. However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things? Will there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?
We also found another implementation from Intel: https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp. In my opinion it seems to be a more mature solution for MCTP and higher PCMI management models. Is there any plan to accept such implementations into the mainline?
Thanks,
Heyi
Hi Heyi,
Quoted 5 lines
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus. However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table.
So we're working on this at the moment.
The new design is here:
https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
The MCTP core patches have been accepted to the upstream kernel, and we're finishing up the i2c/SMBus binding driver at the moment:
https://github.com/CodeConstruct/linux/tree/dev/mctp-i2c-mux
The prototype userspace support code is here:
https://github.com/CodeConstruct/mctp
- which includes the control protocol implementation (mctpd) for endpoint discovery, routing table updates, etc.
The switch over from the demux-based approach to the kernel sockets interface should be fairly straightforward, but Intel are working on a library to abstract that.
Cheers,
Jeremy
Hello Heyi!
Quoted 5 lines · Heyi Guo
Hi,
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.
To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.
For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.
As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:
https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665
Quoted 3 lines
However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?
It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.
Quoted 3 lines
Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?
No.
Quoted 5 lines
We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.
This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:
https://lore.kernel.org/netdev/[email protected]/
This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].
[1] https://github.com/CodeConstruct/mctp
You can read more about the application of the socket syscalls here:
https://lore.kernel.org/netdev/[email protected]/
and here:
https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
Once we have AF_MCTP in the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.
Hope that helps!
Andrew
Hi Andrew, Jeremy,
Appreciate your helpful information. Let me take some time to look into all this document and patches :)
BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?
Thanks,
Heyi
Quoted 68 lines · Andrew Jeffery
Hello Heyi!
Quoted 5 lines · Heyi Guo
Hi,
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.
To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.
For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.
As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:
https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665
Quoted 3 lines
However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?
It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.
Quoted 3 lines
Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?No.
Quoted 4 lines
We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:
https://lore.kernel.org/netdev/[email protected]/
This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].
[1] https://github.com/CodeConstruct/mctp
You can read more about the application of the socket syscalls here:
https://lore.kernel.org/netdev/[email protected]/
and here:
https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
Once we have
AF_MCTPin the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.Hope that helps!
Andrew
Hi Heyi,
Quoted 3 lines
BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?
We're mostly working on the generic infrastructure components at the moment; those are based on their relevant upstream projects rather than as parts of OpenBMC specifically. However, the upstream work is going well, and the OpenBMC integration shouldn't be too difficult there; we'll get that done as necessary.
As for interfaces: I currently have i2c and serial transports working, but have no specific plans for PCIe - no hardware to test on. If this is something you'd be interested in working on yourself, I can provide some pointers.
Regards,
Jeremy
Quoted 11 lines · Jeremy Kerr
Hi Heyi,
Quoted 3 lines
BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?
We're mostly working on the generic infrastructure components at the moment; those are based on their relevant upstream projects rather than as parts of OpenBMC specifically. However, the upstream work is going well, and the OpenBMC integration shouldn't be too difficult there; we'll get that done as necessary.
Quoted 4 lines
As for interfaces: I currently have i2c and serial transports working, but have no specific plans for PCIe - no hardware to test on. If this is something you'd be interested in working on yourself, I can provide some pointers.
Yes, we are both interested in SMBus binding and PCIe VDM binding, and we have supporting hardware platforms (servers with Aspeed 2600 as BMC).
It's my pleasure to make contribution to the community :)
Thanks,
Heyi
Quoted 5 lines
Regards,
Jeremy
Hi Andrew,
Quoted 11 lines · Andrew Jeffery
Hello Heyi!
Quoted 5 lines · Heyi Guo
Hi,
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.
To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.
Could you point out where I can find these forks, including libmctp and kernel? So that we can do some initial test with the current implementation of NVMeSensor in dbus-sensors.
Thanks,
Heyi
Quoted 57 lines
For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.
As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:
https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665
Quoted 3 lines
However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?
It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.
Quoted 3 lines
Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?No.
Quoted 4 lines
We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:
https://lore.kernel.org/netdev/[email protected]/
This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].
[1] https://github.com/CodeConstruct/mctp
You can read more about the application of the socket syscalls here:
https://lore.kernel.org/netdev/[email protected]/
and here:
https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md
Once we have
AF_MCTPin the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.Hope that helps!
Andrew
Quoted 12 lines · Heyi Guo
Hi Andrew,
Quoted 8 lines · Andrew Jeffery
Hello Heyi!
Quoted 5 lines · Heyi Guo
Hi,
We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.
Quoted 7 lines
Quoted 3 lines
To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.
Could you point out where I can find these forks, including libmctp and kernel? So that we can do some initial test with the current implementation of NVMeSensor in dbus-sensors.
The relevant forks are under the Intel-BMC Github org:
Hopefully the Intel folks on Cc can give you pointers on how to get it going.
Cheers,
Andrew