Skip to contentSkip to conversation

Question about NVMe MCTP in dbus-sensors

DISCUSSION Heyi Guo 3 participants 8 messages Aug 6, 2021 – Aug 9, 2021

lore
Share & tools
lore

Heyi Guo

author
#0 lore raw

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus. However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things? Will there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?

We also found another implementation from Intel: https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp. In my opinion it seems to be a more mature solution for MCTP and higher PCMI management models. Is there any plan to accept such implementations into the mainline?

Thanks,

Heyi

#1 lore raw

Hi Heyi,

Quoted 5 lines

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus. However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table.

So we're working on this at the moment.

The new design is here:

https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

The MCTP core patches have been accepted to the upstream kernel, and we're finishing up the i2c/SMBus binding driver at the moment:

https://github.com/CodeConstruct/linux/tree/dev/mctp-i2c-mux

The prototype userspace support code is here:

https://github.com/CodeConstruct/mctp

- which includes the control protocol implementation (mctpd) for endpoint discovery, routing table updates, etc.

The switch over from the demux-based approach to the kernel sockets interface should be fairly straightforward, but Intel are working on a library to abstract that.

Cheers,

Jeremy

#2 lore raw
In reply to Heyi Guo

Hello Heyi!

Quoted 5 lines · Heyi Guo

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.

To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.

For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.

As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:

https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665

Quoted 3 lines

However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?

It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.

Quoted 3 lines

Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?

No.

Quoted 5 lines

We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.

This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:

https://lore.kernel.org/netdev/[email protected]/

This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].

[1] https://github.com/CodeConstruct/mctp

You can read more about the application of the socket syscalls here:

https://lore.kernel.org/netdev/[email protected]/

and here:

https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

Once we have AF_MCTP in the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.

Hope that helps!

Andrew

author
#3 lore raw

Hi Andrew, Jeremy,

Appreciate your helpful information. Let me take some time to look into all this document and patches :)

BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?

Thanks,

Heyi

Quoted 68 lines · Andrew Jeffery

Hello Heyi!

Quoted 5 lines · Heyi Guo

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.

To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.

For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.

As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:

https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665

Quoted 3 lines

However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?

It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.

Quoted 3 lines

Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?

No.

Quoted 4 lines

We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.

This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:

https://lore.kernel.org/netdev/[email protected]/

This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].

[1] https://github.com/CodeConstruct/mctp

You can read more about the application of the socket syscalls here:

https://lore.kernel.org/netdev/[email protected]/

and here:

https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

Once we have AF_MCTP in the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.

Hope that helps!

Andrew

#4 lore raw

Hi Heyi,

Quoted 3 lines

BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?

We're mostly working on the generic infrastructure components at the moment; those are based on their relevant upstream projects rather than as parts of OpenBMC specifically. However, the upstream work is going well, and the OpenBMC integration shouldn't be too difficult there; we'll get that done as necessary.

As for interfaces: I currently have i2c and serial transports working, but have no specific plans for PCIe - no hardware to test on. If this is something you'd be interested in working on yourself, I can provide some pointers.

Regards,

Jeremy

author
#5 lore raw
Quoted 11 lines · Jeremy Kerr

Hi Heyi,

Quoted 3 lines

BTW, when will MCTP be ready in openbmc mainline, including SMBus binding and PCIe binding? Do you have any plan in the future openbmc release?

We're mostly working on the generic infrastructure components at the moment; those are based on their relevant upstream projects rather than as parts of OpenBMC specifically. However, the upstream work is going well, and the OpenBMC integration shouldn't be too difficult there; we'll get that done as necessary.

Quoted 4 lines

As for interfaces: I currently have i2c and serial transports working, but have no specific plans for PCIe - no hardware to test on. If this is something you'd be interested in working on yourself, I can provide some pointers.

Yes, we are both interested in SMBus binding and PCIe VDM binding, and we have supporting hardware platforms (servers with Aspeed 2600 as BMC).

It's my pleasure to make contribution to the community :)

Thanks,

Heyi

Quoted 5 lines

Regards,

Jeremy

author
#6 lore raw
In reply to Andrew Jeffery

Hi Andrew,

Quoted 11 lines · Andrew Jeffery

Hello Heyi!

Quoted 5 lines · Heyi Guo

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.

To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.

Could you point out where I can find these forks, including libmctp and kernel? So that we can do some initial test with the current implementation of NVMeSensor in dbus-sensors.

Thanks,

Heyi

Quoted 57 lines

For the SMBus binding to be merged in upstream libmctp I'd like to see the necessary kernel interfaces merged into the upstream kernel tree, but I don't expect that's going to happen any time soon. More on why below.

As an aside you may be interested in this patch which allows nvmesensor to use the basic management command to fetch temperature data:

https://gerrit.openbmc-project.xyz/c/openbmc/dbus-sensors/+/43665

Quoted 3 lines

However I don't find the code or component who is responsible as a bus owner, to discover endpoints, manager EID and update routing table. Isn't necessary for a central component to do such things?

It's not strictly necessary in that it's valid for systems to be completely defined in terms of static endpoints. Doing so isn't covered by the MCTP spec, and it's also pretty limiting, but it is enough in some circumstances.

Quoted 3 lines

Will
there be access conflict if non-NVMe devices (MCTP capable) are also connected to the same SMBus?

No.

Quoted 4 lines

We also found another implementation from Intel:
https://github.com/Intel-BMC/pmci. It implements mctpd to provide MCTP message transfer interfaces on D-Bus, while PLDM, NVME-MI and others can rely on the D-Bus interfaces instead of libmctp.

This is a direction Intel chose to go however it's not the direction that upstream OpenBMC will be using. The use of a pure userspace solution such as libmctp went far enough to expose the need for a kernel-based solution. We will shortly have that, thanks to the efforts of Jeremy and Matt at Code Construct:

https://lore.kernel.org/netdev/[email protected]/

This series is now in net-next, and Matt and Jeremy have also been developing the in-kernel hardware binding support and necessary userspace components[1].

[1] https://github.com/CodeConstruct/mctp

You can read more about the application of the socket syscalls here:

https://lore.kernel.org/netdev/[email protected]/

and here:

https://github.com/openbmc/docs/blob/master/designs/mctp/mctp-kernel.md

Once we have AF_MCTP in the OpenBMC kernel tree with some binding implementations we'll be switching the userspace applications over to use it.

Hope that helps!

Andrew

#7 lore raw
Quoted 12 lines · Heyi Guo

Hi Andrew,

Quoted 8 lines · Andrew Jeffery

Hello Heyi!

Quoted 5 lines · Heyi Guo

Hi,

We can see that NVMe sensors in dbus-sensors relies on MCTP to get hardware information. It is using libmctp interfaces to initialize MCTP and SMBus.

Quoted 7 lines
Quoted 3 lines

To be clear, it's using a fork of libmctp that implements an SMBus binding via a fork of the kernel that exposes a I2C API that isn't upstream.

Could you point out where I can find these forks, including libmctp and kernel? So that we can do some initial test with the current implementation of NVMeSensor in dbus-sensors.

The relevant forks are under the Intel-BMC Github org:

https://github.com/Intel-BMC/

Hopefully the Intel folks on Cc can give you pointers on how to get it going.

Cheers,

Andrew