Skip to contentSkip to conversation

MCTP Null EID - Physical addressing support - Binding private in API

DISCUSSION Thomaiyar, Richard Marian 2 participants 7 messages Jun 1, 2020 – Jul 20, 2020

lore
Share & tools
lore
#0 lore raw

All,

 MCTP specification doesn’t restrict communication without EID assignment. i.e. MCTP devices can communicate with each other, even without EID’s (Spec limits bridging MCTP commands only – Sec 8.2 Special Endpoints in DSP0236). The scenario can happen in following cases

1. MCTP Bus owner available only in Main power and not on stand-by. In this case, devices are required to communicate using NULL EID, till main power is applied.

2. Till the EID’s are assigned by the Bus Owner, devices can communicate with NULL EID(Source & Dest). Even Bus owner, when it sends out SetEID or query GetEID, it must send with NULL EID (for Dest – Sec 12.3 Set Endpoint ID, 8.17.6 – Reclaiming EID’s from hot-plug devices in DSP0236)

Similarly, as EndPoint device, need physical address of the bus owner, so that when SetEID is received from secondary/backup bus owner, we will know what to do? (Note: Secondary / backup bus owner may send with same bus owner EID, but it’s physical address will be different, and device can request for force Set EID).

3. Devices without bus owner in the network trying to communicate in peer-to-peer – single / simple network.

There are also scenario’s where device’s issue Get EID command to the Bus owner to do a discovery or initiate discovery based on the need. In this case, there can be 2 / 3 devices which will issue this GetEID, and physical addressing is the way to differentiate the same.

 This requires to pass physical addressing information to the MCTP Control command layer or to the upper layer from libmctp. Right now, all this information is dropped as part of binding to MCTP Assembly / disassembly before mctp_bus_rx. Hence, we need to have binding specific private structure in the mctp_pktbuff, which needs to tracked even along with MCTP Message buffer, apart from Source EID. Please comment / put your thoughts on the same.

Regards,

Richard

#1 lore raw

Hi Richard,

Quoted 7 lines · Richard Marian

All,

MCTP specification doesn’t restrict communication without EID
assignment. i.e. MCTP devices can communicate with each other, even without EID’s (Spec limits bridging MCTP commands only – Sec 8.2 Special Endpoints in DSP0236). The scenario can happen in following cases

Before we jump ahead, as far as I can see this is only a problem in libmctp for local bus communications involving at least one device requiring dynamic address allocation.

Obviously that doesn't mean it's not a problem, just that there exists a class of configuration for which this is not an issue. Where it's not an issue we can follow the usual message routing path. My goal is that the flow for sending messages is the same regardless of whether we're communicating with static or dynamic devices irrespective of whether the dynamic device has formally been assigned an address (i.e. we follow the same routing path in both cases).

For static allocation, each endpoint in the network must be aware of the allocated EIDs a-priori, and how this is managed is out of scope of the MCTP base specification. If the EID assignment is known statically then there's no need to worry about the special case of EID 0 as alluded to above.

Finally, conceptually, we can split the process of sending a command into three phases for the purpose of discussing addresses:

1. Command Generation: Performed at the application layer
2. Message Routing: Performed in libmctp core
3. Message Dispatch: Performed in the libmctp binding implementation

Keep this separation in mind. My goal is that we maintain abstractions and do not propagate physical layer information beyond the libmctp core. I'll try to build the case for EID 0 being a corner case, and that we should treat it that way in the libmctp public APIs as well (i.e. not let it impact the usual interfaces for sending messages).

Quoted 4 lines

1. MCTP Bus owner available only in Main power and not on stand-by. In this case, devices are required to communicate using NULL EID, till main power is applied.

Again, only for communication involving a device that requires dynamic EID allocation.

Quoted 3 lines

2. Till the EID’s are assigned by the Bus Owner, devices can communicate with NULL EID(Source & Dest).

Agreed, except for in the static case there's no need to do this because we know the EIDs.

Quoted 4 lines

Even Bus owner, when it sends
out SetEID or query GetEID, it must send with NULL EID (for Dest – Sec 12.3 Set Endpoint ID, 8.17.6 – Reclaiming EID’s from hot-plug devices in DSP0236)

Here's an extract from 8.17.6:

Quoted 6 lines

Attempting to access each endpoint can be accomplished by issuing the Get Endpoint ID command to the physical address of each device and comparing the returned result to the existing entry in the routing table. If there is no response to the command, or if there is a mismatch with the existing routing information, the entry should be cleared and the corresponding EID or EID range should be returned to the "pool" for re-assignment.

Note that it says "can be accomplished" and "or if there is a mismatch". So "must send with NULL EID" is too strong here - if we know an EID we can use it.

This is where breaking down the process of sending a command into the three phases becomes useful: The application uses the EID to specify the device it wants to talk to in the Command Generation phase, while the Message Routing phase performs the mapping of EID to physical device address. If necessary, the EID in the message can be substituted with 0 in the Message Routing phase.

Quoted 4 lines

Similarly, as EndPoint device, need physical address of the bus owner, so that when SetEID is received from secondary/backup bus owner, we will know what to do?

This isn't dependent on the physical address - the specification says that if a device receives a subsequent SetEID to the first it must respond indicating the EID it has already been allocated. This has no tie to the physical address of the device sending the SetEID command. Further, we know both the EID and the physical address of the device that propagated the SetEID from the received packet, so the local routing table should simply be updated with the information captured.

Quoted 6 lines

(Note: Secondary / backup bus owner may send with
same bus owner EID, but it’s physical address will be different, and device can request for force Set EID).

3. Devices without bus owner in the network trying to communicate in peer-to-peer – single / simple network.

Again, if the EIDs have been statically assigned there's no reason not to use them. So again this is the case where communication involves at least one dynamic device.

Quoted 5 lines

There are also scenario’s where device’s issue Get EID command to the Bus owner to do a discovery or initiate discovery based on the need. In this case, there can be 2 / 3 devices which will issue this GetEID, and physical addressing is the way to differentiate the same.

Yep. And this is covered by the proposal I previously outlined to Sumanth (the sequence outlined near the bottom of the reply):

https://lists.ozlabs.org/pipermail/openbmc/2020-April/021293.html

The summary is that for sending messages we populate the route table with a provisional EID assignment before sending the message.

What we lack at the moment in libmctp is proper route table handling and an API to interact (add/remove/inspect) with route entries. However, continuing:

The fact that the EID is marked provisional in the routing table will cause libmctp core to replace the EID in the message with EID 0 before passing the message and the physical address to the appropriate binding.

Going the other direction, upon receiving a message with EID 0, libmctp core will populate the routing table with a provisional EID, replace the source EID 0 in the MCTP header with the provisional EID, then propagate the message up the stack.

Quoted 3 lines

This requires to pass physical addressing information to the MCTP
Control command layer or to the upper layer from libmctp.

I think I've demonstrated above that this might not be not necessary. Please poke holes in what I've proposed!

Andrew

#2 lore raw

Hi Andrew,

Clarified inline, let me know your further comments / questions.

Regards,

Richard

Quoted 5 lines · Andrew Jeffery

Hi Richard,

Quoted 2 lines · Richard Marian

All,

Quoted 8 lines
Quoted 5 lines

MCTP specification doesn’t restrict communication without EID
assignment. i.e. MCTP devices can communicate with each other, even without EID’s (Spec limits bridging MCTP commands only – Sec 8.2 Special Endpoints in DSP0236). The scenario can happen in following cases

Before we jump ahead, as far as I can see this is only a problem in libmctp for local bus communications involving at least one device requiring dynamic address allocation.

[Richard]: Intentionally didn't talk about static EID - networks as this problems won't be there. If you meant to say that libmctp is not designed to handle non-static network case, let me know. I brought this topic to cover it in libmctp.

Quoted 12 lines

Obviously that doesn't mean it's not a problem, just that there exists a class of configuration for which this is not an issue. Where it's not an issue we can follow the usual message routing path. My goal is that the flow for sending messages is the same regardless of whether we're communicating with static or dynamic devices irrespective of whether the dynamic device has formally been assigned an address (i.e. we follow the same routing path in both cases).

For static allocation, each endpoint in the network must be aware of the allocated EIDs a-priori, and how this is managed is out of scope of the MCTP base specification. If the EID assignment is known statically then there's no need to worry about the special case of EID 0 as alluded to above.

Quoted 6 lines

Finally, conceptually, we can split the process of sending a command into three phases for the purpose of discussing addresses:

1. Command Generation: Performed at the application layer
2. Message Routing: Performed in libmctp core
3. Message Dispatch: Performed in the libmctp binding implementation

[Richard]: Yes. We are on same page here.

Quoted 5 lines

Keep this separation in mind. My goal is that we maintain abstractions and do not propagate physical layer information beyond the libmctp core. I'll try to build the case for EID 0 being a corner case, and that we should treat it that way in the libmctp public APIs as well (i.e. not let it impact the usual interfaces for sending messages).

[Richard]: Yes, we are on the same goal, but finding discrepancies in achieving the said cases, and hence this discussion.

Quoted 5 lines
Quoted 3 lines

1. MCTP Bus owner available only in Main power and not on stand-by. In this case, devices are required to communicate using NULL EID, till main power is applied.

Again, only for communication involving a device that requires dynamic EID allocation.

[Richard]: Yes.

Quoted 5 lines
Quoted 3 lines

2. Till the EID’s are assigned by the Bus Owner, devices can communicate with NULL EID(Source & Dest).

Agreed, except for in the static case there's no need to do this because we know the EIDs.

[Richard]: Now for dynamic only EID case, how this will happen is the question here.

Quoted 17 lines
Quoted 5 lines

Even Bus owner, when it sends
out SetEID or query GetEID, it must send with NULL EID (for Dest – Sec 12.3 Set Endpoint ID, 8.17.6 – Reclaiming EID’s from hot-plug devices in DSP0236)

Here's an extract from 8.17.6:

Quoted 6 lines

Attempting to access each endpoint can be accomplished by issuing the Get Endpoint ID command to the physical address of each device and comparing the returned result to the existing entry in the routing table. If there is no response to the command, or if there is a mismatch with the existing routing information, the entry should be cleared and the corresponding EID or EID range should be returned to the "pool" for re-assignment.

Note that it says "can be accomplished" and "or if there is a mismatch". So "must send with NULL EID" is too strong here - if we know an EID we can use it.

Quoted 5 lines

This is where breaking down the process of sending a command into the three phases becomes useful: The application uses the EID to specify the device it wants to talk to in the Command Generation phase, while the Message Routing phase performs the mapping of EID to physical device address. If necessary, the EID in the message can be substituted with 0 in the Message Routing phase.

[Richard]: Not really. This section 8.17.6 talks about reclaiming. i.e Basically, in this case, don't send GetEID command based on EID to physical address mapping, instead send GetEID to the needed physical address with Dest EID Null, and cross verify it's EID. This needs to be performed, to figure out whether EID of the device changed due to card replacement or reset etc. If we don't send with Dest EID as 0, then device may drop the packet, if there is mismatch with it's EID.

Quoted 10 lines
Quoted 3 lines

Similarly, as EndPoint device, need physical address of the bus owner, so that when SetEID is received from secondary/backup bus owner, we will know what to do?

This isn't dependent on the physical address - the specification says that if a device receives a subsequent SetEID to the first it must respond indicating the EID it has already been allocated. This has no tie to the physical address of the device sending the SetEID command. Further, we know both the EID and the physical address of the device that propagated the SetEID from the received packet, so the local routing table should simply be updated with the information captured.

[Richard]: Not really, if the same bus owner sends SetEID, we need to accept that, but if the request comes from different one, then we can't accept that one. Assume bus owner crashed / reset and coming back up, it may start with different EID assignment, based on number of devices at that point and devices needs to identify the bus owner with UUID or physical address to uuid mapped.

Quoted 10 lines
Quoted 7 lines

(Note: Secondary / backup bus owner may send with
same bus owner EID, but it’s physical address will be different, and device can request for force Set EID).

3. Devices without bus owner in the network trying to communicate in peer-to-peer – single / simple network.

Again, if the EIDs have been statically assigned there's no reason not to use them. So again this is the case where communication involves at least one dynamic device.

[Richard]: Yes, talking about non-static network model only.

Quoted 12 lines
Quoted 4 lines

There are also scenario’s where device’s issue Get EID command to the Bus owner to do a discovery or initiate discovery based on the need. In this case, there can be 2 / 3 devices which will issue this GetEID, and physical addressing is the way to differentiate the same.

Yep. And this is covered by the proposal I previously outlined to Sumanth (the sequence outlined near the bottom of the reply):

https://lists.ozlabs.org/pipermail/openbmc/2020-April/021293.html

The summary is that for sending messages we populate the route table with a provisional EID assignment before sending the message.

Quoted 6 lines

What we lack at the moment in libmctp is proper route table handling and an API to interact (add/remove/inspect) with route entries. However, continuing:

The fact that the EID is marked provisional in the routing table will cause libmctp core to replace the EID in the message with EID 0 before passing the message and the physical address to the appropriate binding.

[Richard]: This is as per our earlier discussion in the PMCI WG, but the problem with the provisional is we can't manage as per the need. Say, GetEID needs to be sent to a newly discovered device say PA.X (physicalAddress X), then for this we need to form the command with MCTP transport header with Dest EID as EID.0 (can't put a provisional EID here) (If we send a provisional EID, then the device can drop the packet, when the EID doesn't match). i.e. we need a mechanism to differentiate between provisional & real EID and for the provisional EID, Dest EID must be 0, if that's what we required. Currently MCTP Transport header is filled/decoded by core-mctp, hence it must be aware about this. For this we need to introduce a field, which we call here as binding private.

Quoted 5 lines

Going the other direction, upon receiving a message with EID 0, libmctp core will populate the routing table with a provisional EID, replace the source EID 0 in the MCTP header with the provisional EID, then propagate the message up the stack.

[Richard]: This means that Provisional EID is carved out of the EID Pool. With bridging - (Multiple networks), this can't be handled. We will end up with real EID for the same provisional EID, and hence we won't know when we send back data to which device we need to deliver.

Quoted 4 lines
Quoted 2 lines

This requires to pass physical addressing information to the MCTP
Control command layer or to the upper layer from libmctp.

I think I've demonstrated above that this might not be not necessary. Please poke holes in what I've proposed!

[Richard]:

_*For control command handling:*_  As stated, we need a mechanism other than EID (Note: I meant we may need to introduce one more field, to make the difference - if you are saying that as provisional (i.e. not the same mcpt_eid_t then that's what we are saying as binding private).

_*Now for other Message types*_ say PLDM etc. Other than EID, PLDM doesn't use any of the MCTP Transport header, but we still need to expose the transport header in the upper API, as other OEM types may use it or rely on MCTP transport header for informaiton. Now, having said there are cases where devices behind mux or requires special handling, now we need to see how to handle the same in terms of API (which is second part of this discussion).

Quoted 1 line

Andrew

#3 lore raw

Hi Richard,

I've trimmed out all the bits we agree on.

Quoted 11 lines · Richard Marian
Quoted 1 line · Andrew Jeffery

Hi Richard,

Quoted 1 line · Richard Marian
Quoted 1 line

All,

MCTP specification doesn’t restrict communication without EID
assignment. i.e. MCTP devices can communicate with each other, even without EID’s (Spec limits bridging MCTP commands only – Sec 8.2 Special Endpoints in DSP0236). The scenario can happen in following cases

Quoted 8 lines
Quoted 1 line

Before we jump ahead, as far as I can see this is only a problem in

libmctp for
local bus communications involving at least one device requiring dynamic
address allocation. [Richard]: Intentionally didn't talk about static EID - networks as this problems won't be there. If you meant to say that libmctp is not designed to handle non-static network case, let me know. I brought this topic to cover it in libmctp.

Let's not get ahead of ourselves :)

I _intend_ libmctp to handle non-static network cases just fine. We need to do some work though, mainly implementing support for routing.

Quoted 11 lines
Quoted 1 line
Quoted 1 line

1. MCTP Bus owner available only in Main power and not on stand-by. In

this case, devices are required to communicate using NULL EID, till main power is applied.

Quoted 1 line

Again, only for communication involving a device that requires dynamic EID

allocation. [Richard]: Yes.

Quoted 1 line
Quoted 1 line

2. Till the EID’s are assigned by the Bus Owner, devices can

communicate with NULL EID(Source & Dest).

Quoted 1 line

Agreed, except for in the static case there's no need to do this

because we
know the EIDs. [Richard]: Now for dynamic only EID case, how this will happen is the question here.

It is handled by the provisional EID proposal.

Quoted 16 lines
Quoted 1 line
Quoted 1 line

Even Bus owner, when it sends

out SetEID or query GetEID, it must send with NULL EID (for Dest – Sec 12.3 Set Endpoint ID, 8.17.6 – Reclaiming EID’s from hot-plug devices in DSP0236)

Quoted 3 lines

Here's an extract from 8.17.6:

Quoted 1 line

Attempting to access each endpoint can be accomplished by issuing the Get

Endpoint ID command to the physical address of each device and comparing the returned result to the existing entry in the routing table. If there is no response to the command, or if there is a mismatch with the existing routing information, the entry should be cleared and the corresponding EID or EID range should be returned to the "pool" for re-assignment.

Quoted 1 line

Note that it says "can be accomplished" and "or if there is a mismatch". So

"must send with NULL EID" is too strong here - if we know an EID we can use it.

Quoted 14 lines

This is where breaking down the process of sending a command into the three
phases becomes useful: The application uses the EID to specify the device it
wants to talk to in the Command Generation phase, while the Message Routing
phase performs the mapping of EID to physical device address. If necessary, the
EID in the message can be substituted with 0 in the Message Routing phase.
[Richard]: Not really. This section 8.17.6 talks about reclaiming. i.e
Basically, in this case, don't send GetEID command based on EID to physical address mapping, instead send GetEID to the needed physical address with Dest EID Null, and cross verify it's EID.

That's an option but not a necessity as far as I read it. And if it's not a necessity, we can still use our current understanding of the EID to perform the EID to physical address mapping in the routing phase. If we're unsure of the device's assigned EID, we can always mark what know as was its previous address as provisional before sending the GetEID command. The provisional mark will cause the routing layer to replace the message's destination EID with 0 after resolving the physical address.

Quoted 4 lines

This needs to be
performed, to figure out whether EID of the device changed due to card replacement or reset etc. If we don't send with Dest EID as 0, then device may drop the packet, if there is mismatch with it's EID.

This is why in the provisional EID proposal we substitute the supplied EID with 0 in the routing layer.

Quoted 13 lines
Quoted 1 line
Quoted 1 line

Similarly, as EndPoint device, need physical address of the bus owner,

so that when SetEID is received from secondary/backup bus owner, we will know what to do?

Quoted 1 line

This isn't dependent on the physical address - the specification says

that if a
device receives a subsequent SetEID to the first it must respond indicating the
EID it has already been allocated. This has no tie to the physical address of
the device sending the SetEID command. Further, we know both the EID and the
physical address of the device that propagated the SetEID from the received

Quoted 8 lines

packet, so the local routing table should simply be updated with the information
captured. [Richard]: Not really, if the same bus owner sends SetEID, we need to accept that, but if the request comes from different one, then we can't accept that one. Assume bus owner crashed / reset and coming back up, it may start with different EID assignment, based on number of devices at that point and devices needs to identify the bus owner with UUID or physical address to uuid mapped.

This isn't what the spec says as I read it. Can you point to the sections that made you come to this conclusion about the specific device that's acting as bus owner?

Quoted 12 lines
Quoted 2 lines

Going the other direction, upon receiving a message with EID 0,

libmctp core
will populate the routing table with a provisional EID, replace the source EID
0 in the MCTP header with the provisional EID, then propagate the message up
the stack.
[Richard]: This means that Provisional EID is carved out of the EID
Pool. With bridging - (Multiple networks), this can't be handled. We will end up with real EID for the same provisional EID, and hence we won't know when we send back data to which device we need to deliver.

This case can be detected by the routing layer, however I need to think through the impacts some more. I don't think it precludes the provisional EID approach. To get a bit more specific, provisional EIDs are only used above the routing layer where an inbound message contains a 0 source EID, and for the outbound case where the destination EID should be 0. If an inbound message arrives with a non-zero source EID then we know it is a legitimately assigned EID and we can de-conflict the routing table.

I just need to make sure this works for each scenario.

Quoted 11 lines
Quoted 1 line
Quoted 1 line

This requires to pass physical addressing information to the MCTP

Control command layer or to the upper layer from libmctp.

Quoted 1 line

I think I've demonstrated above that this might not be not necessary. Please

poke holes in what I've proposed!
[Richard]:

_*For control command handling:*_ As stated, we need a mechanism other than EID (Note: I meant we may need to introduce one more field, to make the difference - if you are saying that as provisional (i.e. not the same mcpt_eid_t then that's what we are saying as binding private).

Quoted 4 lines

_*Now for other Message types*_ say PLDM etc. Other than EID, PLDM doesn't use any of the MCTP Transport header, but we still need to expose the transport header in the upper API, as other OEM types may use it or rely on MCTP transport header for informaiton.

Can you please provide concrete examples of this being necessary?

Quoted 2 lines

Now, having
said there are cases where devices behind mux

For muxes it should be possible to determine the necessary mux state between the routing and the binding. The mux configuration should be encoded as part of the physical address. This may mean we need binding "decorators" to handle the case.

Quoted 3 lines

or requires special
handling, now we need to see how to handle the same in terms of API (which is second part of this discussion).

Can you please be more specific here? What do you have in mind?

Cheers,

Andrew

#4 lore raw

Thanks Andrew,

inline comments. Captured this as an agenda for our next work group call discussion https://github.com/openbmc/openbmc/wiki/OpenBMC-PMCI-WG

Regards,

Richard

Quoted 22 lines · Andrew Jeffery

Hi Richard,

I've trimmed out all the bits we agree on.

Quoted 9 lines · Richard Marian
Quoted 1 line · Andrew Jeffery

Hi Richard,

Quoted 1 line · Richard Marian
Quoted 1 line

All,

MCTP specification doesn’t restrict communication without EID
assignment. i.e. MCTP devices can communicate with each other, even without EID’s (Spec limits bridging MCTP commands only – Sec 8.2 Special Endpoints in DSP0236). The scenario can happen in following cases

Quoted 8 lines
Quoted 1 line

Before we jump ahead, as far as I can see this is only a problem in

libmctp for
local bus communications involving at least one device requiring dynamic
address allocation. [Richard]: Intentionally didn't talk about static EID - networks as this problems won't be there. If you meant to say that libmctp is not designed to handle non-static network case, let me know. I brought this topic to cover it in libmctp.

Quoted 4 lines

Let's not get ahead of ourselves :)

I _intend_ libmctp to handle non-static network cases just fine. We need to do some work though, mainly implementing support for routing.

[Richard]: we are on the same page then.

Quoted 42 lines
Quoted 11 lines
Quoted 1 line
Quoted 1 line

1. MCTP Bus owner available only in Main power and not on stand-by. In

this case, devices are required to communicate using NULL EID, till main power is applied.

Quoted 1 line

Again, only for communication involving a device that requires dynamic EID

allocation. [Richard]: Yes.

Quoted 1 line
Quoted 1 line

2. Till the EID’s are assigned by the Bus Owner, devices can

communicate with NULL EID(Source & Dest).

Quoted 1 line

Agreed, except for in the static case there's no need to do this

because we
know the EIDs. [Richard]: Now for dynamic only EID case, how this will happen is the question here.

It is handled by the provisional EID proposal.

Quoted 15 lines
Quoted 1 line
Quoted 1 line

Even Bus owner, when it sends

out SetEID or query GetEID, it must send with NULL EID (for Dest – Sec 12.3 Set Endpoint ID, 8.17.6 – Reclaiming EID’s from hot-plug devices in DSP0236)

Quoted 2 lines

Here's an extract from 8.17.6:

Quoted 1 line

Attempting to access each endpoint can be accomplished by issuing the Get

Endpoint ID command to the physical address of each device and comparing the returned result to the existing entry in the routing table. If there is no response to the command, or if there is a mismatch with the existing routing information, the entry should be cleared and the corresponding EID or EID range should be returned to the "pool" for re-assignment.

Quoted 1 line

Note that it says "can be accomplished" and "or if there is a mismatch". So

"must send with NULL EID" is too strong here - if we know an EID we can use it.

Quoted 14 lines

This is where breaking down the process of sending a command into the three
phases becomes useful: The application uses the EID to specify the device it
wants to talk to in the Command Generation phase, while the Message Routing
phase performs the mapping of EID to physical device address. If necessary, the
EID in the message can be substituted with 0 in the Message Routing phase.
[Richard]: Not really. This section 8.17.6 talks about reclaiming. i.e
Basically, in this case, don't send GetEID command based on EID to physical address mapping, instead send GetEID to the needed physical address with Dest EID Null, and cross verify it's EID.

Quoted 7 lines

That's an option but not a necessity as far as I read it. And if it's not a necessity, we can still use our current understanding of the EID to perform the EID to physical address mapping in the routing phase. If we're unsure of the device's assigned EID, we can always mark what know as was its previous address as provisional before sending the GetEID command. The provisional mark will cause the routing layer to replace the message's destination EID with 0 after resolving the physical address.

[Richard]: If you are saying to use different flag / field, apart from mctp_eid_t for it, then we are on the same page here, and can talk about how that must be done. The main agreement we need to make here is whether same mctp_eid_t must be used or not.

Quoted 28 lines
Quoted 4 lines

This needs to be
performed, to figure out whether EID of the device changed due to card replacement or reset etc. If we don't send with Dest EID as 0, then device may drop the packet, if there is mismatch with it's EID.

This is why in the provisional EID proposal we substitute the supplied EID with 0 in the routing layer.

Quoted 13 lines
Quoted 1 line
Quoted 1 line

Similarly, as EndPoint device, need physical address of the bus owner,

so that when SetEID is received from secondary/backup bus owner, we will know what to do?

Quoted 1 line

This isn't dependent on the physical address - the specification says

that if a
device receives a subsequent SetEID to the first it must respond indicating the
EID it has already been allocated. This has no tie to the physical address of
the device sending the SetEID command. Further, we know both the EID and the
physical address of the device that propagated the SetEID from the received

Quoted 8 lines

packet, so the local routing table should simply be updated with the information
captured. [Richard]: Not really, if the same bus owner sends SetEID, we need to accept that, but if the request comes from different one, then we can't accept that one. Assume bus owner crashed / reset and coming back up, it may start with different EID assignment, based on number of devices at that point and devices needs to identify the bus owner with UUID or physical address to uuid mapped.

Quoted 3 lines

This isn't what the spec says as I read it. Can you point to the sections that made you come to this conclusion about the specific device that's acting as bus owner?

[Richard]: yes, in section 12.3 of the DSP0236. Capturing the excerpt below.

The Set Endpoint ID command is also used to provide the Physical Address and EID of the Bus Owner to an Endpoint. An Endpoint that needs to communicate with the Bus Owner may capture the physical address and EID that was used to deliver the Set Endpoint ID message.

Quoted 20 lines
Quoted 12 lines
Quoted 2 lines

Going the other direction, upon receiving a message with EID 0,

libmctp core
will populate the routing table with a provisional EID, replace the source EID
0 in the MCTP header with the provisional EID, then propagate the message up
the stack.
[Richard]: This means that Provisional EID is carved out of the EID
Pool. With bridging - (Multiple networks), this can't be handled. We will end up with real EID for the same provisional EID, and hence we won't know when we send back data to which device we need to deliver.

This case can be detected by the routing layer, however I need to think through the impacts some more. I don't think it precludes the provisional EID approach. To get a bit more specific, provisional EIDs are only used above the routing layer where an inbound message contains a 0 source EID, and for the outbound case where the destination EID should be 0. If an inbound message arrives with a non-zero source EID then we know it is a legitimately assigned EID and we can de-conflict the routing table.

Quoted 1 line

I just need to make sure this works for each scenario.

[Richard]: We can't know which EID can be used as provisional. It may be real EID in a bridged network. Please educate me, if the same can be achieved without having a conflict.

Quoted 12 lines
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines

This requires to pass physical addressing information to the MCTP

Control command layer or to the upper layer from libmctp.

Quoted 1 line

I think I've demonstrated above that this might not be not necessary. Please

poke holes in what I've proposed!
[Richard]:

_*For control command handling:*_ As stated, we need a mechanism other than EID (Note: I meant we may need to introduce one more field, to make the difference - if you are saying that as provisional (i.e. not the same mcpt_eid_t then that's what we are saying as binding private).

Quoted 5 lines
Quoted 4 lines

_*Now for other Message types*_ say PLDM etc. Other than EID, PLDM doesn't use any of the MCTP Transport header, but we still need to expose the transport header in the upper API, as other OEM types may use it or rely on MCTP transport header for informaiton.

Can you please provide concrete examples of this being necessary?

[Richard]: For control commands, already stated the scenarios and excerpt from specification, let me know if you are looking for any other specific concrete examples.

Quoted 16 lines
Quoted 3 lines

Now, having
said there are cases where devices behind mux

For muxes it should be possible to determine the necessary mux state between the routing and the binding. The mux configuration should be encoded as part of the physical address. This may mean we need binding "decorators" to handle the case.

Quoted 3 lines

or requires special
handling, now we need to see how to handle the same in terms of API (which is second part of this discussion).

Can you please be more specific here? What do you have in mind?

Cheers,

Andrew

#5 lore raw
Quoted 4 lines · Richard Marian

Thanks Andrew,

inline comments. Captured this as an agenda for our next work group call discussion https://github.com/openbmc/openbmc/wiki/OpenBMC-PMCI-WG

I noted in another thread that I've pushed patches implementing bridging and routing along with the provisional EID concept to gerrit:

https://gerrit.openbmc-project.xyz/q/topic:%2522routing%2522+(status:open+OR+status:merged)+project:openbmc/libmctp

In the end the implementation uses a "provisional" flag in both the mctp_eid_t type and the route table as there was one race that couldn't be safely eliminated without it.

Quoted 5 lines

[Richard]: We
can't know which EID can be used as provisional. It may be real EID in a bridged network. Please educate me, if the same can be achieved without having a conflict.

Because the conflict shouldn't matter - it can be detected by the binding receiving the message by looking up the source EID in the route table and checking if the result is a provisional EID.

However, the complication occurs when trying to remove the provisional EID from the route table in order to deliver the message received from the endpoint that has been formally allocated the same EID. If there's an outstanding command whose response hasn't yet been sent which will need the provisional EID entry from the table in order for the message to be routed, we can't deliver the message received from the endpoint which is non-provisionally assigned the EID. It's hard to unwind that properly and still account for issues like deadlocks or application crashes which will lead to a denial of service.

So the result is we have a route table with two separate address spaces, one for formally assigned EIDs and another for provisional EIDs. This partition is managed by a flag on the route table entries in the current implementation.

Andrew

#6 lore raw

Andrew,

inline comments,

Regards,

Richard

Quoted 32 lines · Andrew Jeffery
Quoted 4 lines · Richard Marian

Thanks Andrew,

inline comments. Captured this as an agenda for our next work group call discussion https://github.com/openbmc/openbmc/wiki/OpenBMC-PMCI-WG

I noted in another thread that I've pushed patches implementing bridging and routing along with the provisional EID concept to gerrit:

https://gerrit.openbmc-project.xyz/q/topic:%2522routing%2522+(status:open+OR+status:merged)+project:openbmc/libmctp

In the end the implementation uses a "provisional" flag in both the mctp_eid_t type and the route table as there was one race that couldn't be safely eliminated without it.

Quoted 4 lines

[Richard]: We
can't know which EID can be used as provisional. It may be real EID in a bridged network. Please educate me, if the same can be achieved without having a conflict.

Because the conflict shouldn't matter - it can be detected by the binding receiving the message by looking up the source EID in the route table and checking if the result is a provisional EID.

However, the complication occurs when trying to remove the provisional EID from the route table in order to deliver the message received from the endpoint that has been formally allocated the same EID. If there's an outstanding command whose response hasn't yet been sent which will need the provisional EID entry from the table in order for the message to be routed, we can't deliver the message received from the endpoint which is non-provisionally assigned the EID. It's hard to unwind that properly and still account for issues like deadlocks or application crashes which will lead to a denial of service.

Quoted 3 lines

So the result is we have a route table with two separate address spaces, one for formally assigned EIDs and another for provisional EIDs. This partition is managed by a flag on the route table entries in the current implementation.

[Richard]: Reviewed the same and indicating provisional EID using the flag will cause confusion, as application must use this flag. Applications can't just rely on EID 0 is special, instead it must check for the flag.

Won't it be easier to track with EID 0 itself, as all application knows that it is special EID (no separate EID necessary), and applications using special EID, must be aware about which device it is communicating and should be fine.

I also see that with provisional EID, we are building up of another EID concept similar to MCTP specification, and don't find a way to clear it. We are doing on the assumption that provisional EID will be removed once proper EID is assigned, but we will end up issuing Get Endpoint ID to a device with Null EID (just to make sure that the card has not changed the EID when BMC is in reset - This will happen, when Card also went for a reset or swapped and it is using a static EID X, in which case BMC request with already assigned EID Y will be dropped).

Quoted 2 lines

Andrew