initiator depth and responder resources in a connect request.

Hey Sean,

Dumb question: What exactly do the rdma_conn_param fields initiator_depth and responder_resources mean in an incoming connection request? Does it mean the peer's values for these two resources, or is responder_resources set to match the peer's initiator_depth and initiator_depth set to match the peer's responder_resources? IE Are they swapped so the app can turn around and use them in the rdma_accept()?

(hope that question makes sense)

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The values are swapped. See the man page for rdma_get_cm_event and let me know if it's not clear. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks. Yes. That man page is clear.

I think the rdma_accept() page has a bug though:

----
responder_resources
The maximum number of outstanding RDMA read and atomic opera- tions that the local side will accept from the remote side. Applies only to RDMA_PS_TCP. This value must be less than or equal to the local RDMA device attribute max_qp_rd_atom and the responder_resources value reported in the connect request event.
----

"and the responder_resources value" should be "and greater than or equal to the responder_resources value".

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

thanks - I added the following change to the man page:

-max_qp_rd_atom and the responder_resources value reported in the connect -request event.
+max_qp_rd_atom, but preferably greater than or equal to the responder_resources +value reported in the connect request event.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes - it's fatal. With rdma_accept, the user _can_ specify a lower number of responder_resources than what the client side requested in their initiator_depth. The client must then determine if the responder_resources that were offered is sufficient (at least with IB). This is why I added 'preferably' to the documentation.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

When you say 'client must then determine', do you mean the client application? So the IB CM really just passes along these values and doesn't really enforce any semantics?

The reason I'm asking is we're reviewing the IETF ID that extends MPA connection setup to allow negotiating ORD/IRD. And I think we should not diverge from the way its done in IB. IE: the RDMACM semantics should be the same IMO.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes and yes. (The ib cm is probably a bit brain dead on this, but the librdmacm does check for this error.) From the IB spec (sections intentionally reversed) see the two areas marked *****:

12.7.30 INITIATOR DEPTH
The maximum number of outstanding RDMA Read/Atomic operations the sender will have to the remote QP/EEC. This value may be zero. The maximum number that the HCA can support for a QP/EEC can be determined using the Query HCA verb. See section 11.2.1.2 Query HCA. InfiniBandTM Architecture Release 1.2.1 Communication Management November 2007

*****The recipient of the REQ message should try to choose a number of local Responder Resources that is greater than or equal to the Initiator Depth in the REQ message. If it is unwilling or unable to do so, it may send a REP message containing fewer Responder Resources than the Initiator Depth in the REQ message.*****

12.7.29 RESPONDER RESOURCES
The maximum number of outstanding RDMA Read/Atomic operations the sender will support from the remote QP/EEC. This value may be zero. The maximum number that the HCA can support for a QP/EEC can be determined using the Query HCA verb. See section 11.2.1.2 Query HCA.

The recipient of the REQ message shall choose a local Initiator Depth that does not exceed the Responder Resources offered in the REQ. If the recipient of the REQ message is unwilling or unable to do so, it shall send a REJ message to discontinue the connection establishment.

*****The recipient of the REP message shall decide whether the Responder Resources offered in the REP are sufficient for the Initiator Depth the recipient of the REP wishes to use.***** If not, it shall send a REJ message to discontinue the connection establishment.

What does iwarp do now?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The current IETF specs leave it up to the applications/ULPs to deal with it. Neither ORD or IRD are exchanged as part of MPA or other iWARP protocol negotiations. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Not sure how this is presented by RDMA CM, but the way it works on the wire in IB is..
- Client sends "client's max outgoing RR usage" and "client's max incoming RR available" [which is min(ULP request,HCA limit)]
- Server computes:
server's max outgoing RR usage = min(client's max
incoming RR available,ULP limit,HCA limit)
server's max incoming RR available = min(client's max outoing RR
usage,ULP limit,HCA limit)
and sends it back.
- Client should sanity check:
server's max outgoing RR usage <= client's max incoming RR available server's max incoming RR available >= client's max outoing RR usage
And the ULP should confirm it can function within the RR limit.
- The four values are encoded into two pairs in the CM messages, the client pair is in the REQ and the server pair is in the REP. Both sides end up learning all four values, but only the server's values are used.

Basically, the equations:
server max outgoing RR usage = client max incoming RR available =

     min(server ULP, client ULP, server HCA, client HCA);
 client max outgoing RR usage = server max incoming RR available =
     min(server ULP, client ULP, server HCA, client HCA);

are being computed. Each side's ULP should have the opportunity to specify the maximum number of RRs it will consume at once when posting send WR's, to avoid consuming global RRs in the HCA.

FWIW, I'm not sure this actually works in Linux, last time I looked it only worked if all values were the same, which they generally are since nearly everything seems wired to 4.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html