Notes on XRC

A natural way to pass the SRQ's QPN is to use the "private_data" of the "rdma_conn_param", however this area is limited to 56 octets and since each QPN occupies 3 octets that leaves us with 18 SRQ. I suggest to increase the size of the private data, or to find a more convenient way to pass the SRQ's QPM, one that doesn't use extra ports or out-of-band communication (socket).

Another issue is of backwards compatibility of applications. Assume an application uses a well known port, and want to allow both new XRC enabled clients and old RC clients. It seems that this is possible using the "rdma_create_ep" API with appropriate "rdma_addrinfo" one that "rdma_getaddrinfo" resolved with QP type "IBV_QPT_XRC_RECV" and one with "IBV_QPT_RC" one the same port (I hope this is allowed). A client on the other side will first try to connect to the XRC and if it fails will fall-back to RC.

However, a server usually uses "rdma_create_id" to associate an event handler with an id and then bind the id to an address, and accept connection requests asynchronously, but with these API, AFAIK there is no way to pass the required hints.

I assume I can use the "private_data" of the "rdma_conn_param" to distinguish between XRC enabled clients and servers and old ones, but it doesn't seems clean, as it is implicit and not explicit.

Quoted 6 lines

A natural way to pass the SRQ's QPN is to use the "private_data" of the "rdma_conn_param", however this area is limited to 56 octets and since each QPN occupies 3 octets that leaves us with 18 SRQ. I suggest to increase the size of the private data, or to find a more convenient way to pass the SRQ's QPM, one that doesn't use extra ports or out-of-band communication (socket).

This is an issue for the IBTA to deal with and requires increasing the size of a MAD.

Quoted 9 lines

Another issue is of backwards compatibility of applications. Assume an application uses a well known port, and want to allow both new XRC enabled clients and old RC clients. It seems that this is possible using the "rdma_create_ep" API with appropriate "rdma_addrinfo" one that "rdma_getaddrinfo" resolved with QP type "IBV_QPT_XRC_RECV" and one with "IBV_QPT_RC" one the same port (I hope this is allowed). A client on the other side will first try to connect to the XRC and if it fails will fall-back to RC.

The intent behind rdma_getaddrinfo is to allow multiple QP types to be returned based on user input.

Quoted 5 lines

However, a server usually uses "rdma_create_id" to associate an event handler with an id and then bind the id to an address, and accept connection requests asynchronously, but with these API, AFAIK there is no way to pass the required hints.

Correct - the older rdma cm APIs are not as flexible. That's why new APIs ended up being added.

Quoted 4 lines

I assume I can use the "private_data" of the "rdma_conn_param" to distinguish between XRC enabled clients and servers and old ones, but it doesn't seems clean, as it is implicit and not explicit.

I don't think I'm fully following your private data idea.

- Sean

This is just a workaround I been considering in case using "hints" and asynchronous connection request can't work together. If I want to have servers and clients that need to be backward compatible with RC servers and clients. And assuming the old ones didn't used the private data in the connection request and in the accept. Than the presence of the private data will indicated that these are XRC enabled clients and servers.

Quoted 6 lines

This is just a workaround I been considering in case using "hints" and asynchronous connection request can't work together. If I want to have servers and clients that need to be backward compatible with RC servers and clients. And assuming the old ones didn't used the private data in the connection request and in the accept. Than the presence of the private data will indicated that these are XRC enabled clients and servers.

I don't see how an app can 'just switch' from RC to XRC without having coded for the possibility of XRC. XRC introduces too many new concepts.