RDMA CM + XRC, take two

This is an improved version of my previous patch to add XRC support to RDMA CM.

This version now support the newer rdma CM interface. It needs some polishing but it works.

It modifies the rdma_client/server examples to use xrc instead of rc.

The big hack is that the SRQ number needs to be transmitted to the remote side. This patch hijacks the private data, so it's not acceptable. Ideally the SRQ number should be transmitted either in the REQ or REP packet (depending on which side the sender or the receiver) alongside the QP number. But that would need a change in the specs. Any suggestions ?

Also a good chunk of the patch is to deal with the XRC verbs API. I wonder whether XRC could/should be more integrated into the existing verbs:

- sender should not need a domain,

- there should be 2 types of xrc QPs (send and receive) instead of one,

Quoted 2 lines

This is an improved version of my previous patch to add XRC support to RDMA CM.

I would still like to see support merged upstream and into mainline libibverbs before applying this to my tree.

Quoted 3 lines

This version now support the newer rdma CM interface. It needs some polishing but it works.

It's hard to comment on the patch as an attachment. In several places, if/else statements were added, but indentation was not added to the opposite part of the statement. There was also a rather large amount of code added to rdma_create_qp() that needs to be split into a separate function. (I dislike functions longer than a screen.)

Quoted 8 lines

It modifies the rdma_client/server examples to use xrc instead of rc.

The big hack is that the SRQ number needs to be transmitted to the remote side. This patch hijacks the private data, so it's not acceptable. Ideally the SRQ number should be transmitted either in the REQ or REP packet (depending on which side the sender or the receiver) alongside the QP number. But that would need a change in the specs. Any suggestions ?

What are the QPNs carried in the REQ and REP?

Quoted 1 line

What are the QPNs carried in the REQ and REP?

Those for the XRC QPs. Same as a for regular RC connection.

Quoted 3 lines

Quoted 1 line

What are the QPNs carried in the REQ and REP?

Those for the XRC QPs. Same as a for regular RC connection.

I guess I need to go back and study the specs more to understand why the QPN is in the REQ.

The REQ/REP also carry EECN fields which might be usable here, and maybe that was the intent.

Quoted 1 line

I guess I need to go back and study the specs more to understand why the

QPN is in the REQ.

I looked at the kernel headers; struct cm_req_msg and cm_rep_msg, and they both contain the QP numbers.

Quoted 1 line

The REQ/REP also carry EECN fields which might be usable here, and maybe

that was the intent.

The XRC spec does not make room for the srq number. Which implies it's left to the application to transmit it.

Quoted 5 lines

Quoted 1 line

I guess I need to go back and study the specs more to understand why the

QPN is in the REQ.

I looked at the kernel headers; struct cm_req_msg and cm_rep_msg, and they both contain the QP numbers.

I was referring only to *why* the XRC TGT QP needed to know the XRC INI QPN (at a conceptual level). I think it's for control messages.

Quoted 5 lines

Quoted 1 line

The REQ/REP also carry EECN fields which might be usable here, and maybe

that was the intent.

The XRC spec does not make room for the srq number. Which implies it's left to the application to transmit it.

Agreed - I was wondering if using the EECN would work, but I don't think that it does. There would typically be multiple SRQs targeted by a single XRC INI->TGT connection. Maybe SIDR could be adopted for this purpose as the standard discovery technique.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

There is a connection between the ini and TFT qps they have to exchange qpns so that they can check them later The sewn is also sent and used to route messages to the srq at the target

Sent from my iPhone

Quoted 14 lines · Sean

Quoted 5 lines

Quoted 1 line

I guess I need to go back and study the specs more to understand why the

QPN is in the REQ.

I looked at the kernel headers; struct cm_req_msg and cm_rep_msg, and they both contain the QP numbers.

I was referring only to *why* the XRC TGT QP needed to know the XRC INI QPN (at a conceptual level). I think it's for control messages.

Quoted 5 lines

Quoted 1 line

The REQ/REP also carry EECN fields which might be usable here, and maybe

that was the intent.

The XRC spec does not make room for the srq number. Which implies it's left to the application to transmit it.

Quoted 7 lines

Agreed - I was wondering if using the EECN would work, but I don't think that it does. There would typically be multiple SRQs targeted by a single XRC INI->TGT connection. Maybe SIDR could be adopted for this purpose as the standard discovery technique.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

There is a connection between the ini and TFT qps they have to exchange qpns so that they can check them later

The big hack is that the SRQ number needs to be transmitted to the remote side. This patch hijacks the private data, so it's not acceptable. Ideally the SRQ number should be transmitted either in the REQ or REP packet (depending on which side the sender or the receiver) alongside the QP number. But that would need a change in the specs. Any suggestions ?

Also a good chunk of the patch is to deal with the XRC verbs API. I wonder whether XRC could/should be more integrated into the existing verbs:
- sender should not need a domain,
- there should be 2 types of xrc QPs (send and receive) instead of one,
- *_xrc_rcv_qp verbs should be abstracted under the cover in libibverbs,

Quoted 11 lines

The big hack is that the SRQ number needs to be transmitted to the remote side. This patch hijacks the private data, so it's not acceptable. Ideally the SRQ number should be transmitted either in the REQ or REP packet (depending on which side the sender or the receiver) alongside the QP number. But that would need a change in the specs. Any suggestions ?

Also a good chunk of the patch is to deal with the XRC verbs API. I wonder whether XRC could/should be more integrated into the existing verbs:
- sender should not need a domain,
- there should be 2 types of xrc QPs (send and receive) instead of one,
- *_xrc_rcv_qp verbs should be abstracted under the cover in libibverbs,

I've spent some time reading over the XRC patches in Roland's git tree and the XRC patches to OFED's version of libibverbs. These are some of the ideas that I've jotted down to support XRC through the librdmacm and mainline libibverbs, in no specific order. (There may very well be implementation issues with these.)

1. The IB CM needs to be updated to connect XRC INI QPs to XRC TGT QPs. This should be fairly simple.

2. As mentioned, there's no standard way of obtaining the SRQN. I will submit a comment to the IBTA on this. My recommendation will be to use the IB CM SIDR protocol. 2a. As an optimization, the IB CM REP could optionally return an SRQN in the EECN.
2b. It may be useful if the SIDR REQ carried the XRC INI/TGT QPNs

3. I don't see an easy way to hide the 'XRC domain'. However, if I look at the existing libibverbs and librdmacm APIs, it may be simpler for the user and API compatibility if it were abstracted behind a PD (struct ibv_pd). For example, a kernel XRC domain could be created the first time an XRC object is allocated on a PD. In order to share an XRC domain among multiple processes, we would need a new call (ibv_share_pd? ibv_modify_pd?).

4. There doesn't seem to be a strong reason to expose the XRC TGT QP to user space. A kernel XRC component could accept and manage XRC target connections.

5. Assuming that XRC TGT QP is not exposed, libibverbs would use IBV_QPT_XRC only as the send side QP. In order to support this QPT through the librdmacm, we would need to know what port space XRC QPs use, to know what SID range to map to, if any. I will submit a comment to the IBTA on how XRC makes use of the RDMA IP CM Service.

6. The XRC SRQ is more troubling to fit under the existing APIs. The one idea I had was to treat the XRC SRQ as a QP (struct ibv_qp) rather than like an SRQ (struct ibv_srq). This would require defining an IBV_QPT_SRQ type. The SRQN ends up being a QPN for all purposes, though I don't know if this would cause other issues if the SRQNs and real QPNs overlap.

With these changes, the librdmacm usage model would be:

Passive side:

   rdma_create_ep()  /* qp_type = IBV_QPT_SRQ */
   rdma_listen()  /* listen for SIDR REQ */
   rdma_get_cm_event()
   rdma_accept()

Active side:
rdma_getaddrinfo() /* qp_type = IBV_QPT_XRC */
rdma_create_ep()
rdma_connect() /* connects to XRC TGT QP */

rdma_getaddrinfo() /* qp_type = IBV_QPT_SRQ */
rdma_create_ep()
rdma_connect() /* resolves XRC SRQN */

This doesn't include code necessary to share the 'XRC domain' among multiple processes on the passive side. On the passive side, the kernel XRC component would respond to IB CM REQs based on whether there was an associated SRQ listen.

I realize this model would deviate from the OFED libibverbs APIs, but I don't want to break the librdmacm ABI, or necessarily add an entire new set of APIs just to support XRC.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The typical use case foe xrc is for lots of related processes on each node. In this case it would be helpful to cache these to avoid P^2 effects. The CM seems like a nice place to do this. The usual problem is knowing when to stop.

I'm not thrilled with 3. Seems to me like these concepts don't merge well. The idea behind the implementation is to let apps use the file system permissions to control who can join. Whatever we do has to preserve this.