Extending rdma_server example

Hi,

I took librdmacm/examples/rdma_server.c and converted it to be a persistent server to serve successive requests. However, this does not work and I have the following problem:

fprintfs from my code (lower case) and librdmacm (caps) for a first successful connection and a second one which causes a segfault

rdma_getaddrinfo
rdma_listen
rdma_getrequest <----- wait for first request
RDMA CREATE QP
UCMA CREATE_CQS
CALLED UCMA CREATE CQS
CREATING RECV_CQ <----- CQ created here OK for client connection
CREATING SEND_CQ <----- this looks good
IBV CREATE QP
rdma_reg_msgs
rdma_post_recv
rdma_accept
rdma_get_recv_comp
rdma_post_send
rdma_get_send_comp
rdma_getrequest <----- wait for second request
RDMA CREATE QP
UCMA CREATE_CQS
CALLED UCMA CREATE CQS <----- !!! no recv/send CQ created here
IBV CREATE QP
rdma_reg_msgs
rdma_post_recv
rdma_accept
rdma_get_recv_comp <----- dereference of id->recv_cq = NULL
Segmentation fault

So I'm wondering why the CQs do not get created second time round and it looks like

1) rdma_get_request passes event_id and (listen_id)id_priv->qp_init_attr to rdma_create_qp()

2) rdma_create_qp passes qp_init_attr to ucma_create_cqs()

3) ucma_create_cqs stores a pointer to the created CQs in attr->recv_cq / send_cq, which is the attr of the listen_id

4) serving the second client, ucma_create_cqs checks attr->recv_cq and does not create a pair of CQs

Feels a little odd to me that the listen_id takes a pointer to the CQs created for the client_id in its qp_init_attr.

Can anyone enlighten me? Am I trying to do something (persistent server) that rdma_get_request is not intended to do?

Cheers,
Jonathan Rosser
Senior R&D Engineer
BBC R&D

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

This is a bug in rdma_create_qp, rdma_get_request, or both.

rdma_create_qp will modify the qp_init_attr parameter, but I need to think whether it should modify the cq pointers. The underlying ibv_create_qp call does not, but does modify other values, so I think this call may be okay.

Because rdma_create_qp modifies the qp_init_attr parameters, rdma_get_request ends up modifying the attributes stored with the listen request. This is definitely an error, and I'll need to come up with a fix for this.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

rdma_create_qp modifies the qp_init_attr structure passed in by the user to return the actual QP capabilities that were allocated. If the qp_init_attr does not specify CQs, the librdmacm will allocate CQs for the user and return them.

rdma_get_request will allocate a QP off newly connected rdma_cm_id if the corresponding listen request is associated with a qp_init_attr structure. The librdmacm passes in the listen-> qp_init_attr structure into the rdma_create_qp call. rdma_create_qp ends up modifying the qp_init_attr's associated with the listen. The result is that future calls to rdma_get_request will use the modified qp attributes, rather than those specified by the user.

Fix this by having rdma_get_request pass in a copy of the qp_init_attr, rather than modifying those associated with the listen. Also update the man page for rdma_create_qp to indicate that the qp_init_attr structure may be modified on output.

---
I added a while(1) loop to rdma_server to allow clients to connected repeatedly, and this worked for me. Jonathan, can you see if this works for your testing as well? If so, I'll commit.

 man/rdma_create_qp.3 |    3 +++
 src/cma.c            |    5 ++++-
 2 files changed, 7 insertions(+), 1 deletions(-)

index 9d2de76..7445db3 100644
@@ -39,6 +39,9 @@ a send or receive completion queue is not specified, then a CQ will be
 allocated by the rdma_cm for the QP, along with corresponding completion
 channels.  Completion channels and CQ data created by the rdma_cm are
 exposed to the user through the rdma_cm_id structure.
+.P
+The actual capabilities and properties of the created QP will be
+returned to the user through the qp_init_attr parameter.
 .SH "SEE ALSO"
 rdma_bind_addr(3), rdma_resolve_addr(3), rdma_destroy_qp(3), ibv_create_qp(3),
 ibv_modify_qp(3)

index 253cf80..7ac5335 100755
@@ -1368,7 +1368,10 @@ int rdma_get_request(struct rdma_cm_id *listen, struct rdma_cm_id **id)
 	}
 
 	if (id_priv->qp_init_attr) {
-		ret = rdma_create_qp(event->id, id_priv->pd, id_priv->qp_init_attr);
+		struct ibv_qp_init_attr attr;
+
+		attr = *id_priv->qp_init_attr;
+		ret = rdma_create_qp(event->id, id_priv->pd, &attr);
 		if (ret)
 			goto err;
 	}

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yesterday I tried setting attr->send/recv_cq = NULL in rdma_get_request() which fixes the bug in a somewhat ugly manner. Passing a copy of the attributes is a much tidier solution, and your patch works for me.

Many Thanks,
Jonathan.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html