When is it safe to release connection resources?

Hello,

I'm new to RDMA development and I have a question regarding resource release.
If I understood correctly, when ibv_get_cq_event returns, it holds some sort of lock over the completion queue, which is released when I call ibv_req_notify_cq. This lock is checked also in ibv_destroy_cq, so that:
1) When ibv_destroy_cq returns, I am certain that there is no thread running somewhere between ibv_get_cq_event and ibv_req_notify_cq
2) When ibv_destroy_cq returns, I am certain that ibv_get_cq_event will not return the destroyed cq any more.

Is all this correct?

Thanks
Flavio
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

There is a paragraph in the IBTA that warns that the above sequence can cause WQE and data segment leakage in the HCA for QPs associated with an SRQ. I don't know though whether this can also occur for QPs with a private receive queue - I'm not a HCA firmware expert.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

ibv_get_cq_event will increment a reference count on the CQ that it is returning.

You decrement the count with ibv_ack_cq_events. ibv_req_notify_cq is used to arm the cq, so that a completion generates an interrupt and a new event.

ibv_destroy_cq will block until all outstanding references on the cq have been released. The intent is to protect the user from ibv_get_cq_event from returning a reference to a cq that is being destroyed from another thread, which could result in a crash in the user's code.

correct

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

So in order to safely call ibv_destroy_cq, I should call ibv_ack_cq_events *after* ibv_poll_cq? The example on the man page for ibv_get_cq_event calls it before, is it an error?

Flavio

Il 1/2/2012 17:39 PM, Hefty, Sean ha scritto:

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

You should call ibv_ack_cq_events after ibv_get_cq_events but before ibv_destroy_cq. That can come before or after calling ibv_poll_cq. And, although it's probably easiest to call ack after get, note that you don't need a 1:1 call ratio between get/ack. You can keep a count of the number of times that ibv_get_cq_events returns a specific cq, and then call ibv_ack_cq_events for that amount just before calling ibv_destroy_cq.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I've just had a look at the kernel code that implements all this (uverbs_cmd.c and uverbs_main.c). I haven't found any precautions against ib_uverbs_comp_handler() accessing *uobj after ib_uverbs_destroy_cq() has invoked put_uobj(uobj). Did I miss something ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The kernel operation is different, since it relies on callbacks. When the kernel ib_destroy_cq() returns, we are guaranteed that the completion handler (ib_uverbs_comp_handler) is not executing and will not be called. After destroying the kernel cq, ib_uverbs_destroy_cq() will remove all references to the destroyed cq from the event list.

The issue is that another thread could have retrieved an event for this cq before the cleanup occurs. When ib_uverbs_destroy_cq unwinds back to user space, it returns the total number of events that were retrieved from the kernel. ibv_destroy_cq blocks until the application has processed all cq events, which are indicated by the app calling ibv_ack_cq_events().

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Sorry if I wasn't clear enough, but I was referring to the case where the kernel completion handler gets invoked after ib_uverbs_destroy_cq() has started but before that function has invoked ib_destroy_cq().

More in general - assuming that completion notifications have been enabled - I'm wondering whether it is possible to shut down a queue pair without triggering a race condition if that queue pair hasn't been transitioned to the reset or error state first.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

That should be OK. ib_uverbs_destroy_cq() doesn't really do anything until ib_destroy_cq() has returned, and at that point it is guaranteed that the completion handler for the CQ is done.

destroying a QP is pretty much equivalent to transitioning it to the reset state.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

That sounds like a requirement for low-level IB drivers. Has it already been considered to document this requirement in Documentation/infiniband/core_locking.txt ?

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html