race in ULPs when processing RDMA_CM_EVENT_DEVICE_REMOVAL

Hi Sean,

When the low level driver exercises the hot unplug code (e.g if the user does modprobe -r, pci hot unplug, etc) they would call the rdma-cm remove_one callback, which would go and generate RDMA_CM_EVENT_DEVICE_REMOVAL event for the cma consumers. Now, if the consumer doesn't make sure they destroy all the IB objects created on that ll device instance (e.g mlx4_0) prior to finalizing all processing of the DEVICE_REMOVAL callback, the rdma-cm will let the low level driver green light to finalize its de-registation (destruction of the IB device instance etc) with the IB core, and a call from the consumer to (say) ib_destroy_cq(dev, cq) will crash since that dev object is practically null or the call points to a function/module which doesn't exist any more in the kernel address space - agree?

What would be the correct way to go for consumers, is that making sure they destroy 1st all their IB objects (PDs, MRs, CQs, QPs, etc) prior to destroying the last rdma_cm id on a device removal event? any other idea?

In iSER we don't make sure to destroy all the IB objects prior to acking this event to the rdma-cm and we see crashes under RoCE link layer, where under IB link layer not crashing, most likely some timing which is different between the link layers, but the arch question is still valid.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The user should free any objects that are associated with the device associated with the rdma_cm id. If the user only accesses the device using the rdma_cm id, then all objects should be freed. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

So think on the case, e.g for iser where a ULP maintains a per-device context where they stroe IB objects used for all connections of that device (e.g PD, CQs, DMA-MR, etc) and per connection object where the QP and RDMA-CM ID are storage.

The per-device context is created on demand throughout establishing the 1st connection (e.g after address resolution where the IB device to be used for that connection is "resolved") over a device and destroyed after the last connection is disconnected.

Could we come up with an elegant (or the most not elegant..) way for a secure teardown of the IB elements of that global context in the context of the device removal event?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I would think that this would do it:

- call rdma_destroy_qp() from the callback
- decrement the global reference count
- destroy the global context from the callback when count = 0
- return non-zero from the callback to destroy the rdma cm id

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yep, generally speaking this makes some sense, we need to see how to really make it work, in the presence of some other considerations specific to iser. At least now we understand clearly the problem of the current code, I just wonder why over IB we don't hit the crash... oh well.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html