rdma provider module references
Hey Roland,
I notice that if I have a user rdma application running that has an rdma connection using iw_cxgb3, then the iw_cxgb3 module reference count is bumped and thus it cannot be unloaded. However when I have an NFSRDMA connection that utilizes iw_cxgb3, the module reference count is not bumped, and iw_cxgb3 can erroneously be unloaded while the NFSRDMA connection is still active, causing a crash.
My question is more of what is the right thing to do here.
Should the module refcnt be the way to prohibit a provider from being unloaded?
Thoughts?
Thanks,
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines
I notice that if I have a user rdma application running that has an rdma connection using
iw_cxgb3, then theiw_cxgb3module reference count is bumped and thus it cannot be unloaded. However when I have an NFSRDMA connection that utilizesiw_cxgb3, the module reference count is not bumped, andiw_cxgb3can erroneously be unloaded while the NFSRDMA connection is still active, causing a crash.
What is supposed to happen is that as the HW driver is unloading, it calls ib_unregister_device() first, and this calls each client's .remove() method to have it release everything related to that device.
However I guess NFS/RDMA is behind the RDMA CM, which is supposed to handle device removal. In that code it seems to end up in cma_process_remove(), which appears at first glance to do the right things to destroy all connections etc.
The idea is that RDMA devices should be like net devices, ie you can remove them even if they're in use -- things should just clean up, rather than blocking the module removal. The uverbs case is a bit of a hack because we don't have a way to handle revoking the mmap regions etc yet.
What goes wrong with NFS/RDMA in this scheme? It looks like it should work.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Roland Dreier
Quoted 6 lines
I notice that if I have a user rdma application running that has an rdma connection using
iw_cxgb3, then theiw_cxgb3module reference count is bumped and thus it cannot be unloaded. However when I have an NFSRDMA connection that utilizesiw_cxgb3, the module reference count is not bumped, andiw_cxgb3can erroneously be unloaded while the NFSRDMA connection is still active, causing a crash.What is supposed to happen is that as the HW driver is unloading, it calls
ib_unregister_device()first, and this calls each client's .remove() method to have it release everything related to that device.However I guess NFS/RDMA is behind the RDMA CM, which is supposed to handle device removal. In that code it seems to end up in
cma_process_remove(), which appears at first glance to do the right things to destroy all connections etc.
Quoted 6 lines
The idea is that RDMA devices should be like net devices, ie you can remove them even if they're in use -- things should just clean up, rather than blocking the module removal. The uverbs case is a bit of a hack because we don't have a way to handle revoking the mmap regions etc yet.
Thanks for the description of how it should work.
Quoted 2 lines
What goes wrong with NFS/RDMA in this scheme? It looks like it should work.
I'm still investigating exactly what happens. I'll follow up once I know more.
Thanks!
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Roland Dreier
Quoted 6 lines
I notice that if I have a user rdma application running that has an rdma connection using
iw_cxgb3, then theiw_cxgb3module reference count is bumped and thus it cannot be unloaded. However when I have an NFSRDMA connection that utilizesiw_cxgb3, the module reference count is not bumped, andiw_cxgb3can erroneously be unloaded while the NFSRDMA connection is still active, causing a crash.What is supposed to happen is that as the HW driver is unloading, it calls
ib_unregister_device()first, and this calls each client's .remove() method to have it release everything related to that device.However I guess NFS/RDMA is behind the RDMA CM, which is supposed to handle device removal. In that code it seems to end up in
cma_process_remove(), which appears at first glance to do the right things to destroy all connections etc.
Quoted 8 lines
The idea is that RDMA devices should be like net devices, ie you can remove them even if they're in use -- things should just clean up, rather than blocking the module removal. The uverbs case is a bit of a hack because we don't have a way to handle revoking the mmap regions etc yet.
What goes wrong with NFS/RDMA in this scheme? It looks like it should work.
Here's one stack. From this I assume the offload connection was still active after iw_cxgb3 was unloaded...
Call Trace: <IRQ> [<ffffffff80037136>] kref_get+0x38/0x3d [<ffffffff885fb5b1>] :iw_cxgb3:sched+0x17/0x49 [<ffffffff8824cf37>] :cxgb3:process_rx+0x37/0x8b [<ffffffff8824a3e7>] :cxgb3:process_responses+0xc09/0xc63 [<ffffffff8824ac65>] :cxgb3:napi_rx_handler+0x36/0xa4 [<ffffffff8000c88a>] net_rx_action+0xac/0x1e0 [<ffffffff8824ac15>] :cxgb3:t3_sge_intr_msix_napi+0x173/0x18d [<ffffffff80012409>] __do_softirq+0x89/0x133 [<ffffffff8005f2fc>] call_softirq+0x1c/0x28 [<ffffffff8006dba8>] do_softirq+0x2c/0x85 [<ffffffff8006da30>] do_IRQ+0xec/0xf5 [<ffffffff800575d0>] mwait_idle+0x0/0x4a [<ffffffff8005e615>] ret_from_intr+0x0/0xa <EOI> [<ffffffff80057606>] mwait_idle+0x36/0x4a [<ffffffff800497be>] cpu_idle+0x95/0xb8 [<ffffffff80078997>] start_secondary+0x498/0x4a7
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
However I guess NFS/RDMA is behind the RDMA CM, which is supposed to
Quoted 4 lines
handle device removal. In that code it seems to end up in
cma_process_remove(), which appears at first glance to do the right things to destroy all connections etc.
Function cma_process_remove() calls cma_remove_id_dev() for each cm_id bound to the device being removed. Function cma_remove_id_dev() calls the event handler function for each cm_id and passes a RDMA_CM_EVENT_DEVICE_REMOVAL event. The NFSRDMA server marks the RPC transport as XPT_CLOSE, but doesn't immediately destroy the cm_id in the event handler function. This is in net/sunrpc/xprtrdma/svc_rdma_transport.c / rdma_cma_handler(). That's the issue methinks. Each RDMA kernel user must destroy all the resources in the event handler function itself. These cannot be scheduled or deferred in any way given the current design.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html