NFS/RDMA Release resources in svcrdma when device is removed
Subsystems (3)
RFC PATCH 1/1 NFS/RDMA Release resources in svcrdma when device is removed
When removing underlying RDMA device, the rmmod will hang forever if there are any outstanding NFS/RDMA client mounts. The outstanding NFS/RDMA counts could also prevent the server from shutting down. Further debugging shows that the existing connections are not teared down and resource are not released when receiving RDMA_CM_EVENT_DEVICE_REMOVAL event. It seems the original code missing svc_xprt_put() in RDMA_CM_EVENT_REMOVAL event handler thus svc_xprt_free is never invoked to release the existing connection resources.
The patch has been passed removing, adding device back and forth without stopping NFS/RDMA service. This will also allow a device to be unplugged and swapped out without shutting down NFS service.
--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 1 file changed, 1 insertion(+)
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
@@ -673,6 +673,7 @@ static int rdma_cma_handler(struct rdma_cm_id *cma_id,if (xprt) {set_bit (XPT_CLOSE , &xprt->xpt_flags);svc_xprt_enqueue (xprt); +svc_xprt_put (xprt); }break ;default :
Shirley
Quoted 8 lines · Shirley Ma
When removing underlying RDMA device, the rmmod will hang forever if there are any outstanding NFS/RDMA client mounts. The outstanding NFS/RDMA counts could also prevent the server from shutting down. Further debugging shows that the existing connections are not teared down and resource are not released when receiving
RDMA_CM_EVENT_DEVICE_REMOVALevent. It seems the original code missingsvc_xprt_put()inRDMA_CM_EVENT_REMOVALevent handler thussvc_xprt_freeis never invoked to release the existing connection resources.
Quoted 5 lines
The patch has been passed removing, adding device back and forth without stopping NFS/RDMA service. This will also allow a device to be unplugged and swapped out without shutting down NFS service.
Signed-off-by Shirley Ma <[email protected]>
Quoted 17 lines
---
net/sunrpc/xprtrdma/svc_rdma_transport.c| 1 +1 file changed, 1 insertion(+)net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-0@@ -673,6 +673,7 @@ static int rdma_cma_handler(struct rdma_cm_id *cma_id,if (xprt) {set_bit (XPT_CLOSE , &xprt->xpt_flags);svc_xprt_enqueue (xprt); +svc_xprt_put (xprt); }break ;default :
Quoted 1 line
Shirley
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 11 lines · Chuck Lever
Quoted 8 lines · Shirley Ma
When removing underlying RDMA device, the rmmod will hang forever if there are any outstanding NFS/RDMA client mounts. The outstanding NFS/RDMA counts could also prevent the server from shutting down. Further debugging shows that the existing connections are not teared down and resource are not released when receiving
RDMA_CM_EVENT_DEVICE_REMOVALevent. It seems the original code missingsvc_xprt_put()inRDMA_CM_EVENT_REMOVALevent handler thussvc_xprt_freeis never invoked to release the existing connection resources.
Quoted 3 lines
Quoted 3 lines
The patch has been passed removing, adding device back and forth without stopping NFS/RDMA service. This will also allow a device to be unplugged and swapped out without shutting down NFS service.
And maybe also add:
here.
Quoted 29 lines
Quoted 1 line
Signed-off-by Shirley Ma <[email protected]>Reviewed-by Chuck Lever <[email protected]>Quoted 17 lines
---
net/sunrpc/xprtrdma/svc_rdma_transport.c| 1 +1 file changed, 1 insertion(+)net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-0@@ -673,6 +673,7 @@ static int rdma_cma_handler(struct rdma_cm_id *cma_id,if (xprt) {set_bit (XPT_CLOSE , &xprt->xpt_flags);svc_xprt_enqueue (xprt); +svc_xprt_put (xprt); }break ;default :Quoted 1 line
Shirley
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 14 lines · Chuck Lever
Quoted 11 lines · Chuck Lever
Quoted 8 lines · Shirley Ma
When removing underlying RDMA device, the rmmod will hang forever if there are any outstanding NFS/RDMA client mounts. The outstanding NFS/RDMA counts could also prevent the server from shutting down. Further debugging shows that the existing connections are not teared down and resource are not released when receiving
RDMA_CM_EVENT_DEVICE_REMOVALevent. It seems the original code missingsvc_xprt_put()inRDMA_CM_EVENT_REMOVALevent handler thussvc_xprt_freeis never invoked to release the existing connection resources.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
The patch has been passed removing, adding device back and forth without stopping NFS/RDMA service. This will also allow a device to be unplugged and swapped out without shutting down NFS service.
And maybe also add:
Yes, this patch has addressed above problem too. I forgot this bug.
Quoted 42 lines
here.
Quoted 29 lines
Quoted 1 line
Signed-off-by Shirley Ma <[email protected]>Reviewed-by Chuck Lever <[email protected]>Quoted 17 lines
---
net/sunrpc/xprtrdma/svc_rdma_transport.c| 1 +1 file changed, 1 insertion(+)net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-0@@ -673,6 +673,7 @@ static int rdma_cma_handler(struct rdma_cm_id *cma_id,if (xprt) {set_bit (XPT_CLOSE , &xprt->xpt_flags);svc_xprt_enqueue (xprt); +svc_xprt_put (xprt); }break ;default :Quoted 1 line
Shirley
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]comQuoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com