rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change
Subsystems (3)
PATCH 1/2 rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change
We observed a scenario in IB bonding where RDMA_CM_EVENT_ADDR_CHANGE is followed by RDMA_CM_EVENT_DISCONNECTED on a connected endpoint. This sequence causes a negative reference splat and subsequent tear-down issues due to a duplication in the disconnection path.
This fix aligns with the approach taken in a previous change 4836da219781 ("rpcrdma: fix handling for RDMA_CM_EVENT_DEVICE_REMOVAL"), addressing a similar issue.
--- net/sunrpc/xprtrdma/verbs.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +2 -1
@@ -273,7 +273,8 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)wake_up_all (&ep->re_connect_wait);return 0 ;case RDMA_CM_EVENT_DISCONNECTED : - ep->re_connect_status = -ECONNABORTED ; +if (xchg (&ep->re_connect_status, -ECONNABORTED ) !=1 ) +break ; disconnected:rpcrdma_force_disconnect (ep);return rpcrdma_ep_put (ep);
Signature · 1 line
2.39.3
PATCH 2/2 rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE
It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.
--- net/sunrpc/xprtrdma/verbs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/verbs.c +2 -3
@@ -244,14 +244,13 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)case RDMA_CM_EVENT_DEVICE_REMOVAL :pr_info ("rpcrdma: removing device %s for %pISpc\n" , ep->re_id->device->name, sap); + fallthrough; +case RDMA_CM_EVENT_ADDR_CHANGE :switch (xchg (&ep->re_connect_status, -ENODEV )) {case 0 : goto wake_connect_worker;case 1 : goto disconnected; }return 0 ; -case RDMA_CM_EVENT_ADDR_CHANGE : - ep->re_connect_status = -ENODEV ; - goto disconnected;case RDMA_CM_EVENT_ESTABLISHED :rpcrdma_ep_get (ep); ep->re_connect_status =1 ;
Signature · 1 line
2.39.3
Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE
Quoted 3 lines · Dan Aloni
It would be beneficial to implement handling similar to
RDMA_CM_EVENT_DEVICE_REMOVALin the case whereRDMA_CM_EVENT_ADDR_CHANGEis issued for an unconnected CM.
I think Chuck has a pending series for handling device removals with a dedicated ib_client. So one would have to rebase against the other...
See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal
Other than that, looks good,
Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE
Quoted 7 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni
It would be beneficial to implement handling similar to
RDMA_CM_EVENT_DEVICE_REMOVALin the case whereRDMA_CM_EVENT_ADDR_CHANGEis issued for an unconnected CM.
Quoted 8 lines
I think Chuck has a pending series for handling device removals with a dedicated
ib_client. So one would have to rebase against the other...See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal
Other than that, looks good,
Reviewed-by Sagi Grimberg <[email protected]>
I ran some testing with device-removal branch (merge fe4cef916d91 v6.10-rc7), and found that the RDMA_CM_EVENT_ADDR_CHANGE fix is still required and independent of the branch's content.
And for the second patch I sent, which fixes only a theoretical issue, I think we need the case RDMA_CM_EVENT_ADDR_CHANGE: branch to implement the same logic like the removed case RDMA_CM_EVENT_DEVICE_REMOVAL: branch.
Signature · 1 line
Dan Aloni
Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE
Quoted 10 lines · Dan Aloni
Quoted 9 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni
It would be beneficial to implement handling similar to
RDMA_CM_EVENT_DEVICE_REMOVALin the case whereRDMA_CM_EVENT_ADDR_CHANGEis issued for an unconnected CM.I think Chuck has a pending series for handling device removals with a dedicated
ib_client. So one would have to rebase against the other...
Quoted 7 lines
Quoted 4 lines
See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal
Other than that, looks good,
Reviewed-by Sagi Grimberg <[email protected]>I ran some testing with
device-removalbranch (mergefe4cef916d91v6.10-rc7), and found that theRDMA_CM_EVENT_ADDR_CHANGEfix is still required and independent of the branch's content.
Yes it isn't related, was just mentioning that Chuck is touching this exact area in the code, so perhaps these patches should be against this branch (if indeed it is heading upstream)? Otherwise Chuck can rebase on top of these fixes, which I agree are needed.
Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE
In reply to Sagi GrimbergQuoted 8 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni
It would be beneficial to implement handling similar to
RDMA_CM_EVENT_DEVICE_REMOVALin the case whereRDMA_CM_EVENT_ADDR_CHANGEis issued for an unconnected CM.
Quoted 7 lines
I think Chuck has a pending series for handling device removals with a dedicated
ib_client. So one would have to rebase against the other...See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal
Other than that, looks good,
Reviewed-by Sagi Grimberg <[email protected]>
FYI Anna has taken my "rpcrdma ib_client" series:
http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=shortlog;h=refs/heads/linux-next
PATCH 1/2 "rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change" is a fix that you probably want to make available to the LTS kernels, so it would have to be applied before my patches in order to make that possible. I don't know what Anna's policy is regarding rearranging the patches in her linux-next branch.
Quite possibly she'll just have to rip out my patches so we can give her a fresh series with your work and mine properly ordered.
--
Chuck Lever
Quoted 8 lines · Dan Aloni
We observed a scenario in IB bonding where
RDMA_CM_EVENT_ADDR_CHANGEis followed byRDMA_CM_EVENT_DISCONNECTEDon a connected endpoint. This sequence causes a negative reference splat and subsequent tear-down issues due to a duplication in the disconnection path.This fix aligns with the approach taken in a previous change
4836da219781("rpcrdma: fix handling forRDMA_CM_EVENT_DEVICE_REMOVAL"), addressing a similar issue.
I think a code comment will help here. This whole handler is not very intuitive (but that may be a result of the rdma_cm state machine, the picture in other ulps do not look materially different).