Skip to contentSkip to conversation

rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change

PATCH In review Dan Aloni 3 participants 7 messages Jul 11, 2024

Subsystems (3)
lore
Share & tools
lore

Dan Aloni

author
#0 lore raw

PATCH 1/2 rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change

We observed a scenario in IB bonding where RDMA_CM_EVENT_ADDR_CHANGE is followed by RDMA_CM_EVENT_DISCONNECTED on a connected endpoint. This sequence causes a negative reference splat and subsequent tear-down issues due to a duplication in the disconnection path.

This fix aligns with the approach taken in a previous change 4836da219781 ("rpcrdma: fix handling for RDMA_CM_EVENT_DEVICE_REMOVAL"), addressing a similar issue.

---
 net/sunrpc/xprtrdma/verbs.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +2 -1
index 432557a553e7..e42f5664ecaf 100644
@@ -273,7 +273,8 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)
 		wake_up_all(&ep->re_connect_wait);
 		return 0;
 	case RDMA_CM_EVENT_DISCONNECTED:
-		ep->re_connect_status = -ECONNABORTED;
+		if (xchg(&ep->re_connect_status, -ECONNABORTED) != 1)
+			break;
 disconnected:
 		rpcrdma_force_disconnect(ep);
 		return rpcrdma_ep_put(ep);
Signature · 1 line
2.39.3
author
#1 lore raw

PATCH 2/2 rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE

It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.

---
 net/sunrpc/xprtrdma/verbs.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/verbs.c +2 -3
index e42f5664ecaf..0342a0527733 100644
@@ -244,14 +244,13 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event)
 	case RDMA_CM_EVENT_DEVICE_REMOVAL:
 		pr_info("rpcrdma: removing device %s for %pISpc\n",
 			ep->re_id->device->name, sap);
+		fallthrough;
+	case RDMA_CM_EVENT_ADDR_CHANGE:
 		switch (xchg(&ep->re_connect_status, -ENODEV)) {
 		case 0: goto wake_connect_worker;
 		case 1: goto disconnected;
 		}
 		return 0;
-	case RDMA_CM_EVENT_ADDR_CHANGE:
-		ep->re_connect_status = -ENODEV;
-		goto disconnected;
 	case RDMA_CM_EVENT_ESTABLISHED:
 		rpcrdma_ep_get(ep);
 		ep->re_connect_status = 1;
Signature · 1 line
2.39.3
Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE

Quoted 3 lines · Dan Aloni

It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.

I think Chuck has a pending series for handling device removals with a dedicated ib_client. So one would have to rebase against the other...

See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal

Other than that, looks good,

author
#3 lore raw

Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE

Quoted 7 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni

It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.

Quoted 8 lines

I think Chuck has a pending series for handling device removals with a dedicated
ib_client. So one would have to rebase against the other...

See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal

Other than that, looks good,

I ran some testing with device-removal branch (merge fe4cef916d91 v6.10-rc7), and found that the RDMA_CM_EVENT_ADDR_CHANGE fix is still required and independent of the branch's content.

And for the second patch I sent, which fixes only a theoretical issue, I think we need the case RDMA_CM_EVENT_ADDR_CHANGE: branch to implement the same logic like the removed case RDMA_CM_EVENT_DEVICE_REMOVAL: branch.

Signature · 1 line
Dan Aloni
#4 lore raw

Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE

Quoted 10 lines · Dan Aloni
Quoted 9 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni

It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.

I think Chuck has a pending series for handling device removals with a dedicated
ib_client. So one would have to rebase against the other...

Quoted 7 lines
Quoted 4 lines

See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal

Other than that, looks good,

I ran some testing with device-removal branch (merge fe4cef916d91 v6.10-rc7), and found that the RDMA_CM_EVENT_ADDR_CHANGE fix is still required and independent of the branch's content.

Yes it isn't related, was just mentioning that Chuck is touching this exact area in the code, so perhaps these patches should be against this branch (if indeed it is heading upstream)? Otherwise Chuck can rebase on top of these fixes, which I agree are needed.

#5 lore raw

Re: [PATCH 2/2] rpcrdma: improve handling of RDMA_CM_EVENT_ADDR_CHANGE

In reply to Sagi Grimberg
Quoted 8 lines · Sagi Grimberg
Quoted 3 lines · Dan Aloni

It would be beneficial to implement handling similar to RDMA_CM_EVENT_DEVICE_REMOVAL in the case where RDMA_CM_EVENT_ADDR_CHANGE is issued for an unconnected CM.

Quoted 7 lines

I think Chuck has a pending series for handling device removals with a dedicated ib_client. So one would have to rebase against the other...

See: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.gittopic-device-removal

Other than that, looks good,

FYI Anna has taken my "rpcrdma ib_client" series:

http://git.linux-nfs.org/?p=anna/linux-nfs.git;a=shortlog;h=refs/heads/linux-next

PATCH 1/2 "rpcrdma: fix handling for RDMA_CM_EVENT_DISCONNECTED due to address change" is a fix that you probably want to make available to the LTS kernels, so it would have to be applied before my patches in order to make that possible. I don't know what Anna's policy is regarding rearranging the patches in her linux-next branch.

Quite possibly she'll just have to rip out my patches so we can give her a fresh series with your work and mine properly ordered.

--
Chuck Lever

#6 lore raw
In reply to Dan Aloni
Quoted 8 lines · Dan Aloni

We observed a scenario in IB bonding where RDMA_CM_EVENT_ADDR_CHANGE is followed by RDMA_CM_EVENT_DISCONNECTED on a connected endpoint. This sequence causes a negative reference splat and subsequent tear-down issues due to a duplication in the disconnection path.

This fix aligns with the approach taken in a previous change 4836da219781 ("rpcrdma: fix handling for RDMA_CM_EVENT_DEVICE_REMOVAL"), addressing a similar issue.

I think a code comment will help here. This whole handler is not very intuitive (but that may be a result of the rdma_cm state machine, the picture in other ulps do not look materially different).