Skip to contentSkip to conversation

smbdirect: don't hang on netdev reconfiguration

PATCH Applied Ammar Ratnani 3 participants 8 messages Sep 8 – Sep 22

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover smbdirect: don't hang on netdev reconfiguration

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call rdma_disconnect on a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send the RDMA_CM_EVENT_DISCONNECTED event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.

Artifacts for reproducing the hang and testing my fix are available at

https://github.com/ammrat13/linux-cifs

There, you can find: logs from the "bad" and "good" kernels, the kernel configuration, the VMs' configuration, and a playbook to trigger the hang.

Ammar Ratnani (2):
infiniband: force disconnect if DREP and DREQ fail
infiniband: trace force disconnections

 drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
 drivers/infiniband/core/cma_trace.h |  1 +
 2 files changed, 38 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/2 infiniband: force disconnect if DREP and DREQ fail

When we call rdma_disconnect, it's possible that we fail to send both a DREQ and a DREP. In that case, we currently either do nothing or move into the IB_CM_TIMEWAIT state. Either way, we remain in the RDMA_CM_CONNECT state. Anyone waiting for us to disconnect hangs.

Instead, while we're connected, when we call rdma_disconnect, and fail to send both a DREQ and a DREP, mark the connection as disconnected. Call the upper layer's handler for the disconnect, and destroy the connection if instructed.

Cc Leon Romanovsky <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Namjae Jeon <[email protected]>
Cc Paulo Alcantara <[email protected]>
Cc Stefan Metzmacher <[email protected]>
Cc Tom Talpey <[email protected]>
---
 drivers/infiniband/core/cma.c | 39 +++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
drivers/infiniband/core/cma.c +35 -4
index 73170b15fc3d..1f1a8b4e160a 100644
@@ -4823,6 +4823,34 @@ int rdma_reject(struct rdma_cm_id *id, const void *private_data,
 }
 EXPORT_SYMBOL(rdma_reject);
 
+/*
+ * Called from rdma_disconnect() when we know we're not receiving a callback.
+ * Emulate what that callback (in cma_ib_handler()) would have done.
+ *
+ * NOTE: Despite the name, this function will only actually disconnect when
+ * we're in the RDMA_CM_CONNECT state. Else, it does nothing.
+ */
+static int cma_force_disconnect(struct rdma_id_private *id_priv)
+{
+	struct cma_work *work;
+
+	work = kzalloc_obj(*work);
+	if (!work)
+		return -ENOMEM;
+
+	INIT_WORK(&work->work, cma_work_handler);
+	work->old_state = RDMA_CM_CONNECT;
+	work->new_state = RDMA_CM_DISCONNECT;
+	work->event.event = RDMA_CM_EVENT_DISCONNECTED;
+	work->event.status = -ECONNABORTED;
+
+	cma_id_get(id_priv);
+	work->id = id_priv;
+
+	queue_work(cma_wq, &work->work);
+	return 0;
+}
+
 int rdma_disconnect(struct rdma_cm_id *id)
 {
 	struct rdma_id_private *id_priv;
@@ -4838,12 +4866,15 @@ int rdma_disconnect(struct rdma_cm_id *id)
 			goto out;
 		/* Initiate or respond to a disconnect. */
 		trace_cm_disconnect(id_priv);
-		if (ib_send_cm_dreq(id_priv->cm_id.ib, NULL, 0)) {
-			if (!ib_send_cm_drep(id_priv->cm_id.ib, NULL, 0))
-				trace_cm_sent_drep(id_priv);
-		} else {
+		if (!ib_send_cm_dreq(id_priv->cm_id.ib, NULL, 0)) {
 			trace_cm_sent_dreq(id_priv);
+			goto out;
+		}
+		if (!ib_send_cm_drep(id_priv->cm_id.ib, NULL, 0)) {
+			trace_cm_sent_drep(id_priv);
+			goto out;
 		}
+		ret = cma_force_disconnect(id_priv);
 	} else if (rdma_cap_iw_cm(id->device, id->port_num)) {
 		ret = iw_cm_disconnect(id_priv->cm_id.iw, 0);
 	} else
Signature · 1 line
2.55.0
#2 lore raw

PATCH 2/2 infiniband: trace force disconnections

In reply to Ammar Ratnani

When we failed to send both a DREQ and a DREP, we have to infer that by noting the absence of the sent_dreq and sent_drep trace events. It would be better if we could see this case by the presence of a trace event instead. Implement that.

Cc Leon Romanovsky <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Namjae Jeon <[email protected]>
Cc Paulo Alcantara <[email protected]>
Cc Stefan Metzmacher <[email protected]>
Cc Tom Talpey <[email protected]>
---
 drivers/infiniband/core/cma.c       | 2 ++
 drivers/infiniband/core/cma_trace.h | 1 +
 2 files changed, 3 insertions(+)
drivers/infiniband/core/cma.c +2 -0
index 1f1a8b4e160a..c4eab48a4a9f 100644
@@ -4834,6 +4834,8 @@ static int cma_force_disconnect(struct rdma_id_private *id_priv)
 {
 	struct cma_work *work;
 
+	trace_cm_force_disconnect(id_priv);
+
 	work = kzalloc_obj(*work);
 	if (!work)
 		return -ENOMEM;
drivers/infiniband/core/cma_trace.h +1 -0
index 3456d5f3aa47..3d05d8e80856 100644
@@ -61,6 +61,7 @@ DEFINE_CMA_FSM_EVENT(send_sidr_rep);
 DEFINE_CMA_FSM_EVENT(disconnect);
 DEFINE_CMA_FSM_EVENT(sent_drep);
 DEFINE_CMA_FSM_EVENT(sent_dreq);
+DEFINE_CMA_FSM_EVENT(force_disconnect);
 DEFINE_CMA_FSM_EVENT(id_destroy);
 
 TRACE_EVENT(cm_id_attach,
Signature · 1 line
2.55.0
#3 lore raw
In reply to Ammar Ratnani

Hi Ammar,

thanks for the fixes!
I let the rdma maintainers comment on the patches in detail, but it's good to fix these problems where they happen.

Quoted 26 lines

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call rdma_disconnect on a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send the RDMA_CM_EVENT_DISCONNECTED event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.

Quoted 3 lines

Artifacts for reproducing the hang and testing my fix are available at

https://github.com/ammrat13/linux-cifs

Given you have some automation to reproduce it I'm wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/[email protected]/

That was reported as fix for ksmbd, but I guess it will also happen for the case your're seeing with rxe. And your fixes are likely also good for ksmbd.

Thanks!
metze

#4 lore raw

Thank you for the very quick response, Stefan!

Quoted 3 lines

Given you have some automation to reproduce it I'm wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/[email protected]/

I tried running my setup for iWARP. I used the old "bad" kernel, and the only thing I changed was that I used an siw device instead of an rxe. I do not get a hang in that case. To be more precise, I see the following at the end of dmesg:

    [   40.006460] siw0: Port: 1 Link DOWN
    [   40.012912] siw0: Port: 1 Link ACTIVE

But I don't see any message of the form "INFO: task ... blocked for more than ... seconds." Additionally, I'm able to list and read a file off the CIFS mount, even after bringing the interface down and up.

For this test, I used commit 2beb1b31a12b ("Merge tag 'bpf-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf"). This is a later tree than what I used in my last email, but I confirmed that I get the same behavior I described in that email.

Thank you,
- Ammar Ratnani

#5 lore raw
In reply to Stefan Metzmacher
Quoted 5 lines · Stefan Metzmacher

Hi Ammar,

thanks for the fixes!
I let the rdma maintainers comment on the patches in detail, but it's good to fix these problems where they happen.

I think that patches are ok.

Thanks

#6 lore raw
In reply to Ammar Ratnani
Quoted 15 lines · Ammar Ratnani

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Quoted 1 line

[...]

Applied, thanks!

[1/2] infiniband: force disconnect if DREP and DREQ fail

      https://git.kernel.org/rdma/rdma/c/ba88a28b47a703

[2/2] infiniband: trace force disconnections
(no commit info)

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
#7 lore raw
In reply to Ammar Ratnani
Quoted 28 lines · Ammar Ratnani

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call rdma_disconnect on a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send the RDMA_CM_EVENT_DISCONNECTED event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.

Artifacts for reproducing the hang and testing my fix are available at

Quoted 8 lines

https://github.com/ammrat13/linux-cifs

There, you can find: logs from the "bad" and "good" kernels, the kernel configuration, the VMs' configuration, and a playbook to trigger the hang.

Ammar Ratnani (2):
infiniband: force disconnect if DREP and DREQ fail
infiniband: trace force disconnections

I squashed the second patch into the first one, fixed the subject, removed the useless AI-generated comments, placed trace call into right place, and pushed the result.

Thanks

Quoted 7 lines
 drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
 drivers/infiniband/core/cma_trace.h |  1 +
 2 files changed, 38 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0