Skip to content

smbdirect: don't hang on netdev reconfiguration

Ammar Ratnani linux-cifs linux-rdma lkml 1 version Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest2 patches 5 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/2infiniband: force disconnect if DREP and DREQ faildrivers/infiniband/core/cma.c+35 -40 replies
2/2infiniband: trace force disconnectionsdrivers/infiniband/core/cma.c drivers/infiniband/core/cma_trace.h+3 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/2] smbdirect: don't hang on netdev reconfiguration

When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call rdma_disconnect on a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send the RDMA_CM_EVENT_DISCONNECTED event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.

Artifacts for reproducing the hang and testing my fix are available at

https://github.com/ammrat13/linux-cifs

There, you can find: logs from the "bad" and "good" kernels, the kernel configuration, the VMs' configuration, and a playbook to trigger the hang.

Ammar Ratnani (2):
infiniband: force disconnect if DREP and DREQ fail
infiniband: trace force disconnections

 drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
 drivers/infiniband/core/cma_trace.h |  1 +
 2 files changed, 38 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0