smbdirect: don't hang on netdev reconfiguration
Subsystems (3)
cover smbdirect: don't hang on netdev reconfiguration
When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just ls the mount point in this state gives EHOSTDOWN.
I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the RDMA_CM_CONNECT state, and no callback moves it out. The SMB Direct layer never gets the RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.
Fix this in the CMA. If we call rdma_disconnect on a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send the RDMA_CM_EVENT_DISCONNECTED event to SMB Direct.
I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.
Artifacts for reproducing the hang and testing my fix are available at
https://github.com/ammrat13/linux-cifs
There, you can find: logs from the "bad" and "good" kernels, the kernel configuration, the VMs' configuration, and a playbook to trigger the hang.
Ammar Ratnani (2):
infiniband: force disconnect if DREP and DREQ fail
infiniband: trace force disconnections
drivers/infiniband/core/cma.c | 41 ++++++++++++++++++++++++++--- drivers/infiniband/core/cma_trace.h | 1 + 2 files changed, 38 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0
PATCH 1/2 infiniband: force disconnect if DREP and DREQ fail
When we call rdma_disconnect, it's possible that we fail to send both a DREQ and a DREP. In that case, we currently either do nothing or move into the IB_CM_TIMEWAIT state. Either way, we remain in the RDMA_CM_CONNECT state. Anyone waiting for us to disconnect hangs.
Instead, while we're connected, when we call rdma_disconnect, and fail to send both a DREQ and a DREP, mark the connection as disconnected. Call the upper layer's handler for the disconnect, and destroy the connection if instructed.
--- drivers/infiniband/core/cma.c | 39 +++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)
drivers/infiniband/core/cma.c +35 -4
@@ -4823,6 +4823,34 @@ int rdma_reject(struct rdma_cm_id *id, const void *private_data, }EXPORT_SYMBOL (rdma_reject); +/* +* Called from rdma_disconnect() when we know we're not receiving a callback. +* Emulate what that callback (in cma_ib_handler()) would have done. +* +* NOTE: Despite the name, this function will only actually disconnect when +* we're in the RDMA_CM_CONNECT state. Else, it does nothing. +*/ +static int cma_force_disconnect (struct rdma_id_private *id_priv) +{ +struct cma_work *work; + + work =kzalloc_obj (*work); +if (!work) +return -ENOMEM ; + +INIT_WORK (&work->work, cma_work_handler); + work->old_state =RDMA_CM_CONNECT ; + work->new_state =RDMA_CM_DISCONNECT ; + work->event.event =RDMA_CM_EVENT_DISCONNECTED ; + work->event.status = -ECONNABORTED ; + +cma_id_get (id_priv); + work->id = id_priv; + +queue_work (cma_wq, &work->work); +return 0 ; +} +int rdma_disconnect (struct rdma_cm_id *id) {struct rdma_id_private *id_priv;
@@ -4838,12 +4866,15 @@ int rdma_disconnect(struct rdma_cm_id *id) goto out;/* Initiate or respond to a disconnect. */ trace_cm_disconnect (id_priv); -if (ib_send_cm_dreq (id_priv->cm_id.ib,NULL ,0 )) { -if (!ib_send_cm_drep (id_priv->cm_id.ib,NULL ,0 )) -trace_cm_sent_drep (id_priv); - }else { +if (!ib_send_cm_dreq (id_priv->cm_id.ib,NULL ,0 )) {trace_cm_sent_dreq (id_priv); + goto out; + } +if (!ib_send_cm_drep (id_priv->cm_id.ib,NULL ,0 )) { +trace_cm_sent_drep (id_priv); + goto out; } + ret =cma_force_disconnect (id_priv); }else if (rdma_cap_iw_cm (id->device, id->port_num)) { ret =iw_cm_disconnect (id_priv->cm_id.iw,0 ); }else
Signature · 1 line
2.55.0
PATCH 2/2 infiniband: trace force disconnections
In reply to Ammar RatnaniWhen we failed to send both a DREQ and a DREP, we have to infer that by noting the absence of the sent_dreq and sent_drep trace events. It would be better if we could see this case by the presence of a trace event instead. Implement that.
--- drivers/infiniband/core/cma.c | 2 ++ drivers/infiniband/core/cma_trace.h | 1 + 2 files changed, 3 insertions(+)
drivers/infiniband/core/cma.c +2 -0
@@ -4834,6 +4834,8 @@ static int cma_force_disconnect(struct rdma_id_private *id_priv) {struct cma_work *work; +trace_cm_force_disconnect (id_priv); + work =kzalloc_obj (*work);if (!work)return -ENOMEM ;
drivers/infiniband/core/cma_trace.h +1 -0
@@ -61,6 +61,7 @@ DEFINE_CMA_FSM_EVENT(send_sidr_rep);DEFINE_CMA_FSM_EVENT (disconnect);DEFINE_CMA_FSM_EVENT (sent_drep);DEFINE_CMA_FSM_EVENT (sent_dreq); +DEFINE_CMA_FSM_EVENT (force_disconnect);DEFINE_CMA_FSM_EVENT (id_destroy); TRACE_EVENT(cm_id_attach,
Signature · 1 line
2.55.0
Hi Ammar,
thanks for the fixes!
I let the rdma maintainers comment on the patches in detail, but it's good to fix these problems where they happen.
Quoted 26 lines
When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just
lsthe mount point in this state gives EHOSTDOWN.I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the
RDMA_CM_CONNECTstate, and no callback moves it out. The SMB Direct layer never gets theRDMA_CM_EVENT_DISCONNECTEDit's waiting for, and hangs.Fix this in the CMA. If we call
rdma_disconnecton a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send theRDMA_CM_EVENT_DISCONNECTEDevent to SMB Direct.I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit
9f0346dcbea3("Merge tag 'driver-core-7.3-rc2' ofgit://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.
Quoted 3 lines
Artifacts for reproducing the hang and testing my fix are available at
Given you have some automation to reproduce it I'm wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/[email protected]/
That was reported as fix for ksmbd, but I guess it will also happen for the case your're seeing with rxe. And your fixes are likely also good for ksmbd.
Thanks!
metze
Thank you for the very quick response, Stefan!
Quoted 3 lines
Given you have some automation to reproduce it I'm wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/[email protected]/
I tried running my setup for iWARP. I used the old "bad" kernel, and the only thing I changed was that I used an siw device instead of an rxe. I do not get a hang in that case. To be more precise, I see the following at the end of dmesg:
[ 40.006460] siw0: Port: 1 Link DOWN
[ 40.012912] siw0: Port: 1 Link ACTIVEBut I don't see any message of the form "INFO: task ... blocked for more than ... seconds." Additionally, I'm able to list and read a file off the CIFS mount, even after bringing the interface down and up.
For this test, I used commit 2beb1b31a12b ("Merge tag 'bpf-fixes' ofgit://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf"). This is a later tree than what I used in my last email, but I confirmed that I get the same behavior I described in that email.
Thank you,
- Ammar Ratnani
Quoted 5 lines · Stefan Metzmacher
Hi Ammar,
thanks for the fixes!
I let the rdma maintainers comment on the patches in detail, but it's good to fix these problems where they happen.
I think that patches are ok.
Thanks
Leon Romanovsky
AppliedQuoted 15 lines · Ammar Ratnani
When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just
lsthe mount point in this state gives EHOSTDOWN.I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the
RDMA_CM_CONNECTstate, and no callback moves it out. The SMB Direct layer never gets theRDMA_CM_EVENT_DISCONNECTEDit's waiting for, and hangs.
Quoted 1 line
[...]
Applied, thanks!
[1/2] infiniband: force disconnect if DREP and DREQ fail
https://git.kernel.org/rdma/rdma/c/ba88a28b47a703
[2/2] infiniband: trace force disconnections
(no commit info)
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>
Quoted 28 lines · Ammar Ratnani
When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel thread hang if I "configure" its slave network device by taking its link down and bringing it back up. Attempting to just
lsthe mount point in this state gives EHOSTDOWN.I believe the following is the root-cause of the hang: When the link is taken down, the corresponding GID Table Entry is marked as pending deletion and has its slave ndev set to NULL. All sends on RDMA connections still using that GID Table Entry fail at MAD creation. SMB Direct eventually detects this and tries to disconnect / reconnect to recover. Unfortunately, disconnecting requires successfully sending either a DREQ or a DREP. Since neither of them even post, the connection remains in the
RDMA_CM_CONNECTstate, and no callback moves it out. The SMB Direct layer never gets theRDMA_CM_EVENT_DISCONNECTEDit's waiting for, and hangs.Fix this in the CMA. If we call
rdma_disconnecton a connected connection and we fail to send both a DREP and a DREQ; disconnect, and thereby send theRDMA_CM_EVENT_DISCONNECTEDevent to SMB Direct.I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a mainline kernel. On commit
9f0346dcbea3("Merge tag 'driver-core-7.3-rc2' ofgit://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately disconnects and reconnects when the slave device is brought down then up. Listing and reading files from the mount point also work afterwards.Artifacts for reproducing the hang and testing my fix are available at
Quoted 8 lines
https://github.com/ammrat13/linux-cifs
There, you can find: logs from the "bad" and "good" kernels, the kernel configuration, the VMs' configuration, and a playbook to trigger the hang.
Ammar Ratnani (2):
infiniband: force disconnect if DREP and DREQ fail
infiniband: trace force disconnections
I squashed the second patch into the first one, fixed the subject, removed the useless AI-generated comments, placed trace call into right place, and pushed the result.
Thanks
Quoted 7 lines
drivers/infiniband/core/cma.c | 41 ++++++++++++++++++++++++++--- drivers/infiniband/core/cma_trace.h | 1 + 2 files changed, 38 insertions(+), 4 deletions(-)Signature · 1 line
2.55.0