Skip to contentSkip to conversation

RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()

PATCH In review Deepanshu Kartikey 4 participants 6 messages Nov 1 – Nov 4 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()

rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.

syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:

  WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src
  Call Trace:
   rdma_resolve_addr
   rds_ib_conn_path_connect
   rds_connect_worker

This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

Reported-by [email protected]
Fixes 732d41c545bb ("RDMA/cma: Make the locking for automatic state transition more clear")
---
 drivers/infiniband/core/cma.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/core/cma.c +2 -2
index 337a49d1acf7..46741c1fd34b 100644
@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv,
 		ret = cma_bind_addr(&id_priv->id, src_addr, dst_addr);
 		if (ret)
 			return ret;
-		if (WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
-					   RDMA_CM_ADDR_QUERY)))
+		if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
+				   RDMA_CM_ADDR_QUERY))
 			return -EINVAL;
 
 	} else {
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] RDMA/cma: missing comment in resolve_prepare_src() promised by commit message --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] RDMA/cma: missing comment in resolve_prepare_src() promised by commit message --

commit b9b0fc41a9bfe112cf7965b88795f6ea037bc2e0
Author: Deepanshu Kartikey <[email protected]>

RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()

This commit removes the WARN_ON() in resolve_prepare_src() when transitioning an RDMA CM ID from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY after cma_bind_addr(), as a concurrent device removal can legitimately change the state to RDMA_CM_DEVICE_REMOVAL.

Quoted 2 lines

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

[ ... ]

drivers/infiniband/core/cma.c static int resolve_prepare_src(struct rdma_id_private *id_priv, · line 4122

Quoted 13 lines
drivers/infiniband/core/cma.c +2 -2
index 337a49d1acf72..a30f367753241 100644
@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv,
 		ret = cma_bind_addr(&id_priv->id, src_addr, dst_addr);
 		if (ret)
 			return ret;
-		if (WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
-					   RDMA_CM_ADDR_QUERY)))
+		if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
+				   RDMA_CM_ADDR_QUERY))
 			return -EINVAL;

[Severity: Low]
This isn't a bug, but the commit message mentions documenting in a comment that the state may change because of device removal, whereas the diff in resolve_prepare_src() in drivers/infiniband/core/cma.c only removes the WARN_ON() around cma_comp_exch(). Was that comment accidentally left out of the patch, or should the commit message be updated?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Deepanshu Kartikey
Quoted 17 lines · Deepanshu Kartikey

rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.

syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:

   WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src
   Call Trace:
    rdma_resolve_addr
    rds_ib_conn_path_connect
    rds_connect_worker
Quoted 8 lines

This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

Resend this becase some error occurred.

For this specific case, we can consider this behavior expected rather than an error and remove the WARN_ON().

However, I'm not sure whether the WARN_ON() is still necessary for other test cases.

Yanjun Zhu

Quoted 25 lines
Reported-by [email protected]
Fixes 732d41c545bb ("RDMA/cma: Make the locking for automatic state transition more clear")
---
  drivers/infiniband/core/cma.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/core/cma.c +2 -2
index 337a49d1acf7..46741c1fd34b 100644
@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv,
  		ret = cma_bind_addr(&id_priv->id, src_addr, dst_addr);
  		if (ret)
  			return ret;
-		if (WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
-					   RDMA_CM_ADDR_QUERY)))
+		if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
+				   RDMA_CM_ADDR_QUERY))
  			return -EINVAL;
 
  	} else {
Signature · 2 lines
Best Regards,
Yanjun.Zhu
#3 lore raw
Quoted 29 lines · Zhu Yanjun
Quoted 17 lines · Deepanshu Kartikey

rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.

syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:

   WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src
   Call Trace:
    rdma_resolve_addr
    rds_ib_conn_path_connect
    rds_connect_worker
Quoted 8 lines

This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

Quoted 8 lines

Resend this becase some error occurred.

For this specific case, we can consider this behavior expected rather than an error and remove the WARN_ON().

However, I'm not sure whether the WARN_ON() is still necessary for other test cases.

Hi Yanjun,

Thanks for the review. After cma_bind_addr() succeeds the state is ADDR_BOUND, and the only legitimate change before the second exchange is device removal (cma_send_device_removal_put() sets DEVICE_REMOVAL under id_priv->lock without comparing the old state). The other ways to fail are ULP misuse (a concurrent destroy or a concurrent call on the same ID), and those still get -EINVAL returned to the caller, as before. Only the stack trace is dropped.

I can send a v2 that adds a comment explaining the race, and keeps the WARN_ON() for states other than DEVICE_REMOVAL if you or the maintainers prefer. Please let me know.

Thanks,
Deepanshu

#4 lore raw
Quoted 46 lines · Deepanshu Kartikey
Quoted 27 lines · Zhu Yanjun
Quoted 17 lines · Deepanshu Kartikey

rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.

syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:

    WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src
    Call Trace:
     rdma_resolve_addr
     rds_ib_conn_path_connect
     rds_connect_worker
Quoted 8 lines

This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

Quoted 8 lines

Resend this becase some error occurred.

For this specific case, we can consider this behavior expected rather than an error and remove the WARN_ON().

However, I'm not sure whether the WARN_ON() is still necessary for other test cases.

Hi Yanjun,

Thanks for the review. After cma_bind_addr() succeeds the state is ADDR_BOUND, and the only legitimate change before the second exchange is device removal (cma_send_device_removal_put() sets DEVICE_REMOVAL under id_priv->lock without comparing the old state). The other ways to fail are ULP misuse (a concurrent destroy or a concurrent call on the same ID), and those still get -EINVAL returned to the caller, as before. Only the stack trace is dropped.

Quoted 3 lines

I can send a v2 that adds a comment explaining the race, and keeps the WARN_ON() for states other than DEVICE_REMOVAL if you or the maintainers prefer. Please let me know.

I think sending a v2 would be better. It would be good to keep the WARN_ON()

check for other unexpected states and add a comment explaining why DEVICE_REMOVAL

is a legitimate state in this race condition. This preserves the existing error detection

while making the code easier to understand and maintain.

Yanjun Zhu

Quoted 3 lines

Thanks,
Deepanshu

Signature · 2 lines
Best Regards,
Yanjun.Zhu
#5 lore raw
In reply to Deepanshu Kartikey
Quoted 8 lines · Deepanshu Kartikey

rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.

Quoted 8 lines

syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:

  WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src
  Call Trace:
   rdma_resolve_addr
   rds_ib_conn_path_connect
   rds_connect_worker

Please never trim kernel dump stack and add full splat.

Quoted 6 lines

This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.

Why is this a valid race? The expectation is that when a device is removed, the ib_client is notified first and then deleted. RDS should stop all communication with the underlying device so it can be unregistered cleanly.

Quoted 3 lines

Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.

I think that it is not correct and the issue is in RDS.

Thanks

Quoted 28 lines
Reported-by [email protected]
Fixes 732d41c545bb ("RDMA/cma: Make the locking for automatic state transition more clear")
---
 drivers/infiniband/core/cma.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/core/cma.c +2 -2
index 337a49d1acf7..46741c1fd34b 100644
@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv,
 		ret = cma_bind_addr(&id_priv->id, src_addr, dst_addr);
 		if (ret)
 			return ret;
-		if (WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
-					   RDMA_CM_ADDR_QUERY)))
+		if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
+				   RDMA_CM_ADDR_QUERY))
 			return -EINVAL;
 
 	} else {
Signature · 1 line
2.43.0