RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()
PATCH 1/1 RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()
rdma_resolve_addr() on an unbound ID (RDMA_CM_IDLE) first binds it with cma_bind_addr() and then moves it from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY with a second cma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, so cma_send_device_removal_put() can find it and set RDMA_CM_DEVICE_REMOVAL before the second step runs. The exchange then fails and hits the WARN_ON() added by the commit below.
syzbot triggers this with RDS over rxe, where rds_connect_worker() calls rdma_resolve_addr() while the rxe device is being deleted:
WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src Call Trace: rdma_resolve_addr rds_ib_conn_path_connect rds_connect_worker
This is a valid race and not a kernel bug. If removal wins before the bind, cma_bind_addr() already fails and the error is returned to the caller, and err_state in the same function already ignores a failed state exchange. The caller gets -EINVAL, and RDS destroys the ID on any error from rdma_resolve_addr(), so nothing is leaked.
Drop the WARN_ON() and keep returning -EINVAL, and document in a comment that the state may change because of device removal.
--- drivers/infiniband/core/cma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/core/cma.c +2 -2
@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv, ret =cma_bind_addr (&id_priv->id, src_addr, dst_addr);if (ret)return ret; -if (WARN_ON (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , -RDMA_CM_ADDR_QUERY ))) +if (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , +RDMA_CM_ADDR_QUERY ))return -EINVAL ; }else {
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] RDMA/cma: missing comment in resolve_prepare_src() promised by commit message --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] RDMA/cma: missing comment in resolve_prepare_src() promised by commit message --
commit b9b0fc41a9bfe112cf7965b88795f6ea037bc2e0
Author: Deepanshu Kartikey <[email protected]>
RDMA/cma: Don't warn when device removal races with rdma_resolve_addr()
This commit removes the WARN_ON() in resolve_prepare_src() when transitioning an RDMA CM ID from RDMA_CM_ADDR_BOUND to RDMA_CM_ADDR_QUERY after cma_bind_addr(), as a concurrent device removal can legitimately change the state to RDMA_CM_DEVICE_REMOVAL.
Quoted 2 lines
Drop the
WARN_ON()and keep returning-EINVAL, and document in a comment that the state may change because of device removal.
[ ... ]
drivers/infiniband/core/cma.c static int resolve_prepare_src(struct rdma_id_private *id_priv, · line 4122
Quoted 13 lines
drivers/infiniband/core/cma.c +2
-2@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv, ret =cma_bind_addr (&id_priv->id, src_addr, dst_addr);if (ret)return ret; -if (WARN_ON (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , -RDMA_CM_ADDR_QUERY ))) +if (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , +RDMA_CM_ADDR_QUERY ))return -EINVAL ;
[Severity: Low]
This isn't a bug, but the commit message mentions documenting in a comment that the state may change because of device removal, whereas the diff in resolve_prepare_src() in drivers/infiniband/core/cma.c only removes the WARN_ON() around cma_comp_exch(). Was that comment accidentally left out of the patch, or should the commit message be updated?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 17 lines · Deepanshu Kartikey
rdma_resolve_addr()on an unbound ID (RDMA_CM_IDLE) first binds it withcma_bind_addr()and then moves it fromRDMA_CM_ADDR_BOUNDtoRDMA_CM_ADDR_QUERYwith a secondcma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, socma_send_device_removal_put()can find it and setRDMA_CM_DEVICE_REMOVALbefore the second step runs. The exchange then fails and hits theWARN_ON()added by the commit below.syzbot triggers this with RDS over rxe, where
rds_connect_worker()callsrdma_resolve_addr()while the rxe device is being deleted:WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src Call Trace: rdma_resolve_addr rds_ib_conn_path_connect rds_connect_worker
Quoted 8 lines
This is a valid race and not a kernel bug. If removal wins before the bind,
cma_bind_addr()already fails and the error is returned to the caller, anderr_statein the same function already ignores a failed state exchange. The caller gets-EINVAL, and RDS destroys the ID on any error fromrdma_resolve_addr(), so nothing is leaked.Drop the
WARN_ON()and keep returning-EINVAL, and document in a comment that the state may change because of device removal.
Resend this becase some error occurred.
For this specific case, we can consider this behavior expected rather than an error and remove the WARN_ON().
However, I'm not sure whether the WARN_ON() is still necessary for other test cases.
Yanjun Zhu
Quoted 25 lines
Reported-by [email protected]Fixes 732d41c545bb ("RDMA/cma: Make the locking for automatic state transition more clear")Tested-by [email protected]Signed-off-by Deepanshu Kartikey <[email protected]>--- drivers/infiniband/core/cma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/infiniband/core/cma.c +2
-2@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv, ret =cma_bind_addr (&id_priv->id, src_addr, dst_addr);if (ret)return ret; -if (WARN_ON (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , -RDMA_CM_ADDR_QUERY ))) +if (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , +RDMA_CM_ADDR_QUERY ))return -EINVAL ; }else {
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 29 lines · Zhu Yanjun
Quoted 17 lines · Deepanshu Kartikey
rdma_resolve_addr()on an unbound ID (RDMA_CM_IDLE) first binds it withcma_bind_addr()and then moves it fromRDMA_CM_ADDR_BOUNDtoRDMA_CM_ADDR_QUERYwith a secondcma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, socma_send_device_removal_put()can find it and setRDMA_CM_DEVICE_REMOVALbefore the second step runs. The exchange then fails and hits theWARN_ON()added by the commit below.syzbot triggers this with RDS over rxe, where
rds_connect_worker()callsrdma_resolve_addr()while the rxe device is being deleted:WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src Call Trace: rdma_resolve_addr rds_ib_conn_path_connect rds_connect_workerQuoted 8 lines
This is a valid race and not a kernel bug. If removal wins before the bind,
cma_bind_addr()already fails and the error is returned to the caller, anderr_statein the same function already ignores a failed state exchange. The caller gets-EINVAL, and RDS destroys the ID on any error fromrdma_resolve_addr(), so nothing is leaked.Drop the
WARN_ON()and keep returning-EINVAL, and document in a comment that the state may change because of device removal.
Quoted 8 lines
Resend this becase some error occurred.
For this specific case, we can consider this behavior expected rather than an error and remove the
WARN_ON().However, I'm not sure whether the
WARN_ON()is still necessary for other test cases.
Hi Yanjun,
Thanks for the review. After cma_bind_addr() succeeds the state is ADDR_BOUND, and the only legitimate change before the second exchange is device removal (cma_send_device_removal_put() sets DEVICE_REMOVAL under id_priv->lock without comparing the old state). The other ways to fail are ULP misuse (a concurrent destroy or a concurrent call on the same ID), and those still get -EINVAL returned to the caller, as before. Only the stack trace is dropped.
I can send a v2 that adds a comment explaining the race, and keeps the WARN_ON() for states other than DEVICE_REMOVAL if you or the maintainers prefer. Please let me know.
Thanks,
Deepanshu
Quoted 46 lines · Deepanshu Kartikey
Quoted 27 lines · Zhu Yanjun
Quoted 17 lines · Deepanshu Kartikey
rdma_resolve_addr()on an unbound ID (RDMA_CM_IDLE) first binds it withcma_bind_addr()and then moves it fromRDMA_CM_ADDR_BOUNDtoRDMA_CM_ADDR_QUERYwith a secondcma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, socma_send_device_removal_put()can find it and setRDMA_CM_DEVICE_REMOVALbefore the second step runs. The exchange then fails and hits theWARN_ON()added by the commit below.syzbot triggers this with RDS over rxe, where
rds_connect_worker()callsrdma_resolve_addr()while the rxe device is being deleted:WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src Call Trace: rdma_resolve_addr rds_ib_conn_path_connect rds_connect_workerQuoted 8 lines
This is a valid race and not a kernel bug. If removal wins before the bind,
cma_bind_addr()already fails and the error is returned to the caller, anderr_statein the same function already ignores a failed state exchange. The caller gets-EINVAL, and RDS destroys the ID on any error fromrdma_resolve_addr(), so nothing is leaked.Drop the
WARN_ON()and keep returning-EINVAL, and document in a comment that the state may change because of device removal.Quoted 8 lines
Resend this becase some error occurred.
For this specific case, we can consider this behavior expected rather than an error and remove the
WARN_ON().However, I'm not sure whether the
WARN_ON()is still necessary for other test cases.Hi Yanjun,
Thanks for the review. After
cma_bind_addr()succeeds the state isADDR_BOUND, and the only legitimate change before the second exchange is device removal (cma_send_device_removal_put()setsDEVICE_REMOVALunderid_priv->lockwithout comparing the old state). The other ways to fail are ULP misuse (a concurrent destroy or a concurrent call on the same ID), and those still get-EINVALreturned to the caller, as before. Only the stack trace is dropped.
Quoted 3 lines
I can send a v2 that adds a comment explaining the race, and keeps the
WARN_ON()for states other thanDEVICE_REMOVALif you or the maintainers prefer. Please let me know.
I think sending a v2 would be better. It would be good to keep the WARN_ON()
check for other unexpected states and add a comment explaining why DEVICE_REMOVAL
is a legitimate state in this race condition. This preserves the existing error detection
while making the code easier to understand and maintain.
Yanjun Zhu
Quoted 3 lines
Thanks,
Deepanshu
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 8 lines · Deepanshu Kartikey
rdma_resolve_addr()on an unbound ID (RDMA_CM_IDLE) first binds it withcma_bind_addr()and then moves it fromRDMA_CM_ADDR_BOUNDtoRDMA_CM_ADDR_QUERYwith a secondcma_comp_exch(). These are two separate steps. Once the ID is bound it is attached to the device, socma_send_device_removal_put()can find it and setRDMA_CM_DEVICE_REMOVALbefore the second step runs. The exchange then fails and hits theWARN_ON()added by the commit below.
Quoted 8 lines
syzbot triggers this with RDS over rxe, where
rds_connect_worker()callsrdma_resolve_addr()while the rxe device is being deleted:WARNING: drivers/infiniband/core/cma.c:4120 at resolve_prepare_src Call Trace: rdma_resolve_addr rds_ib_conn_path_connect rds_connect_worker
Please never trim kernel dump stack and add full splat.
Quoted 6 lines
This is a valid race and not a kernel bug. If removal wins before the bind,
cma_bind_addr()already fails and the error is returned to the caller, anderr_statein the same function already ignores a failed state exchange. The caller gets-EINVAL, and RDS destroys the ID on any error fromrdma_resolve_addr(), so nothing is leaked.
Why is this a valid race? The expectation is that when a device is removed, the ib_client is notified first and then deleted. RDS should stop all communication with the underlying device so it can be unregistered cleanly.
Quoted 3 lines
Drop the
WARN_ON()and keep returning-EINVAL, and document in a comment that the state may change because of device removal.
I think that it is not correct and the issue is in RDS.
Thanks
Quoted 28 lines
Reported-by [email protected]Fixes 732d41c545bb ("RDMA/cma: Make the locking for automatic state transition more clear")Tested-by [email protected]Signed-off-by Deepanshu Kartikey <[email protected]>--- drivers/infiniband/core/cma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/infiniband/core/cma.c +2
-2@@ -4117,8 +4117,8 @@ static int resolve_prepare_src(struct rdma_id_private *id_priv, ret =cma_bind_addr (&id_priv->id, src_addr, dst_addr);if (ret)return ret; -if (WARN_ON (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , -RDMA_CM_ADDR_QUERY ))) +if (!cma_comp_exch (id_priv,RDMA_CM_ADDR_BOUND , +RDMA_CM_ADDR_QUERY ))return -EINVAL ; }else {Signature · 1 line
2.43.0