Skip to contentSkip to conversation

RDMA/cma: Fix hang when cma_netevent_callback fails to queue_work

PATCH Applied Leon Romanovsky 4 participants 6 messages May 21, 2025 – May 26, 2025 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/cma: Fix hang when cma_netevent_callback fails to queue_work

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Fix this by calling cma_id_put() if queue_work fails.

Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")
---
 drivers/infiniband/core/cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cma.c +2 -1
index ab31eefa916b3..274cfbd5aaba7 100644
@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self,
 			   neigh->ha, ETH_ALEN))
 			continue;
 		cma_id_get(current_id);
-		queue_work(cma_wq, &current_id->id.net_work);
+		if (!queue_work(cma_wq, &current_id->id.net_work))
+			cma_id_put(current_id);
 	}
 out:
 	spin_unlock_irqrestore(&id_table_lock, flags);
Signature · 1 line
2.49.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 7 lines · Leon Romanovsky

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

Quoted 8 lines

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Fix this by calling cma_id_put() if queue_work fails.

Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")

IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, &current_id->id.net_work);"

i.e. Fixes: 925d046e7e52 ("RDMA/core: Add a netevent notifier to cma")

and not another bug fix (45f5dcdd0497) which did not introduce the problem being described in this patch (a missing cma_id_put() when queue_work() fails).

Otherwise the fix looks good to me:

Thanks,
Sharath

Quoted 22 lines
---
 drivers/infiniband/core/cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cma.c +2 -1
index ab31eefa916b3..274cfbd5aaba7 100644
@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self,
 			   neigh->ha, ETH_ALEN))
 			continue;
 		cma_id_get(current_id);
-		queue_work(cma_wq, &current_id->id.net_work);
+		if (!queue_work(cma_wq, &current_id->id.net_work))
+			cma_id_put(current_id);
 	}
 out:
 	spin_unlock_irqrestore(&id_table_lock, flags);
#2 lore raw
Quoted 16 lines · Sharath Srinivasan
Quoted 14 lines · Leon Romanovsky

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Fix this by calling cma_id_put() if queue_work fails.

Quoted 8 lines
Quoted 1 line
Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")

IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, &current_id->id.net_work);"

i.e. Fixes: 925d046e7e52 ("RDMA/core: Add a netevent notifier to cma")

and not another bug fix (45f5dcdd0497) which did not introduce the problem being described in this patch (a missing cma_id_put() when queue_work() fails).

It is not, according to the queue_work() description and implementation, that function call can fail only if this work already exist. Before commit 45f5dcdd0497 that cma_netevent_work was always new and hence can't fail. This is why queue_work() returned value is almost never checked in the kernel.

Thanks

Quoted 30 lines

Otherwise the fix looks good to me:

Thanks,
Sharath

Quoted 22 lines
---
 drivers/infiniband/core/cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cma.c +2 -1
index ab31eefa916b3..274cfbd5aaba7 100644
@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self,
 			   neigh->ha, ETH_ALEN))
 			continue;
 		cma_id_get(current_id);
-		queue_work(cma_wq, &current_id->id.net_work);
+		if (!queue_work(cma_wq, &current_id->id.net_work))
+			cma_id_put(current_id);
 	}
 out:
 	spin_unlock_irqrestore(&id_table_lock, flags);
#3 lore raw
Quoted 24 lines · Leon Romanovsky
Quoted 23 lines · Sharath Srinivasan
Quoted 7 lines · Leon Romanovsky

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

Quoted 8 lines

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Fix this by calling cma_id_put() if queue_work fails.

Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")

IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, &current_id->id.net_work);"

i.e. Fixes: 925d046e7e52 ("RDMA/core: Add a netevent notifier to cma")

Quoted 9 lines
Quoted 1 line

and not another bug fix (45f5dcdd0497) which did not introduce the problem being described in this patch (a missing cma_id_put() when queue_work() fails).

It is not, according to the queue_work() description and implementation, that function call can fail only if this work already exist. Before commit 45f5dcdd0497 that cma_netevent_work was always new and hence can't fail. This is why queue_work() returned value is almost never checked in the kernel.

Thanks

Thanks for clarifying. Makes sense to say "Fixes: 45f5dcdd0497".

Regards,
Sharath

Quoted 30 lines
Quoted 30 lines

Otherwise the fix looks good to me:

Thanks,
Sharath

Quoted 22 lines
---
 drivers/infiniband/core/cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cma.c +2 -1
index ab31eefa916b3..274cfbd5aaba7 100644
@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self,
 			   neigh->ha, ETH_ALEN))
 			continue;
 		cma_id_get(current_id);
-		queue_work(cma_wq, &current_id->id.net_work);
+		if (!queue_work(cma_wq, &current_id->id.net_work))
+			cma_id_put(current_id);
 	}
 out:
 	spin_unlock_irqrestore(&id_table_lock, flags);
Reviewed-by on v1 1/1
#4 lore raw
In reply to Leon Romanovsky
Quoted 13 lines · Leon Romanovsky

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Quoted 7 lines

Fix this by calling cma_id_put() if queue_work fails.

Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")

LGTM,

Signature · 2 lines
Regards,
Kalesh AP
maintainer Applied
#5 lore raw
In reply to Leon Romanovsky
Quoted 21 lines · Leon Romanovsky

From: Jack Morgenstein <[email protected]>

The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.

However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).

Fix this by calling cma_id_put() if queue_work fails.

Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")
Quoted 3 lines
---
 drivers/infiniband/core/cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Applied to for-next, thanks

Jason