RDMA/cma: Fix hang when cma_netevent_callback fails to queue_work
PATCH 1/1 RDMA/cma: Fix hang when cma_netevent_callback fails to queue_work
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when cma_netevent_callback was called for a cma_id while work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.
However, it left a problem when queue_work fails (because the item is still pending in the work queue from a previous call). In this case, cma_id_put (which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).
Fix this by calling cma_id_put() if queue_work fails.
--- drivers/infiniband/core/cma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cma.c +2 -1
@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self, neigh->ha, ETH_ALEN))continue ;cma_id_get (current_id); -queue_work (cma_wq, ¤t_id->id.net_work); +if (!queue_work (cma_wq, ¤t_id->id.net_work)) +cma_id_put (current_id); } out:spin_unlock_irqrestore (&id_table_lock, flags);
Signature · 1 line
2.49.0
Quoted 7 lines · Leon Romanovsky
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when
cma_netevent_callbackwas called for acma_idwhile work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.
Quoted 8 lines
However, it left a problem when
queue_workfails (because the item is still pending in the work queue from a previous call). In this case,cma_id_put(which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).Fix this by calling
cma_id_put()ifqueue_workfails.Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")
IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, ¤t_id->id.net_work);"
i.e. Fixes: 925d046e7e52 ("RDMA/core: Add a netevent notifier to cma")
and not another bug fix (45f5dcdd0497) which did not introduce the problem being described in this patch (a missing cma_id_put() when queue_work() fails).
Otherwise the fix looks good to me:
Thanks,
Sharath
Quoted 22 lines
Signed-off-by Jack Morgenstein <[email protected]>Signed-off-by Feng Liu <[email protected]>Reviewed-by Vlad Dumitrescu <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/infiniband/core/cma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cma.c +2
-1@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self, neigh->ha, ETH_ALEN))continue ;cma_id_get (current_id); -queue_work (cma_wq, ¤t_id->id.net_work); +if (!queue_work (cma_wq, ¤t_id->id.net_work)) +cma_id_put (current_id); } out:spin_unlock_irqrestore (&id_table_lock, flags);
Quoted 16 lines · Sharath Srinivasan
Quoted 14 lines · Leon Romanovsky
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when
cma_netevent_callbackwas called for acma_idwhile work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.However, it left a problem when
queue_workfails (because the item is still pending in the work queue from a previous call). In this case,cma_id_put(which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).Fix this by calling
cma_id_put()ifqueue_workfails.
Quoted 8 lines
Quoted 1 line
Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, ¤t_id->id.net_work);"i.e. Fixes:
925d046e7e52("RDMA/core: Add a netevent notifier to cma")and not another bug fix (
45f5dcdd0497) which did not introduce the problem being described in this patch (a missingcma_id_put()whenqueue_work()fails).
It is not, according to the queue_work() description and implementation, that function call can fail only if this work already exist. Before commit 45f5dcdd0497 that cma_netevent_work was always new and hence can't fail. This is why queue_work() returned value is almost never checked in the kernel.
Thanks
Quoted 30 lines
Otherwise the fix looks good to me:
Reviewed-by Sharath Srinivasan <[email protected]>Thanks,
SharathQuoted 22 lines
Signed-off-by Jack Morgenstein <[email protected]>Signed-off-by Feng Liu <[email protected]>Reviewed-by Vlad Dumitrescu <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/infiniband/core/cma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cma.c +2
-1@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self, neigh->ha, ETH_ALEN))continue ;cma_id_get (current_id); -queue_work (cma_wq, ¤t_id->id.net_work); +if (!queue_work (cma_wq, ¤t_id->id.net_work)) +cma_id_put (current_id); } out:spin_unlock_irqrestore (&id_table_lock, flags);
Quoted 24 lines · Leon Romanovsky
Quoted 23 lines · Sharath Srinivasan
Quoted 7 lines · Leon Romanovsky
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when
cma_netevent_callbackwas called for acma_idwhile work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.Quoted 8 lines
However, it left a problem when
queue_workfails (because the item is still pending in the work queue from a previous call). In this case,cma_id_put(which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).Fix this by calling
cma_id_put()ifqueue_workfails.Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")IMO the above Fixes: tag should point to the commit that introduced the line:
"queue_work(cma_wq, ¤t_id->id.net_work);"i.e. Fixes:
925d046e7e52("RDMA/core: Add a netevent notifier to cma")
Quoted 9 lines
Quoted 1 line
and not another bug fix (
45f5dcdd0497) which did not introduce the problem being described in this patch (a missingcma_id_put()whenqueue_work()fails).It is not, according to the
queue_work()description and implementation, that function call can fail only if this work already exist. Before commit45f5dcdd0497thatcma_netevent_workwas always new and hence can't fail. This is whyqueue_work()returned value is almost never checked in the kernel.Thanks
Thanks for clarifying. Makes sense to say "Fixes: 45f5dcdd0497".
Regards,
Sharath
Quoted 30 lines
Quoted 30 lines
Otherwise the fix looks good to me:
Reviewed-by Sharath Srinivasan <[email protected]>Thanks,
SharathQuoted 22 lines
Signed-off-by Jack Morgenstein <[email protected]>Signed-off-by Feng Liu <[email protected]>Reviewed-by Vlad Dumitrescu <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- drivers/infiniband/core/cma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cma.c +2
-1@@ -5245,7 +5245,8 @@ static int cma_netevent_callback(struct notifier_block *self, neigh->ha, ETH_ALEN))continue ;cma_id_get (current_id); -queue_work (cma_wq, ¤t_id->id.net_work); +if (!queue_work (cma_wq, ¤t_id->id.net_work)) +cma_id_put (current_id); } out:spin_unlock_irqrestore (&id_table_lock, flags);
Quoted 13 lines · Leon Romanovsky
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when
cma_netevent_callbackwas called for acma_idwhile work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.However, it left a problem when
queue_workfails (because the item is still pending in the work queue from a previous call). In this case,cma_id_put(which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).
Quoted 7 lines
Fix this by calling
cma_id_put()ifqueue_workfails.Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")Signed-off-by Jack Morgenstein <[email protected]>Signed-off-by Feng Liu <[email protected]>Reviewed-by Vlad Dumitrescu <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>
LGTM,
Signature · 2 lines
Regards, Kalesh AP
Quoted 21 lines · Leon Romanovsky
From: Jack Morgenstein <[email protected]>
The cited commit fixed a crash when
cma_netevent_callbackwas called for acma_idwhile work on that id from a previous call had not yet started. The work item was re-initialized in the second call, which corrupted the work item currently in the work queue.However, it left a problem when
queue_workfails (because the item is still pending in the work queue from a previous call). In this case,cma_id_put(which is called in the work handler) is therefore not called. This results in a userspace process hang (zombie process).Fix this by calling
cma_id_put()ifqueue_workfails.Fixes 45f5dcdd0497 ("RDMA/cma: Fix workqueue crash in cma_netevent_work_handler")Signed-off-by Jack Morgenstein <[email protected]>Signed-off-by Feng Liu <[email protected]>Reviewed-by Vlad Dumitrescu <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>Reviewed-by Sharath Srinivasan <[email protected]>Reviewed-by Kalesh AP <[email protected]>
Quoted 3 lines
--- drivers/infiniband/core/cma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Applied to for-next, thanks
Jason