workqueue: Fix spurious sanity check failures in destroy_workqueue()
Tejun Heo
maintainerPATCH 1/1 workqueue: Fix spurious sanity check failures in destroy_workqueue()
Before actually destrying a workqueue, destroy_workqueue() checks whether it's actually idle. If it isn't, it prints out a bunch of warning messages and leaves the workqueue dangling. It unfortunately has a couple issues.
* Mayday list queueing increments pwq's refcnts which gets detected as busy and fails the sanity checks. However, because mayday list queueing is asynchronous, this condition can happen without any actual work items left in the workqueue.
* Sanity check failure leaves the sysfs interface behind too which can lead to init failure of newer instances of the workqueue.
This patch fixes the above two by
* If a workqueue has a rescuer, disable and kill the rescuer before sanity checks. Disabling and killing is guaranteed to flush the existing mayday list.
* Remove sysfs interface before sanity checks.
---
Applying to wq/for-5.4.
Thanks.
kernel/workqueue.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)
kernel/workqueue.c +19 -5
@@ -4318,9 +4318,28 @@ void destroy_workqueue(struct workqueue_struct *wq)struct pool_workqueue *pwq;int node; +/* +* Remove it from sysfs first so that sanity check failure doesn't +* lead to sysfs name conflicts. +*/ +workqueue_sysfs_unregister (wq); +/* drain it before proceeding with destruction */ drain_workqueue (wq); +/* kill rescuer, if sanity checks fail, leave it w/o rescuer */ +if (wq->rescuer) { +struct worker *rescuer = wq->rescuer; + +/* this prevents new queueing */ +spin_lock_irq (&wq_mayday_lock); + wq->rescuer =NULL ; +spin_unlock_irq (&wq_mayday_lock); + +/* rescuer will empty maydays list before exiting */ +kthread_stop (rescuer->task); + } +/* sanity checks */ mutex_lock (&wq->mutex);for_each_pwq (pwq, wq) {
@@ -4352,11 +4371,6 @@ void destroy_workqueue(struct workqueue_struct *wq)list_del_rcu (&wq->list);mutex_unlock (&wq_pool_mutex); -workqueue_sysfs_unregister (wq); - -if (wq->rescuer) -kthread_stop (wq->rescuer->task); -if (!(wq->flags &WQ_UNBOUND )) {wq_unregister_lockdep (wq);/*
Signature · 1 line
2.17.1
Looks good to me.
There is one test in show_pwq()
"""
worker == pwq->wq->rescuer ? "(RESCUER)" : "",
"""
I'm wondering if it needs to be updated to
"""worker->rescue_wq ? "(RESCUER)" : "",
"""
And document "/* MD: rescue worker */" might be better than current "/* I: rescue worker */", although ->rescuer can be accessed without wq_mayday_lock lock in some code.
Quoted 82 lines · Tejun Heo
Before actually destrying a workqueue,
destroy_workqueue()checks whether it's actually idle. If it isn't, it prints out a bunch of warning messages and leaves the workqueue dangling. It unfortunately has a couple issues.* Mayday list queueing increments pwq's refcnts which gets detected as busy and fails the sanity checks. However, because mayday list queueing is asynchronous, this condition can happen without any actual work items left in the workqueue.
* Sanity check failure leaves the sysfs interface behind too which can lead to init failure of newer instances of the workqueue.
This patch fixes the above two by
* If a workqueue has a rescuer, disable and kill the rescuer before sanity checks. Disabling and killing is guaranteed to flush the existing mayday list.
* Remove sysfs interface before sanity checks.
Signed-off-by Tejun Heo <[email protected]>Reported-by Marcin Pawlowski <[email protected]>Reported-by "Williams, Gerald S" <[email protected]>---
Applying to wq/for-5.4.Thanks.
kernel/workqueue.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)kernel/workqueue.c +19
-6@@ -4318,9 +4318,28 @@ void destroy_workqueue(struct workqueue_struct *wq)struct pool_workqueue *pwq;int node; +/* +* Remove it from sysfs first so that sanity check failure doesn't +* lead to sysfs name conflicts. +*/ +workqueue_sysfs_unregister (wq); +/* drain it before proceeding with destruction */ drain_workqueue (wq); +/* kill rescuer, if sanity checks fail, leave it w/o rescuer */ +if (wq->rescuer) { +struct worker *rescuer = wq->rescuer; + +/* this prevents new queueing */ +spin_lock_irq (&wq_mayday_lock); + wq->rescuer =NULL ; +spin_unlock_irq (&wq_mayday_lock); + +/* rescuer will empty maydays list before exiting */ +kthread_stop (rescuer->task); + } +/* sanity checks */ mutex_lock (&wq->mutex);for_each_pwq (pwq, wq) {@@ -4352,11 +4371,6 @@ void destroy_workqueue(struct workqueue_struct *wq)list_del_rcu (&wq->list);mutex_unlock (&wq_pool_mutex); -workqueue_sysfs_unregister (wq); - -if (wq->rescuer) -kthread_stop (wq->rescuer->task); -if (!(wq->flags &WQ_UNBOUND )) {wq_unregister_lockdep (wq);/* -- 2.17.1
Tejun Heo
maintainerHello,
Quoted 10 lines · Lai Jiangshan
Looks good to me.
There is one test in
show_pwq()
"""
worker ==pwq->wq->rescuer? "(RESCUER)" : "",
"""
I'm wondering if it needs to be updated to
"""worker->rescue_wq? "(RESCUER)" : "",
"""
Hmm... yeah, good point. Let's do that.
Quoted 3 lines
And document "/* MD: rescue worker */" might be better than current "/* I: rescue worker */", although ->rescuer can be accessed without
wq_mayday_locklock in some code.
Will apply this one too.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 1/1 workqueue: Minor follow-ups to the rescuer destruction change
From 30ae2fc0a75eb5f1a38bbee7355d8e3bc823a6e1 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Fri, 20 Sep 2019 14:09:14 -0700
* Now that wq->rescuer may be cleared while rescuer is still there, switch show_pwq() debug printout to test worker->rescue_wq to identify rescuers intead of testing wq->rescuer.
* Update comment on ->rescuer locking.
---
Applied to workqueue/for-5.4-fixes.
Thanks.
kernel/workqueue.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/workqueue.c +2 -2
@@ -248,7 +248,7 @@ struct workqueue_struct {struct list_head flusher_overflow;/* WQ: flush overflow list */ struct list_head maydays;/* MD: pwqs requesting rescue */ -struct worker *rescuer;/* I: rescue worker */ +struct worker *rescuer;/* MD: rescue worker */ int nr_drainers;/* WQ: drain in progress */ int saved_max_active;/* WQ: saved pwq max_active */
@@ -4672,7 +4672,7 @@ static void show_pwq(struct pool_workqueue *pwq)pr_cont ("%s %d%s:%ps" , comma ?"," :"" ,task_pid_nr (worker->task), - worker == pwq->wq->rescuer ?"(RESCUER)" :"" , + worker->rescue_wq ?"(RESCUER)" :"" , worker->current_func);list_for_each_entry (work, &worker->scheduled, entry)pr_cont_work (false, work);
Signature · 1 line
2.17.1