workqueue: ensure all flush_work() completed when being destoryed
PATCH 1/1 workqueue: ensure all flush_work() completed when being destoryed
In old days, worker threads are not shared among different workqueues and destroy_workqueue() used kthread_stop() to destroy all workers before going to destroy workqueue structures. And kthread_stop() can ensure the scheduled (worker->scheduled) work items and the linked work items queued by flush_work() to be completed.
For a workqueue to be completed/unused for wq users means that all queued works have completed and all flush_work() have return, and the workqueue is legitimate to passed to destroy_workqueue().
Bute22bee782b3b("workqueue: implement concurrency managed dynamic worker pool") made worker pools and workers shared among different
workqueues and kthread_stop() is not used to sync the completion of the work items. destroy_workqueue() uses drain_workqueue()
to drain user work items, but internal work items queued by flush_work() is not drained due to they don't have colors.
So problems may occur when wq_barrier_func() does complete(&barr->done) and the wokenup wq-user code does destroy_workqueue(). destroy_workqueue() can be scheduled eariler than the proccess_one_work() to do the put_pwq(), so that the sanity check in destroy_workqueue() can see the no yet put pwq->refcnt and blame false positively.
The problem can be easily fixed by removing the WORK_NO_COLOR and making the internal work item queued by flush_work() inherit the color of the work items to be flushed. It would definitely revert the design and the benefits of the WORK_NO_COLOR.
The color-based flush_workqueue() was designed so well that we can also easily to adapt it to flush WORK_NO_COLOR. This is the approach taken by this patch which introduces a flush_no_color() to flush all the WORK_NO_COLOR works.
--- include/linux/workqueue.h | 5 +-- kernel/workqueue.c | 72 +++++++++++++++++++++++++++++++++------ 2 files changed, 64 insertions(+), 13 deletions(-)
include/linux/workqueue.h +3 -2
@@ -55,8 +55,9 @@ enum {* The last color is no color used for works which don't * participate in workqueue flushing. */ -WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ) -1 , -WORK_NO_COLOR =WORK_NR_COLORS , +WORK_NR_COLORS = (1 <<WORK_STRUCT_COLOR_BITS ), +WORK_NR_FLUSH_COLORS = (WORK_NR_COLORS -1 ), +WORK_NO_COLOR =WORK_NR_FLUSH_COLORS ,/* not bound to any CPU, prefer the local CPU */ WORK_CPU_UNBOUND = NR_CPUS,
kernel/workqueue.c +61 -11
105 lines not on this page · raw message
Signature · 1 line
2.20.1
Tejun Heo
maintainerHello, Lai.
kernel/workqueue.c void flush_workqueue(struct workqueue_struct *wq) · line 2923
Quoted 2 lines · Lai Jiangshan
kernel/workqueue.c +2
-0+static void flush_no_color (struct workqueue_struct *wq) +{
...
I'm not too sure about using the colored flushing system for this. Given that the requirements are a lot simpler, I'd prefer keep it separate and dumb rather than intertwining it with regular flushes. e.g. each wq can keep the number of flush work items in flight and a simple wake up mechanism for the task doing destroy_workqueue().
Thanks.
Signature · 1 line
tejun
PATCH 1/1 workqueue: ensure all flush_work() completed when being destoryed
In reply to Lai JiangshanIn old days, worker threads are not shared among different workqueues and destroy_workqueue() used kthread_stop() to destroy all workers before going to destroy workqueue structures. And kthread_stop() can ensure the scheduled (worker->scheduled) work items and the linked work items queued by flush_work() to be completed.
For a workqueue to be completed/unused for wq users means that all queued works have completed and all flush_work() have return, and the workqueue is legitimate to passed to destroy_workqueue().
Bute22bee782b3b("workqueue: implement concurrency managed dynamic worker pool") made worker pools and workers shared among different
workqueues and kthread_stop() is not used to sync the completion of the work items. destroy_workqueue() uses drain_workqueue()
to drain user work items, but internal work items queued by flush_work() is not drained due to they don't have colors.
So problems may occur when wq_barrier_func() does complete(&barr->done) and the wokenup wq-user code does destroy_workqueue(). destroy_workqueue() can be scheduled eariler than the proccess_one_work() to do the put_pwq(), so that the sanity check in destroy_workqueue() can see the no yet put pwq->refcnt and blame false positively.
The problem can be easily fixed by removing the WORK_NO_COLOR and making the internal work item queued by flush_work() inherit the color of the work items to be flushed. It would definitely revert the design and the benefits of the WORK_NO_COLOR.
The patch simply adds an atomic counter for in-flight flush_work() and a completion for destroy_workqueue() waiting for them.
---
Changed from V1:
Change from flush_no_color based mechanism to atomic+completion based as TJ suggested.
kernel/workqueue.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
kernel/workqueue.c +20 -1
@@ -253,6 +253,9 @@ struct workqueue_struct {int nr_drainers;/* WQ: drain in progress */ int saved_max_active;/* WQ: saved pwq max_active */ +atomic_t nr_flush_work;/* flush work in progress */ +struct completion flush_work_done;/* sync flush_work() */ +struct workqueue_attrs *unbound_attrs;/* PW: only for unbound wqs */ struct pool_workqueue *dfl_pwq;/* PW: only for unbound wqs */
@@ -1154,6 +1157,12 @@ static void pwq_activate_first_delayed(struct pool_workqueue *pwq)pwq_activate_delayed_work (work); } +static void dec_nr_in_flight_flush_work (struct workqueue_struct *wq) +{ +if (atomic_dec_and_test (&wq->nr_flush_work)) +complete (&wq->flush_work_done); +} +/** * pwq_dec_nr_in_flight - decrement pwq's nr_in_flight * @pwq: pwq of interest
@@ -1168,8 +1177,10 @@ static void pwq_activate_first_delayed(struct pool_workqueue *pwq)static void pwq_dec_nr_in_flight (struct pool_workqueue *pwq,int color) {/* uncolored work items don't participate in flushing or nr_active */ -if (color ==WORK_NO_COLOR ) +if (color ==WORK_NO_COLOR ) { +dec_nr_in_flight_flush_work (pwq->wq); goto out_put; + } pwq->nr_in_flight[color]--;
@@ -2682,6 +2693,7 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, }debug_work_activate (&barr->work); +atomic_inc (&pwq->wq->nr_flush_work);insert_work (pwq, &barr->work, head,work_color_to_flags (WORK_NO_COLOR ) | linked); }
@@ -4278,6 +4290,9 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,wq_init_lockdep (wq);INIT_LIST_HEAD (&wq->list); +atomic_set (&wq->nr_flush_work,1 ); +init_completion (&wq->flush_work_done); +if (alloc_and_link_pwqs (wq) <0 ) goto err_unreg_lockdep;
@@ -4354,6 +4369,10 @@ void destroy_workqueue(struct workqueue_struct *wq)/* drain it before proceeding with destruction */ drain_workqueue (wq); +/* flush all uncompleted internal work items queued by flush_work() */ +dec_nr_in_flight_flush_work (wq); +wait_for_completion (&wq->flush_work_done); +/* kill rescuer, if sanity checks fail, leave it w/o rescuer */ if (wq->rescuer) {struct worker *rescuer = wq->rescuer;
Signature · 1 line
2.20.1
Tejun Heo
maintainerHello, Lai.
kernel/workqueue.c static void pwq_activate_first_delayed(struct pool_workqueue *pwq) · line 1162
Quoted 3 lines · Lai Jiangshan
kernel/workqueue.c +3
-0+static void dec_nr_in_flight_flush_work (struct workqueue_struct *wq) +{ +if (atomic_dec_and_test (&wq->nr_flush_work))
Do you think it'd make sense to put this in pwq so that it can be synchronized with the pool lock instead of using a separate atomic counter?
Makes sense to me otherwise.
Thanks.
Signature · 1 line
tejun
Lai Jiangshan
reviewerQuoted 11 lines · Tejun Heo
Hello, Lai.
kernel/workqueue.c static void pwq_activate_first_delayed(struct pool_workqueue *pwq) · line 1162
Quoted 3 lines · Lai Jiangshan
kernel/workqueue.c +3
-0+static void dec_nr_in_flight_flush_work (struct workqueue_struct *wq) +{ +if (atomic_dec_and_test (&wq->nr_flush_work))Do you think it'd make sense to put this in pwq so that it can be synchronized with the pool lock instead of using a separate atomic counter?
Hello, Tejun
When the counter is put in pwq, there will be a per-pwq counter, a per-pwq completion or pointer and a flush_workqueue_prep_pwqs()-like function (although simpler) to set up them before waiting for them.
It would sound like the V1 patch. per-pwq counter may have better performance over per-wq atomic, it seems too tiny to add code complicity. V1 has the simplest pwq_dec_nr_in_flight() except there is too much WARN_ON_ONCE() in flush_no_color().
Thanks
Lai
Quoted 6 lines
Makes sense to me otherwise.
Thanks.
--
tejun