workqueue: Implement system-wide max_active for unbound workqueues
Tejun Heo
maintainerPATCH v3 1/1 workqueue: Implement system-wide max_active for unbound workqueues
Hello,
This is v3. Changes from v2
(http://lkml.kernel.org/r/[email protected]):
- Per-node max_active is now pre-computed and stored in wq_node_nr_active->max. It also calculates max_active based on the intersection of online CPUs and the workqueue's effective CPUs to distribute CPUs fairly on workqueues with a cpumask which is not balanced across nodes. (Lai)
- wq_dump.py is updated to print out per-node nr/max_active counters.
- wq->node_nr_active is now a flex array. (Lai)
v1(http://lkml.kernel.org/r/[email protected]) -> v2:
- wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai.
- __queue_work() is updated to alwyas delay the work item if there already are inactive work items on the pwq. This prevents work item reordering inside the pwq when max_active is increased thus maintaining execution order for ordered workqueues. This issue was noticed by Lai.
- In 0008-workqueue-Introduce-struct-wq_node_nr_active.patch, Lai pointed out that pwq_tryinc_nr_active() incorrectly dropped pwq->max_active check. Restored. As the next patch replaces the max_active enforcement mechanism, this doesn't change the end result.
- 0010-workqueue-Reimplement-ordered-workqueue-using-shared.patch was broken and could reorder work items in ordered workqueues leading to severe perf regressions and hangs with certain workloads. Dropped.
A pool_workqueue (pwq) represents the connection between a workqueue and a worker_pool. One of the roles that a pwq plays is enforcement of the max_active concurrency limit. Before 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues"), there was one pwq per each CPU for per-cpu workqueues and per each NUMA node for unbound workqueues, which was a natural result of per-cpu workqueues being served by per-cpu pools and unbound by per-NUMA pools.
In terms of max_active enforcement, this was, while not perfect, workable. For per-cpu workqueues, it was fine. For unbound, it wasn't great in that NUMA machines would get max_active that's multiplied by the number of nodes but didn't cause huge problems because NUMA machines are relatively rare and the node count is usually pretty low.
However, cache layouts are more complex now and sharing a worker pool across a whole node didn't really work well for unbound workqueues. Thus, a series of commits culminating on 8639ecebc9b1 ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") implemented more flexible affinity mechanism for unbound workqueues which enables using e.g. last-level-cache aligned pools. In the process, 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") made unbound workqueues use per-cpu pwqs like per-cpu workqueues.
While the change was necessary to enable more flexible affinity scopes, this came with the side effect of blowing up the effective max_active for unbound workqueues. Before, the effective max_active for unbound workqueues was multiplied by the number of nodes. After, by the number of CPUs.
636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") claims that this should generally be okay. It is okay for users which self-regulates concurrency level which are the vast majority; however, there are enough use cases which actually depend on max_active to prevent the level of concurrency from going bonkers including several IO handling workqueues that can issue a work item for each in-flight IO. With targeted benchmarks, the misbehavior can easily be exposed as reported in
http://lkml.kernel.org/r/dbu6wiwu3sdhmhikb2w6lns7b27gbobfavhjj57kwi2quafgwl@htjcc5oikcr3.
Unfortunately, there is no way to express what these use cases need using per-cpu max_active. A CPU may issue most of in-flight IOs, so we don't want to set max_active too low but as soon as we increase max_active a bit, we can end up with unreasonable number of in-flight work items when many CPUs issue IOs at the same time. ie. The acceptable lowest max_active is higher than the acceptable highest max_active.
Ideally, max_active for an unbound workqueue should be system-wide so that the users can regulate the total level of concurrency regardless of node and cache layout. The reasons workqueue hasn't implemented that yet are:
- One max_active enforcement decouples from pool boundaires, chaining execution after a work item finishes requires inter-pool operations which would require lock dancing, which is nasty.
- Sharing a single nr_active count across the whole system can be pretty expensive on NUMA machines.
- Per-pwq enforcement had been more or less okay while we were using per-node pools.
It looks like we no longer can avoid decoupling max_active enforcement from pool boundaries. This patchset implements system-wide nr_active mechanism with the following design characteristics:
- To avoid sharing a single counter across multiple nodes, the configured max_active is split across nodes according to the proportion of online CPUs per node. e.g. A node with twice more online CPUs will get twice higher portion of max_active.
- Workqueue used to be able to process a chain of interdependent work items which is as long as max_active. We can't do this anymore as max_active is distributed across the nodes. Instead, a new parameter min_active is introduced which determines the minimum level of concurrency within a node regardless of how max_active distribution comes out to be.
It is set to the smaller of max_active and WQ_DFL_MIN_ACTIVE which is 8. This can lead to higher effective max_weight than configured and also deadlocks if a workqueue was depending on being able to handle chains of interdependent work items that are longer than 8.
I believe these should be fine given that the number of CPUs in each NUMA node is usually higher than 8 and work item chain longer than 8 is pretty unlikely. However, if these assumptions turn out to be wrong, we'll need to add an interface to adjust min_active.
- Each unbound wq has an array of struct wq_node_nr_active which tracks per-node nr_active. When its pwq wants to run a work item, it has to obtain the matching node's nr_active. If over the node's max_active, the pwq is queued on wq_node_nr_active->pending_pwqs. As work items finish, the completion path round-robins the pending pwqs activating the first inactive work item of each, which involves some pool lock dancing and kicking other pools. It's not the simplest code but doesn't look too bad.
This patchset includes the following patches:
0001-workqueue-Move-pwq-max_active-to-wq-max_active.patch 0002-workqueue-Factor-out-pwq_is_empty.patch 0003-workqueue-Replace-pwq_activate_inactive_work-with-__.patch 0004-workqueue-Move-nr_active-handling-into-helpers.patch 0005-workqueue-Make-wq_adjust_max_active-round-robin-pwqs.patch 0006-workqueue-RCU-protect-wq-dfl_pwq-and-implement-acces.patch 0007-workqueue-Move-pwq_dec_nr_in_flight-to-the-end-of-wo.patch 0008-workqueue-Introduce-struct-wq_node_nr_active.patch 0009-workqueue-Implement-system-wide-nr_active-enforcemen.patch 0010-tools-workqueue-wq_dump.py-Add-node_nr-max_active-du.patch
This pachset is also available in the following git branch.
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git unbound-system-wide-max_active-v3
diffstat follows. Thanks.
include/linux/workqueue.h | 35 +++ kernel/workqueue.c | 749 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------- tools/workqueue/wq_dump.py | 41 +++ 3 files changed, 691 insertions(+), 134 deletions(-)
--
tejun
Tejun Heo
maintainerPATCH 1/10 workqueue: Move pwq->max_active to wq->max_active
max_active is a workqueue-wide setting and the configured value is stored in wq->saved_max_active; however, the effective value was stored in pwq->max_active. While this is harmless, it makes max_active update process more complicated and gets in the way of the planned max_active semantic updates for unbound workqueues.
This patches moves pwq->max_active to wq->max_active. This simplifies the code and makes freezing and noop max_active updates cheaper too. No user-visible behavior change is intended.
As wq->max_active is updated while holding wq mutex but read without any locking, it now uses WRITE/READ_ONCE(). A new locking locking rule WO is added for it.
v2: wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai.
--- kernel/workqueue.c | 133 ++++++++++++++++++++++----------------------- 1 file changed, 66 insertions(+), 67 deletions(-)
kernel/workqueue.c +66 -67
@@ -143,6 +143,9 @@ enum {* * WR: wq->mutex protected for writes. RCU protected for reads. * +* WO: wq->mutex protected for writes. Updated with WRITE_ONCE() and can be read +* with READ_ONCE() without locking. +* * MD: wq_mayday_lock protected. * * WD: Used internally by the watchdog.
@@ -250,7 +253,6 @@ struct pool_workqueue {* is marked with WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works. */ int nr_active;/* L: nr of active works */ -int max_active;/* L: max active works */ struct list_head inactive_works;/* L: inactive works */ struct list_head pwqs_node;/* WR: node on wq->pwqs */ struct list_head mayday_node;/* MD: node on wq->maydays */
@@ -298,7 +300,8 @@ struct workqueue_struct {struct worker *rescuer;/* MD: rescue worker */ int nr_drainers;/* WQ: drain in progress */ -int saved_max_active;/* WQ: saved pwq max_active */ +int max_active;/* WO: max active works */ +int saved_max_active;/* WQ: saved max_active */ struct workqueue_attrs *unbound_attrs;/* PW: only for unbound wqs */ struct pool_workqueue *dfl_pwq;/* PW: only for unbound wqs */
@@ -1492,7 +1495,7 @@ static void pwq_dec_nr_in_flight(struct pool_workqueue *pwq, unsigned long work_ pwq->nr_active--;if (!list_empty (&pwq->inactive_works)) {/* one down, submit an inactive one */ -if (pwq->nr_active < pwq->max_active) +if (pwq->nr_active <READ_ONCE (pwq->wq->max_active))pwq_activate_first_inactive (pwq); } }
@@ -1793,7 +1796,13 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, pwq->nr_in_flight[pwq->work_color]++; work_flags =work_color_to_flags (pwq->work_color); -if (likely (pwq->nr_active < pwq->max_active)) { +/* +* Limit the number of concurrently active work items to max_active. +* @work must also queue behind existing inactive work items to maintain +* ordering when max_active changes. See wq_adjust_max_active(). +*/ +if (list_empty (&pwq->inactive_works) && + pwq->nr_active <READ_ONCE (pwq->wq->max_active)) {if (list_empty (&pool->worklist)) pool->watchdog_ts = jiffies;
@@ -4143,50 +4152,6 @@ static void pwq_release_workfn(struct kthread_work *work) } } -/** -* pwq_adjust_max_active - update a pwq's max_active to the current setting -* @pwq: target pool_workqueue -* -* If @pwq isn't freezing, set @pwq->max_active to the associated -* workqueue's saved_max_active and activate inactive work items -* accordingly. If @pwq is freezing, clear @pwq->max_active to zero. -*/ -static void pwq_adjust_max_active (struct pool_workqueue *pwq) -{ -struct workqueue_struct *wq = pwq->wq; -bool freezable = wq->flags &WQ_FREEZABLE ; -unsigned long flags; - -/* for @wq->saved_max_active */ -lockdep_assert_held (&wq->mutex); - -/* fast exit for non-freezable wqs */ -if (!freezable && pwq->max_active == wq->saved_max_active) -return ; - -/* this function can be called during early boot w/ irq disabled */ -raw_spin_lock_irqsave (&pwq->pool->lock, flags); - -/* -* During [un]freezing, the caller is responsible for ensuring that -* this function is called at least once after @workqueue_freezing -* is updated and visible. -*/ -if (!freezable || !workqueue_freezing) { - pwq->max_active = wq->saved_max_active; - -while (!list_empty (&pwq->inactive_works) && - pwq->nr_active < pwq->max_active) -pwq_activate_first_inactive (pwq); - -kick_pool (pwq->pool); - }else { - pwq->max_active =0 ; - } - -raw_spin_unlock_irqrestore (&pwq->pool->lock, flags); -} -/* initialize newly allocated @pwq which is associated with @wq and @pool */ static void init_pwq (struct pool_workqueue *pwq,struct workqueue_struct *wq,struct worker_pool *pool)
@@ -4219,9 +4184,6 @@ static void link_pwq(struct pool_workqueue *pwq)/* set the matching work_color */ pwq->work_color = wq->work_color; -/* sync max_active to the current setting */ -pwq_adjust_max_active (pwq); -/* link in @pwq */ list_add_rcu (&pwq->pwqs_node, &wq->pwqs); }
@@ -4662,6 +4624,52 @@ static int init_rescuer(struct workqueue_struct *wq)return 0 ; } +/** +* wq_adjust_max_active - update a wq's max_active to the current setting +* @wq: target workqueue +* +* If @wq isn't freezing, set @wq->max_active to the saved_max_active and +* activate inactive work items accordingly. If @wq is freezing, clear +* @wq->max_active to zero. +*/ +static void wq_adjust_max_active (struct workqueue_struct *wq) +{ +struct pool_workqueue *pwq; + +lockdep_assert_held (&wq->mutex); + +if ((wq->flags &WQ_FREEZABLE ) && workqueue_freezing) { +WRITE_ONCE (wq->max_active,0 ); +return ; + } + +if (wq->max_active == wq->saved_max_active) +return ; + +/* +* Update @wq->max_active and then kick inactive work items if more +* active work items are allowed. This doesn't break work item ordering +* because new work items are always queued behind existing inactive +* work items if there are any. +*/ +WRITE_ONCE (wq->max_active, wq->saved_max_active); + +for_each_pwq (pwq, wq) { +unsigned long flags; + +/* this function can be called during early boot w/ irq disabled */ +raw_spin_lock_irqsave (&pwq->pool->lock, flags); + +while (!list_empty (&pwq->inactive_works) && + pwq->nr_active < wq->max_active) +pwq_activate_first_inactive (pwq); + +kick_pool (pwq->pool); + +raw_spin_unlock_irqrestore (&pwq->pool->lock, flags); + } +} + __printf(1, 4) struct workqueue_struct *alloc_workqueue(const char *fmt, unsigned int flags,
@@ -4669,7 +4677,6 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, {va_list args;struct workqueue_struct *wq; -struct pool_workqueue *pwq;int len;/*
@@ -4708,6 +4715,7 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,/* init wq */ wq->flags = flags; + wq->max_active = max_active; wq->saved_max_active = max_active;mutex_init (&wq->mutex);atomic_set (&wq->nr_pwqs_to_flush,0 );
@@ -4736,8 +4744,7 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,mutex_lock (&wq_pool_mutex);mutex_lock (&wq->mutex); -for_each_pwq (pwq, wq) -pwq_adjust_max_active (pwq); +wq_adjust_max_active (wq);mutex_unlock (&wq->mutex);list_add_tail_rcu (&wq->list, &workqueues);
@@ -4875,8 +4882,6 @@ EXPORT_SYMBOL_GPL(destroy_workqueue);*/ void workqueue_set_max_active (struct workqueue_struct *wq,int max_active) { -struct pool_workqueue *pwq; -/* disallow meddling with max_active for ordered workqueues */ if (WARN_ON (wq->flags & __WQ_ORDERED_EXPLICIT))return ;
@@ -4887,9 +4892,7 @@ void workqueue_set_max_active(struct workqueue_struct *wq, int max_active) wq->flags &= ~__WQ_ORDERED; wq->saved_max_active = max_active; - -for_each_pwq (pwq, wq) -pwq_adjust_max_active (pwq); +wq_adjust_max_active (wq);mutex_unlock (&wq->mutex); }
@@ -5136,8 +5139,8 @@ static void show_pwq(struct pool_workqueue *pwq)pr_info (" pwq %d:" , pool->id);pr_cont_pool_info (pool); -pr_cont (" active=%d/%d refcnt=%d%s\n" , - pwq->nr_active, pwq->max_active, pwq->refcnt, +pr_cont (" active=%d refcnt=%d%s\n" , + pwq->nr_active, pwq->refcnt, !list_empty (&pwq->mayday_node) ?" MAYDAY" :"" );hash_for_each (pool->busy_hash, bkt, worker, hentry) {
@@ -5685,7 +5688,6 @@ EXPORT_SYMBOL_GPL(work_on_cpu_safe_key);void freeze_workqueues_begin (void ) {struct workqueue_struct *wq; -struct pool_workqueue *pwq;mutex_lock (&wq_pool_mutex);
@@ -5694,8 +5696,7 @@ void freeze_workqueues_begin(void)list_for_each_entry (wq, &workqueues, list) {mutex_lock (&wq->mutex); -for_each_pwq (pwq, wq) -pwq_adjust_max_active (pwq); +wq_adjust_max_active (wq);mutex_unlock (&wq->mutex); }
@@ -5760,7 +5761,6 @@ bool freeze_workqueues_busy(void)void thaw_workqueues (void ) {struct workqueue_struct *wq; -struct pool_workqueue *pwq;mutex_lock (&wq_pool_mutex);
@@ -5772,8 +5772,7 @@ void thaw_workqueues(void)/* restore max_active and repopulate worklist */ list_for_each_entry (wq, &workqueues, list) {mutex_lock (&wq->mutex); -for_each_pwq (pwq, wq) -pwq_adjust_max_active (pwq); +wq_adjust_max_active (wq);mutex_unlock (&wq->mutex); }
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 2/10 workqueue: Factor out pwq_is_empty()
In reply to Tejun Heo"!pwq->nr_active && list_empty(&pwq->inactive_works)" test is repeated multiple times. Let's factor it out into pwq_is_empty().
--- kernel/workqueue.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/workqueue.c +9 -4
@@ -1456,6 +1456,11 @@ static void put_pwq_unlocked(struct pool_workqueue *pwq) } } +static bool pwq_is_empty (struct pool_workqueue *pwq) +{ +return !pwq->nr_active &&list_empty (&pwq->inactive_works); +} +static void pwq_activate_inactive_work (struct work_struct *work) {struct pool_workqueue *pwq =get_work_pwq (work);
@@ -3326,7 +3331,7 @@ void drain_workqueue(struct workqueue_struct *wq)bool drained;raw_spin_lock_irq (&pwq->pool->lock); - drained = !pwq->nr_active &&list_empty (&pwq->inactive_works); + drained =pwq_is_empty (pwq);raw_spin_unlock_irq (&pwq->pool->lock);if (drained)
@@ -4776,7 +4781,7 @@ static bool pwq_busy(struct pool_workqueue *pwq)if ((pwq != pwq->wq->dfl_pwq) && (pwq->refcnt >1 ))return true; -if (pwq->nr_active || !list_empty (&pwq->inactive_works)) +if (!pwq_is_empty (pwq))return true;return false;
@@ -5214,7 +5219,7 @@ void show_one_workqueue(struct workqueue_struct *wq)unsigned long flags;for_each_pwq (pwq, wq) { -if (pwq->nr_active || !list_empty (&pwq->inactive_works)) { +if (!pwq_is_empty (pwq)) { idle = false;break ; }
@@ -5226,7 +5231,7 @@ void show_one_workqueue(struct workqueue_struct *wq)for_each_pwq (pwq, wq) {raw_spin_lock_irqsave (&pwq->pool->lock, flags); -if (pwq->nr_active || !list_empty (&pwq->inactive_works)) { +if (!pwq_is_empty (pwq)) {/* * Defer printing to avoid deadlocks in console * drivers that queue work while holding locks
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 3/10 workqueue: Replace pwq_activate_inactive_work() with [__]pwq_activate_work()
In reply to Tejun HeoTo prepare for unbound nr_active handling improvements, move work activation part of pwq_activate_inactive_work() into __pwq_activate_work() and add pwq_activate_work() which tests WORK_STRUCT_INACTIVE and updates nr_active.
pwq_activate_first_inactive() and try_to_grab_pending() are updated to use pwq_activate_work(). The latter conversion is functionally identical. For the former, this conversion adds an unnecessary WORK_STRUCT_INACTIVE testing. This is temporary and will be removed by the next patch.
--- kernel/workqueue.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)
kernel/workqueue.c +25 -6
@@ -1461,16 +1461,36 @@ static bool pwq_is_empty(struct pool_workqueue *pwq)return !pwq->nr_active &&list_empty (&pwq->inactive_works); } -static void pwq_activate_inactive_work (struct work_struct *work) +static void __pwq_activate_work (struct pool_workqueue *pwq, +struct work_struct *work) { -struct pool_workqueue *pwq =get_work_pwq (work); -trace_workqueue_activate_work (work);if (list_empty (&pwq->pool->worklist)) pwq->pool->watchdog_ts = jiffies;move_linked_works (work, &pwq->pool->worklist,NULL );__clear_bit (WORK_STRUCT_INACTIVE_BIT ,work_data_bits (work)); +} + +/** +* pwq_activate_work - Activate a work item if inactive +* @pwq: pool_workqueue @work belongs to +* @work: work item to activate +* +* Returns %true if activated. %false if already active. +*/ +static bool pwq_activate_work (struct pool_workqueue *pwq, +struct work_struct *work) +{ +struct worker_pool *pool = pwq->pool; + +lockdep_assert_held (&pool->lock); + +if (!(*work_data_bits (work) &WORK_STRUCT_INACTIVE )) +return false; + pwq->nr_active++; +__pwq_activate_work (pwq, work); +return true; }static void pwq_activate_first_inactive (struct pool_workqueue *pwq)
@@ -1478,7 +1498,7 @@ static void pwq_activate_first_inactive(struct pool_workqueue *pwq)struct work_struct *work =list_first_entry (&pwq->inactive_works, struct work_struct, entry); -pwq_activate_inactive_work (work); +pwq_activate_work (pwq, work); }/**
@@ -1616,8 +1636,7 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,* management later on and cause stall. Make sure the work * item is activated before grabbing. */ -if (*work_data_bits (work) &WORK_STRUCT_INACTIVE ) -pwq_activate_inactive_work (work); +pwq_activate_work (pwq, work);list_del_init (&work->entry);pwq_dec_nr_in_flight (pwq, *work_data_bits (work));
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 4/10 workqueue: Move nr_active handling into helpers
In reply to Tejun Heo__queue_work(), pwq_dec_nr_in_flight() and wq_adjust_max_active() were open-coding nr_active handling, which is fine given that the operations are trivial. However, the planned unbound nr_active update will make them more complicated, so let's move them into helpers.
- pwq_tryinc_nr_active() is added. It increments nr_active if under max_active limit and return a boolean indicating whether inc was successful. Note that the function is structured to accommodate future changes. __queue_work() is updated to use the new helper.
- pwq_activate_first_inactive() is updated to use pwq_tryinc_nr_active() and thus no longer assumes that nr_active is under max_active and returns a boolean to indicate whether a work item has been activated.
- wq_adjust_max_active() no longer tests directly whether a work item can be activated. Instead, it's updated to use the return value of pwq_activate_first_inactive() to tell whether a work item has been activated.
- nr_active decrement and activating the first inactive work item is factored into pwq_dec_nr_active().
v3: - WARN_ON_ONCE(!WORK_STRUCT_INACTIVE) added to __pwq_activate_work() as
now we're calling the function unconditionally from pwq_activate_first_inactive().
v2: - wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai.
--- kernel/workqueue.c | 86 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 67 insertions(+), 19 deletions(-)
kernel/workqueue.c +67 -19
@@ -1464,11 +1464,14 @@ static bool pwq_is_empty(struct pool_workqueue *pwq)static void __pwq_activate_work (struct pool_workqueue *pwq,struct work_struct *work) { +unsigned long *wdb =work_data_bits (work); + +WARN_ON_ONCE (!(*wdb &WORK_STRUCT_INACTIVE ));trace_workqueue_activate_work (work);if (list_empty (&pwq->pool->worklist)) pwq->pool->watchdog_ts = jiffies;move_linked_works (work, &pwq->pool->worklist,NULL ); -__clear_bit (WORK_STRUCT_INACTIVE_BIT ,work_data_bits (work)); +__clear_bit (WORK_STRUCT_INACTIVE_BIT , wdb); }/**
@@ -1493,12 +1496,66 @@ static bool pwq_activate_work(struct pool_workqueue *pwq,return true; } -static void pwq_activate_first_inactive (struct pool_workqueue *pwq) +/** +* pwq_tryinc_nr_active - Try to increment nr_active for a pwq +* @pwq: pool_workqueue of interest +* +* Try to increment nr_active for @pwq. Returns %true if an nr_active count is +* successfully obtained. %false otherwise. +*/ +static bool pwq_tryinc_nr_active (struct pool_workqueue *pwq) +{ +struct workqueue_struct *wq = pwq->wq; +struct worker_pool *pool = pwq->pool; +bool obtained; + +lockdep_assert_held (&pool->lock); + + obtained = pwq->nr_active <READ_ONCE (wq->max_active); + +if (obtained) + pwq->nr_active++; +return obtained; +} + +/** +* pwq_activate_first_inactive - Activate the first inactive work item on a pwq +* @pwq: pool_workqueue of interest +* +* Activate the first inactive work item of @pwq if available and allowed by +* max_active limit. +* +* Returns %true if an inactive work item has been activated. %false if no +* inactive work item is found or max_active limit is reached. +*/ +static bool pwq_activate_first_inactive (struct pool_workqueue *pwq) +{ +struct work_struct *work = +list_first_entry_or_null (&pwq->inactive_works, + struct work_struct, entry); + +if (work &&pwq_tryinc_nr_active (pwq)) { +__pwq_activate_work (pwq, work); +return true; + }else { +return false; + } +} + +/** +* pwq_dec_nr_active - Retire an active count +* @pwq: pool_workqueue of interest +* +* Decrement @pwq's nr_active and try to activate the first inactive work item. +*/ +static void pwq_dec_nr_active (struct pool_workqueue *pwq) { -struct work_struct *work =list_first_entry (&pwq->inactive_works, - struct work_struct, entry); +struct worker_pool *pool = pwq->pool; -pwq_activate_work (pwq, work); +lockdep_assert_held (&pool->lock); + + pwq->nr_active--; +pwq_activate_first_inactive (pwq); }/**
@@ -1516,14 +1573,8 @@ static void pwq_dec_nr_in_flight(struct pool_workqueue *pwq, unsigned long work_ {int color =get_work_color (work_data); -if (!(work_data &WORK_STRUCT_INACTIVE )) { - pwq->nr_active--; -if (!list_empty (&pwq->inactive_works)) { -/* one down, submit an inactive one */ -if (pwq->nr_active <READ_ONCE (pwq->wq->max_active)) -pwq_activate_first_inactive (pwq); - } - } +if (!(work_data &WORK_STRUCT_INACTIVE )) +pwq_dec_nr_active (pwq); pwq->nr_in_flight[color]--;
@@ -1825,13 +1876,11 @@ static void __queue_work(int cpu, struct workqueue_struct *wq,* @work must also queue behind existing inactive work items to maintain * ordering when max_active changes. See wq_adjust_max_active(). */ -if (list_empty (&pwq->inactive_works) && - pwq->nr_active <READ_ONCE (pwq->wq->max_active)) { +if (list_empty (&pwq->inactive_works) &&pwq_tryinc_nr_active (pwq)) {if (list_empty (&pool->worklist)) pool->watchdog_ts = jiffies;trace_workqueue_activate_work (work); - pwq->nr_active++;insert_work (pwq, work, &pool->worklist, work_flags);kick_pool (pool); }else {
@@ -4684,9 +4733,8 @@ static void wq_adjust_max_active(struct workqueue_struct *wq)/* this function can be called during early boot w/ irq disabled */ raw_spin_lock_irqsave (&pwq->pool->lock, flags); -while (!list_empty (&pwq->inactive_works) && - pwq->nr_active < wq->max_active) -pwq_activate_first_inactive (pwq); +while (pwq_activate_first_inactive (pwq)) + ;kick_pool (pwq->pool);
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 5/10 workqueue: Make wq_adjust_max_active() round-robin pwqs while activating
In reply to Tejun Heowq_adjust_max_active() needs to activate work items after max_active is increased. Previously, it did that by visiting each pwq once activating all that could be activated. While this makes sense with per-pwq nr_active, nr_active will be shared across multiple pwqs for unbound wqs. Then, we'd want to round-robin through pwqs to be fairer.
In preparation, this patch makes wq_adjust_max_active() round-robin pwqs while activating. While the activation ordering changes, this shouldn't cause user-noticeable behavior changes.
--- kernel/workqueue.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
kernel/workqueue.c +19 -12
@@ -4707,7 +4707,7 @@ static int init_rescuer(struct workqueue_struct *wq)*/ static void wq_adjust_max_active (struct workqueue_struct *wq) { -struct pool_workqueue *pwq; +bool activated;lockdep_assert_held (&wq->mutex);
@@ -4727,19 +4727,26 @@ static void wq_adjust_max_active(struct workqueue_struct *wq)*/ WRITE_ONCE (wq->max_active, wq->saved_max_active); -for_each_pwq (pwq, wq) { -unsigned long flags; - -/* this function can be called during early boot w/ irq disabled */ -raw_spin_lock_irqsave (&pwq->pool->lock, flags); - -while (pwq_activate_first_inactive (pwq)) - ; +/* +* Round-robin through pwq's activating the first inactive work item +* until max_active is filled. +*/ +do { +struct pool_workqueue *pwq; -kick_pool (pwq->pool); + activated = false; +for_each_pwq (pwq, wq) { +unsigned long flags; -raw_spin_unlock_irqrestore (&pwq->pool->lock, flags); - } +/* can be called during early boot w/ irq disabled */ +raw_spin_lock_irqsave (&pwq->pool->lock, flags); +if (pwq_activate_first_inactive (pwq)) { + activated = true; +kick_pool (pwq->pool); + } +raw_spin_unlock_irqrestore (&pwq->pool->lock, flags); + } + }while (activated); }__printf (1 ,4 )
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 6/10 workqueue: RCU protect wq->dfl_pwq and implement accessors for it
In reply to Tejun Heowq->cpu_pwq is RCU protected but wq->dfl_pwq isn't. This is okay because currently wq->dfl_pwq is used only accessed to install it into wq->cpu_pwq which doesn't require RCU access. However, we want to be able to access wq->dfl_pwq under RCU in the future to access its __pod_cpumask and the code can be made easier to read by making the two pwq fields behave in the same way.
- Make wq->dfl_pwq RCU protected.
- Add unbound_pwq_slot() and unbound_pwq() which can access both ->dfl_pwq and ->cpu_pwq. The former returns the double pointer that can be used access and update the pwqs. The latter performs locking check and dereferences the double pointer.
- pwq accesses and updates are converted to use unbound_pwq[_slot]().
--- kernel/workqueue.c | 64 +++++++++++++++++++++++++++++----------------- 1 file changed, 40 insertions(+), 24 deletions(-)
kernel/workqueue.c +40 -24
@@ -304,7 +304,7 @@ struct workqueue_struct {int saved_max_active;/* WQ: saved max_active */ struct workqueue_attrs *unbound_attrs;/* PW: only for unbound wqs */ -struct pool_workqueue *dfl_pwq;/* PW: only for unbound wqs */ +struct pool_workqueue __rcu *dfl_pwq;/* PW: only for unbound wqs */ #ifdef CONFIG_SYSFS struct wq_device *wq_dev;/* I: for sysfs interface */
@@ -635,6 +635,23 @@ static int worker_pool_assign_id(struct worker_pool *pool)return ret; } +static struct pool_workqueue __rcu ** +unbound_pwq_slot (struct workqueue_struct *wq,int cpu) +{ +if (cpu >=0 ) +return per_cpu_ptr (wq->cpu_pwq, cpu); +else +return &wq->dfl_pwq; +} + +/* @cpu < 0 for dfl_pwq */ +static struct pool_workqueue *unbound_pwq (struct workqueue_struct *wq,int cpu) +{ +return rcu_dereference_check (*unbound_pwq_slot (wq, cpu), +lockdep_is_held (&wq_pool_mutex) || +lockdep_is_held (&wq->mutex)); +} +static unsigned int work_color_to_flags (int color) {return color <<WORK_STRUCT_COLOR_SHIFT ;
@@ -4325,10 +4342,11 @@ static void wq_calc_pod_cpumask(struct workqueue_attrs *attrs, int cpu, "possible intersect\n"); } -/* install @pwq into @wq's cpu_pwq and return the old pwq */ +/* install @pwq into @wq and return the old pwq, @cpu < 0 for dfl_pwq */ static struct pool_workqueue *install_unbound_pwq (struct workqueue_struct *wq,int cpu,struct pool_workqueue *pwq) { +struct pool_workqueue __rcu **slot =unbound_pwq_slot (wq, cpu);struct pool_workqueue *old_pwq;lockdep_assert_held (&wq_pool_mutex);
@@ -4337,8 +4355,8 @@ static struct pool_workqueue *install_unbound_pwq(struct workqueue_struct *wq,/* link_pwq() can handle duplicate calls */ link_pwq (pwq); - old_pwq =rcu_access_pointer (*per_cpu_ptr (wq->cpu_pwq, cpu)); -rcu_assign_pointer (*per_cpu_ptr (wq->cpu_pwq, cpu), pwq); + old_pwq =rcu_access_pointer (*slot); +rcu_assign_pointer (*slot, pwq);return old_pwq; }
@@ -4438,14 +4456,11 @@ static void apply_wqattrs_commit(struct apply_wqattrs_ctx *ctx)copy_workqueue_attrs (ctx->wq->unbound_attrs, ctx->attrs); -/* save the previous pwq and install the new one */ +/* save the previous pwqs and install the new ones */ for_each_possible_cpu (cpu) ctx->pwq_tbl[cpu] =install_unbound_pwq (ctx->wq, cpu, ctx->pwq_tbl[cpu]); - -/* @dfl_pwq might not have been used, ensure it's linked */ -link_pwq (ctx->dfl_pwq); -swap (ctx->wq->dfl_pwq, ctx->dfl_pwq); + ctx->dfl_pwq =install_unbound_pwq (ctx->wq,-1 , ctx->dfl_pwq);mutex_unlock (&ctx->wq->mutex); }
@@ -4555,9 +4570,7 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu,/* nothing to do if the target cpumask matches the current pwq */ wq_calc_pod_cpumask (target_attrs, cpu, off_cpu); - pwq =rcu_dereference_protected (*per_cpu_ptr (wq->cpu_pwq, cpu), -lockdep_is_held (&wq_pool_mutex)); -if (wqattrs_equal (target_attrs, pwq->pool->attrs)) +if (wqattrs_equal (target_attrs,unbound_pwq (wq, cpu)->pool->attrs))return ;/* create a new pwq */
@@ -4575,10 +4588,11 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu, use_dfl_pwq:mutex_lock (&wq->mutex); -raw_spin_lock_irq (&wq->dfl_pwq->pool->lock); -get_pwq (wq->dfl_pwq); -raw_spin_unlock_irq (&wq->dfl_pwq->pool->lock); - old_pwq =install_unbound_pwq (wq, cpu, wq->dfl_pwq); + pwq =unbound_pwq (wq,-1 ); +raw_spin_lock_irq (&pwq->pool->lock); +get_pwq (pwq); +raw_spin_unlock_irq (&pwq->pool->lock); + old_pwq =install_unbound_pwq (wq, cpu, pwq); out_unlock:mutex_unlock (&wq->mutex);put_pwq_unlocked (old_pwq);
@@ -4616,10 +4630,13 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)cpus_read_lock ();if (wq->flags & __WQ_ORDERED) { +struct pool_workqueue *dfl_pwq; + ret =apply_workqueue_attrs (wq, ordered_wq_attrs[highpri]);/* there should only be single pwq for ordering guarantee */ -WARN (!ret && (wq->pwqs.next != &wq->dfl_pwq->pwqs_node || - wq->pwqs.prev != &wq->dfl_pwq->pwqs_node), + dfl_pwq =rcu_access_pointer (wq->dfl_pwq); +WARN (!ret && (wq->pwqs.next != &dfl_pwq->pwqs_node || + wq->pwqs.prev != &dfl_pwq->pwqs_node),"ordering guarantee broken for workqueue %s\n" , wq->name); }else { ret =apply_workqueue_attrs (wq, unbound_std_wq_attrs[highpri]);
@@ -4853,7 +4870,7 @@ static bool pwq_busy(struct pool_workqueue *pwq)if (pwq->nr_in_flight[i])return true; -if ((pwq != pwq->wq->dfl_pwq) && (pwq->refcnt >1 )) +if ((pwq !=rcu_access_pointer (pwq->wq->dfl_pwq)) && (pwq->refcnt >1 ))return true;if (!pwq_is_empty (pwq))return true;
@@ -4937,13 +4954,12 @@ void destroy_workqueue(struct workqueue_struct *wq)rcu_read_lock ();for_each_possible_cpu (cpu) { - pwq =rcu_access_pointer (*per_cpu_ptr (wq->cpu_pwq, cpu)); -RCU_INIT_POINTER (*per_cpu_ptr (wq->cpu_pwq, cpu),NULL ); -put_pwq_unlocked (pwq); +put_pwq_unlocked (unbound_pwq (wq, cpu)); +RCU_INIT_POINTER (*unbound_pwq_slot (wq, cpu),NULL ); } -put_pwq_unlocked (wq->dfl_pwq); - wq->dfl_pwq =NULL ; +put_pwq_unlocked (unbound_pwq (wq,-1 )); +RCU_INIT_POINTER (*unbound_pwq_slot (wq,-1 ),NULL );rcu_read_unlock (); }
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 7/10 workqueue: Move pwq_dec_nr_in_flight() to the end of work item handling
In reply to Tejun HeoThe planned shared nr_active handling for unbound workqueues will make pwq_dec_nr_active() sometimes drop the pool lock temporarily to acquire other pool locks, which is necessary as retirement of an nr_active count from one pool may need kick off an inactive work item in another pool.
This patch moves pwq_dec_nr_in_flight() call in try_to_grab_pending() to the end of work item handling so that work item state changes stay atomic. process_one_work() which is the other user of pwq_dec_nr_in_flight() already calls it at the end of work item handling. Comments are added to both call sites and pwq_dec_nr_in_flight().
This shouldn't cause any behavior changes.
--- kernel/workqueue.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/workqueue.c +10 -1
@@ -1583,6 +1583,11 @@ static void pwq_dec_nr_active(struct pool_workqueue *pwq)* A work either has completed or is removed from pending queue, * decrement nr_in_flight of its pwq and handle workqueue flushing. * +* NOTE: +* For unbound workqueues, this function may temporarily drop @pwq->pool->lock +* and thus should be called after all other state updates for the in-flight +* work item is complete. +* * CONTEXT: * raw_spin_lock_irq(pool->lock). */
@@ -1707,11 +1712,13 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork,pwq_activate_work (pwq, work);list_del_init (&work->entry); -pwq_dec_nr_in_flight (pwq, *work_data_bits (work));/* work->data points to pwq iff queued, point to pool */ set_work_pool_and_keep_pending (work, pool->id); +/* must be the last step, see the function comment */ +pwq_dec_nr_in_flight (pwq, *work_data_bits (work)); +raw_spin_unlock (&pool->lock);rcu_read_unlock ();return 1 ;
@@ -2777,6 +2784,8 @@ __acquires(&pool->lock) worker->current_func =NULL ; worker->current_pwq =NULL ; worker->current_color =INT_MAX ; + +/* must be the last step, see the function comment */ pwq_dec_nr_in_flight (pwq, work_data); }
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 8/10 workqueue: Introduce struct wq_node_nr_active
In reply to Tejun HeoCurrently, for both percpu and unbound workqueues, max_active applies per-cpu, which is a recent change for unbound workqueues. The change for unbound workqueues was a significant departure from the previous behavior of per-node application. It made some use cases create undesirable number of concurrent work items and left no good way of fixing them. To address the problem, workqueue is implementing a NUMA node segmented global nr_active mechanism, which will be explained further in the next patch.
As a preparation, this patch introduces struct wq_node_nr_active. It's a data structured allocated for each workqueue and NUMA node pair and currently only tracks the workqueue's number of active work items on the node. This is split out from the next patch to make it easier to understand and review.
Note that there is an extra wq_node_nr_active allocated for the invalid node nr_node_ids which is used to track nr_active for pools which don't have NUMA node associated such as the default fallback system-wide pool.
This doesn't cause any behavior changes visible to userland yet. The next patch will expand to implement the control mechanism on top.
v3: - Use flexible array for wq->node_nr_active as suggested by Lai.
v2: - wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai.
- Lai pointed out that pwq_tryinc_nr_active() incorrectly dropped pwq->max_active check. Restored. As the next patch replaces the max_active enforcement mechanism, this doesn't change the end result.
--- kernel/workqueue.c | 139 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 132 insertions(+), 7 deletions(-)
kernel/workqueue.c +132 -7
@@ -280,6 +280,16 @@ struct wq_flusher {struct wq_device ; +/* +* Unlike in a per-cpu workqueue where max_active limits its concurrency level +* on each CPU, in an unbound workqueue, max_active applies to the whole system. +* As sharing a single nr_active across multiple sockets can be very expensive, +* the counting and enforcement is per NUMA node. +*/ +struct wq_node_nr_active { +atomic_t nr;/* per-node nr_active count */ +}; +/* * The externally visible workqueue. It relays the issued work items to * the appropriate worker_pool through its pool_workqueues.
@@ -326,6 +336,7 @@ struct workqueue_struct {/* hot fields used during command issue, aligned to cacheline */ unsigned int flags____cacheline_aligned ;/* WQ: WQ_* flags */ struct pool_workqueue __percpu __rcu **cpu_pwq;/* I: per-cpu pwqs */ +struct wq_node_nr_active *node_nr_active[];/* I: per-node nr_active */ };static struct kmem_cache *pwq_cache;
@@ -1421,6 +1432,31 @@ work_func_t wq_worker_last_func(struct task_struct *task)return worker->last_func; } +/** +* wq_node_nr_active - Determine wq_node_nr_active to use +* @wq: workqueue of interest +* @node: NUMA node, can be %NUMA_NO_NODE +* +* Determine wq_node_nr_active to use for @wq on @node. Returns: +* +* - %NULL for per-cpu workqueues as they don't need to use shared nr_active. +* +* - node_nr_active[nr_node_ids] if @node is %NUMA_NO_NODE. +* +* - Otherwise, node_nr_active[@node]. +*/ +static struct wq_node_nr_active *wq_node_nr_active (struct workqueue_struct *wq, +int node) +{ +if (!(wq->flags &WQ_UNBOUND )) +return NULL ; + +if (node ==NUMA_NO_NODE ) + node = nr_node_ids; + +return wq->node_nr_active[node]; +} +/** * get_pwq - get an extra reference on the specified pool_workqueue * @pwq: pool_workqueue to get
@@ -1502,12 +1538,17 @@ static bool pwq_activate_work(struct pool_workqueue *pwq, struct work_struct *work) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna;lockdep_assert_held (&pool->lock);if (!(*work_data_bits (work) &WORK_STRUCT_INACTIVE ))return false; + nna =wq_node_nr_active (pwq->wq, pool->node); +if (nna) +atomic_inc (&nna->nr); + pwq->nr_active++;__pwq_activate_work (pwq, work);return true;
@@ -1524,14 +1565,18 @@ static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq) {struct workqueue_struct *wq = pwq->wq;struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (wq, pool->node);bool obtained;lockdep_assert_held (&pool->lock); obtained = pwq->nr_active <READ_ONCE (wq->max_active); -if (obtained) +if (obtained) { pwq->nr_active++; +if (nna) +atomic_inc (&nna->nr); + }return obtained; }
@@ -1568,10 +1613,26 @@ static bool pwq_activate_first_inactive(struct pool_workqueue *pwq)static void pwq_dec_nr_active (struct pool_workqueue *pwq) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (pwq->wq, pool->node);lockdep_assert_held (&pool->lock); +/* +* @pwq->nr_active should be decremented for both percpu and unbound +* workqueues. +*/ pwq->nr_active--; + +/* +* For a percpu workqueue, it's simple. Just need to kick the first +* inactive work item on @pwq itself. +*/ +if (!nna) { +pwq_activate_first_inactive (pwq); +return ; + } + +atomic_dec (&nna->nr);pwq_activate_first_inactive (pwq); }
@@ -4036,12 +4097,62 @@ static void wq_free_lockdep(struct workqueue_struct *wq) }#endif +static void free_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +int node; + +for_each_node (node) { +kfree (nna_ar[node]); + nna_ar[node] =NULL ; + } + +kfree (nna_ar[nr_node_ids]); + nna_ar[nr_node_ids] =NULL ; +} + +static void init_node_nr_active (struct wq_node_nr_active *nna) +{ +atomic_set (&nna->nr,0 ); +} + +/* +* Each node's nr_active counter will be accessed mostly from its own node and +* should be allocated in the node. +*/ +static int alloc_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +struct wq_node_nr_active *nna; +int node; + +for_each_node (node) { + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL , node); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[node] = nna; + } + +/* [nr_node_ids] is used as the fallback */ + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL ,NUMA_NO_NODE ); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[nr_node_ids] = nna; + +return 0 ; + +err_free: +free_node_nr_active (nna_ar); +return -ENOMEM ; +} +static void rcu_free_wq (struct rcu_head *rcu) {struct workqueue_struct *wq = container_of(rcu, struct workqueue_struct, rcu);wq_free_lockdep (wq); +free_node_nr_active (wq->node_nr_active);free_percpu (wq->cpu_pwq);free_workqueue_attrs (wq->unbound_attrs);kfree (wq);
@@ -4782,7 +4893,8 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, {va_list args;struct workqueue_struct *wq; -int len; +size_t wq_size; +int name_len;/* * Unbound && max_active == 1 used to imply ordered, which is no longer
@@ -4798,7 +4910,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, flags |=WQ_UNBOUND ;/* allocate wq and format name */ - wq =kzalloc (sizeof (*wq),GFP_KERNEL ); +if (flags &WQ_UNBOUND ) + wq_size =struct_size (wq, node_nr_active, nr_node_ids +1 ); +else + wq_size =sizeof (*wq); + + wq =kzalloc (wq_size,GFP_KERNEL );if (!wq)return NULL ;
@@ -4809,11 +4926,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, }va_start (args, max_active); - len =vsnprintf (wq->name,sizeof (wq->name), fmt, args); + name_len =vsnprintf (wq->name,sizeof (wq->name), fmt, args);va_end (args); -if (len >=WQ_NAME_LEN ) -pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , wq->name); +if (name_len >=WQ_NAME_LEN ) +pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , + wq->name); max_active = max_active ?:WQ_DFL_ACTIVE ; max_active =wq_clamp_max_active (max_active, flags, wq->name);
@@ -4832,8 +4950,13 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,wq_init_lockdep (wq);INIT_LIST_HEAD (&wq->list); +if (flags &WQ_UNBOUND ) { +if (alloc_node_nr_active (wq->node_nr_active) <0 ) + goto err_unreg_lockdep; + } +if (alloc_and_link_pwqs (wq) <0 ) - goto err_unreg_lockdep; + goto err_free_node_nr_active;if (wq_online &&init_rescuer (wq) <0 ) goto err_destroy;
@@ -4858,6 +4981,8 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,return wq; +err_free_node_nr_active: +free_node_nr_active (wq->node_nr_active); err_unreg_lockdep:wq_unregister_lockdep (wq);wq_free_lockdep (wq);
Signature · 1 line
2.43.0
Lai Jiangshan
reviewerRe: [PATCH 08/10] workqueue: Introduce struct wq_node_nr_active
Hello, Tejun
Quoted 17 lines · Tejun Heo
kernel/workqueue.c +13
-0@@ -4036,12 +4097,62 @@ static void wq_free_lockdep(struct workqueue_struct *wq) }#endif +static void free_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +int node; + +for_each_node (node) { +kfree (nna_ar[node]); + nna_ar[node] =NULL ; + } + +kfree (nna_ar[nr_node_ids]); + nna_ar[nr_node_ids] =NULL ; +} +
[....]
Quoted 7 lines
static void
rcu_free_wq(struct rcu_head *rcu){ struct workqueue_struct *wq = container_of(rcu, struct workqueue_struct, rcu); wq_free_lockdep(wq);+
free_node_nr_active(wq->node_nr_active);
for percpu workqueue, free_node_nr_active() will access out of bound.
Quoted 3 lines
free_percpu(wq->cpu_pwq);
free_workqueue_attrs(wq->unbound_attrs);
kfree(wq);[......]
Quoted 16 lines
kernel/workqueue.c +6
-1@@ -4832,8 +4950,13 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,wq_init_lockdep (wq);INIT_LIST_HEAD (&wq->list); +if (flags &WQ_UNBOUND ) { +if (alloc_node_nr_active (wq->node_nr_active) <0 ) + goto err_unreg_lockdep; + } +if (alloc_and_link_pwqs (wq) <0 ) - goto err_unreg_lockdep; + goto err_free_node_nr_active;if (wq_online &&init_rescuer (wq) <0 ) goto err_destroy;
kernel/workqueue.c struct workqueue_struct *alloc_workqueue(const char *fmt, · line 4985
Quoted 5 lines
kernel/workqueue.c +2
-0@@ -4858,6 +4981,8 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,return wq; +err_free_node_nr_active: +free_node_nr_active (wq->node_nr_active);
for percpu workqueue, free_node_nr_active() will access out of bound.
Thanks
Lai
Quoted 6 lines
kernel/workqueue.c +0
-1err_unreg_lockdep:wq_unregister_lockdep (wq);wq_free_lockdep (wq); --2.43.0
Tejun Heo
maintainerPATCH v4 8/10 workqueue: Introduce struct wq_node_nr_active
From 91ccc6e7233bb10a9c176aa4cc70d6f432a441a5 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Mon, 29 Jan 2024 08:11:24 -1000
Currently, for both percpu and unbound workqueues, max_active applies per-cpu, which is a recent change for unbound workqueues. The change for unbound workqueues was a significant departure from the previous behavior of per-node application. It made some use cases create undesirable number of concurrent work items and left no good way of fixing them. To address the problem, workqueue is implementing a NUMA node segmented global nr_active mechanism, which will be explained further in the next patch.
As a preparation, this patch introduces struct wq_node_nr_active. It's a data structured allocated for each workqueue and NUMA node pair and currently only tracks the workqueue's number of active work items on the node. This is split out from the next patch to make it easier to understand and review.
Note that there is an extra wq_node_nr_active allocated for the invalid node nr_node_ids which is used to track nr_active for pools which don't have NUMA node associated such as the default fallback system-wide pool.
This doesn't cause any behavior changes visible to userland yet. The next patch will expand to implement the control mechanism on top.
v4: - Fixed out-of-bound access when freeing per-cpu workqueues.
v3: - Use flexible array for wq->node_nr_active as suggested by Lai.
v2: - wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai.
- Lai pointed out that pwq_tryinc_nr_active() incorrectly dropped pwq->max_active check. Restored. As the next patch replaces the max_active enforcement mechanism, this doesn't change the end result.
--- kernel/workqueue.c | 142 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 135 insertions(+), 7 deletions(-)
kernel/workqueue.c +135 -7
@@ -284,6 +284,16 @@ struct wq_flusher {struct wq_device ; +/* +* Unlike in a per-cpu workqueue where max_active limits its concurrency level +* on each CPU, in an unbound workqueue, max_active applies to the whole system. +* As sharing a single nr_active across multiple sockets can be very expensive, +* the counting and enforcement is per NUMA node. +*/ +struct wq_node_nr_active { +atomic_t nr;/* per-node nr_active count */ +}; +/* * The externally visible workqueue. It relays the issued work items to * the appropriate worker_pool through its pool_workqueues.
@@ -330,6 +340,7 @@ struct workqueue_struct {/* hot fields used during command issue, aligned to cacheline */ unsigned int flags____cacheline_aligned ;/* WQ: WQ_* flags */ struct pool_workqueue __percpu __rcu **cpu_pwq;/* I: per-cpu pwqs */ +struct wq_node_nr_active *node_nr_active[];/* I: per-node nr_active */ };static struct kmem_cache *pwq_cache;
@@ -1425,6 +1436,31 @@ work_func_t wq_worker_last_func(struct task_struct *task)return worker->last_func; } +/** +* wq_node_nr_active - Determine wq_node_nr_active to use +* @wq: workqueue of interest +* @node: NUMA node, can be %NUMA_NO_NODE +* +* Determine wq_node_nr_active to use for @wq on @node. Returns: +* +* - %NULL for per-cpu workqueues as they don't need to use shared nr_active. +* +* - node_nr_active[nr_node_ids] if @node is %NUMA_NO_NODE. +* +* - Otherwise, node_nr_active[@node]. +*/ +static struct wq_node_nr_active *wq_node_nr_active (struct workqueue_struct *wq, +int node) +{ +if (!(wq->flags &WQ_UNBOUND )) +return NULL ; + +if (node ==NUMA_NO_NODE ) + node = nr_node_ids; + +return wq->node_nr_active[node]; +} +/** * get_pwq - get an extra reference on the specified pool_workqueue * @pwq: pool_workqueue to get
@@ -1506,12 +1542,17 @@ static bool pwq_activate_work(struct pool_workqueue *pwq, struct work_struct *work) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna;lockdep_assert_held (&pool->lock);if (!(*work_data_bits (work) &WORK_STRUCT_INACTIVE ))return false; + nna =wq_node_nr_active (pwq->wq, pool->node); +if (nna) +atomic_inc (&nna->nr); + pwq->nr_active++;__pwq_activate_work (pwq, work);return true;
@@ -1528,14 +1569,18 @@ static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq) {struct workqueue_struct *wq = pwq->wq;struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (wq, pool->node);bool obtained;lockdep_assert_held (&pool->lock); obtained = pwq->nr_active <READ_ONCE (wq->max_active); -if (obtained) +if (obtained) { pwq->nr_active++; +if (nna) +atomic_inc (&nna->nr); + }return obtained; }
@@ -1572,10 +1617,26 @@ static bool pwq_activate_first_inactive(struct pool_workqueue *pwq)static void pwq_dec_nr_active (struct pool_workqueue *pwq) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (pwq->wq, pool->node);lockdep_assert_held (&pool->lock); +/* +* @pwq->nr_active should be decremented for both percpu and unbound +* workqueues. +*/ pwq->nr_active--; + +/* +* For a percpu workqueue, it's simple. Just need to kick the first +* inactive work item on @pwq itself. +*/ +if (!nna) { +pwq_activate_first_inactive (pwq); +return ; + } + +atomic_dec (&nna->nr);pwq_activate_first_inactive (pwq); }
@@ -4039,11 +4100,63 @@ static void wq_free_lockdep(struct workqueue_struct *wq) }#endif +static void free_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +int node; + +for_each_node (node) { +kfree (nna_ar[node]); + nna_ar[node] =NULL ; + } + +kfree (nna_ar[nr_node_ids]); + nna_ar[nr_node_ids] =NULL ; +} + +static void init_node_nr_active (struct wq_node_nr_active *nna) +{ +atomic_set (&nna->nr,0 ); +} + +/* +* Each node's nr_active counter will be accessed mostly from its own node and +* should be allocated in the node. +*/ +static int alloc_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +struct wq_node_nr_active *nna; +int node; + +for_each_node (node) { + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL , node); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[node] = nna; + } + +/* [nr_node_ids] is used as the fallback */ + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL ,NUMA_NO_NODE ); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[nr_node_ids] = nna; + +return 0 ; + +err_free: +free_node_nr_active (nna_ar); +return -ENOMEM ; +} +static void rcu_free_wq (struct rcu_head *rcu) {struct workqueue_struct *wq = container_of(rcu, struct workqueue_struct, rcu); +if (wq->flags &WQ_UNBOUND ) +free_node_nr_active (wq->node_nr_active); +wq_free_lockdep (wq);free_percpu (wq->cpu_pwq);free_workqueue_attrs (wq->unbound_attrs);
@@ -4785,7 +4898,8 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, {va_list args;struct workqueue_struct *wq; -int len; +size_t wq_size; +int name_len;/* * Unbound && max_active == 1 used to imply ordered, which is no longer
@@ -4801,7 +4915,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, flags |=WQ_UNBOUND ;/* allocate wq and format name */ - wq =kzalloc (sizeof (*wq),GFP_KERNEL ); +if (flags &WQ_UNBOUND ) + wq_size =struct_size (wq, node_nr_active, nr_node_ids +1 ); +else + wq_size =sizeof (*wq); + + wq =kzalloc (wq_size,GFP_KERNEL );if (!wq)return NULL ;
@@ -4812,11 +4931,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, }va_start (args, max_active); - len =vsnprintf (wq->name,sizeof (wq->name), fmt, args); + name_len =vsnprintf (wq->name,sizeof (wq->name), fmt, args);va_end (args); -if (len >=WQ_NAME_LEN ) -pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , wq->name); +if (name_len >=WQ_NAME_LEN ) +pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , + wq->name); max_active = max_active ?:WQ_DFL_ACTIVE ; max_active =wq_clamp_max_active (max_active, flags, wq->name);
@@ -4835,8 +4955,13 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,wq_init_lockdep (wq);INIT_LIST_HEAD (&wq->list); +if (flags &WQ_UNBOUND ) { +if (alloc_node_nr_active (wq->node_nr_active) <0 ) + goto err_unreg_lockdep; + } +if (alloc_and_link_pwqs (wq) <0 ) - goto err_unreg_lockdep; + goto err_free_node_nr_active;if (wq_online &&init_rescuer (wq) <0 ) goto err_destroy;
@@ -4861,6 +4986,9 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,return wq; +err_free_node_nr_active: +if (wq->flags &WQ_UNBOUND ) +free_node_nr_active (wq->node_nr_active); err_unreg_lockdep:wq_unregister_lockdep (wq);wq_free_lockdep (wq);
Signature · 1 line
2.43.0
Re: [PATCH v4 08/10] workqueue: Introduce struct wq_node_nr_active
Hi Tejun,
Quoted 299 lines · Tejun Heo
Quoted 1 line
From
91ccc6e7233bb10a9c176aa4cc70d6f432a441a5Mon Sep 17 00:00:00 2001From: Tejun Heo <[email protected]>
Date: Mon, 29 Jan 2024 08:11:24 -1000Currently, for both percpu and unbound workqueues,
max_activeapplies per-cpu, which is a recent change for unbound workqueues. The change for unbound workqueues was a significant departure from the previous behavior of per-node application. It made some use cases create undesirable number of concurrent work items and left no good way of fixing them. To address the problem, workqueue is implementing a NUMA node segmented globalnr_activemechanism, which will be explained further in the next patch.As a preparation, this patch introduces struct
wq_node_nr_active. It's a data structured allocated for each workqueue and NUMA node pair and currently only tracks the workqueue's number of active work items on the node. This is split out from the next patch to make it easier to understand and review.Note that there is an extra
wq_node_nr_activeallocated for the invalid nodenr_node_idswhich is used to tracknr_activefor pools which don't have NUMA node associated such as the default fallback system-wide pool.This doesn't cause any behavior changes visible to userland yet. The next patch will expand to implement the control mechanism on top.
v4: - Fixed out-of-bound access when freeing per-cpu workqueues.
v3: - Use flexible array for
wq->node_nr_activeas suggested by Lai.v2: -
wq->max_activenow uses WRITE/READ_ONCE() as suggested by Lai.- Lai pointed out that
pwq_tryinc_nr_active()incorrectly droppedpwq->max_activecheck. Restored. As the next patch replaces themax_activeenforcement mechanism, this doesn't change the end result.Signed-off-by Tejun Heo <[email protected]>Reviewed-by Lai Jiangshan <[email protected]>--- kernel/workqueue.c | 142 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 135 insertions(+), 7 deletions(-)kernel/workqueue.c +135
-7@@ -284,6 +284,16 @@ struct wq_flusher {struct wq_device ; +/* +* Unlike in a per-cpu workqueue where max_active limits its concurrency level +* on each CPU, in an unbound workqueue, max_active applies to the whole system. +* As sharing a single nr_active across multiple sockets can be very expensive, +* the counting and enforcement is per NUMA node. +*/ +struct wq_node_nr_active { +atomic_t nr;/* per-node nr_active count */ +}; +/* * The externally visible workqueue. It relays the issued work items to * the appropriate worker_pool through its pool_workqueues. @@ -330,6 +340,7 @@ struct workqueue_struct {/* hot fields used during command issue, aligned to cacheline */ unsigned int flags____cacheline_aligned ;/* WQ: WQ_* flags */ struct pool_workqueue __percpu __rcu **cpu_pwq;/* I: per-cpu pwqs */ +struct wq_node_nr_active *node_nr_active[];/* I: per-node nr_active */ };static struct kmem_cache *pwq_cache;@@ -1425,6 +1436,31 @@ work_func_t wq_worker_last_func(struct task_struct *task)return worker->last_func; } +/** +* wq_node_nr_active - Determine wq_node_nr_active to use +* @wq: workqueue of interest +* @node: NUMA node, can be %NUMA_NO_NODE +* +* Determine wq_node_nr_active to use for @wq on @node. Returns: +* +* - %NULL for per-cpu workqueues as they don't need to use shared nr_active. +* +* - node_nr_active[nr_node_ids] if @node is %NUMA_NO_NODE. +* +* - Otherwise, node_nr_active[@node]. +*/ +static struct wq_node_nr_active *wq_node_nr_active (struct workqueue_struct *wq, +int node) +{ +if (!(wq->flags &WQ_UNBOUND )) +return NULL ; + +if (node ==NUMA_NO_NODE ) + node = nr_node_ids; + +return wq->node_nr_active[node]; +} +/** * get_pwq - get an extra reference on the specified pool_workqueue * @pwq: pool_workqueue to get @@ -1506,12 +1542,17 @@ static bool pwq_activate_work(struct pool_workqueue *pwq, struct work_struct *work) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna;lockdep_assert_held (&pool->lock);if (!(*work_data_bits (work) &WORK_STRUCT_INACTIVE ))return false; + nna =wq_node_nr_active (pwq->wq, pool->node); +if (nna) +atomic_inc (&nna->nr); + pwq->nr_active++;__pwq_activate_work (pwq, work);return true;@@ -1528,14 +1569,18 @@ static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq) {struct workqueue_struct *wq = pwq->wq;struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (wq, pool->node);bool obtained;lockdep_assert_held (&pool->lock); obtained = pwq->nr_active <READ_ONCE (wq->max_active); -if (obtained) +if (obtained) { pwq->nr_active++; +if (nna) +atomic_inc (&nna->nr); + }return obtained; }@@ -1572,10 +1617,26 @@ static bool pwq_activate_first_inactive(struct pool_workqueue *pwq)static void pwq_dec_nr_active (struct pool_workqueue *pwq) {struct worker_pool *pool = pwq->pool; +struct wq_node_nr_active *nna =wq_node_nr_active (pwq->wq, pool->node);lockdep_assert_held (&pool->lock); +/* +* @pwq->nr_active should be decremented for both percpu and unbound +* workqueues. +*/ pwq->nr_active--; + +/* +* For a percpu workqueue, it's simple. Just need to kick the first +* inactive work item on @pwq itself. +*/ +if (!nna) { +pwq_activate_first_inactive (pwq); +return ; + } + +atomic_dec (&nna->nr);pwq_activate_first_inactive (pwq); }@@ -4039,11 +4100,63 @@ static void wq_free_lockdep(struct workqueue_struct *wq) }#endif +static void free_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +int node; + +for_each_node (node) { +kfree (nna_ar[node]); + nna_ar[node] =NULL ; + } + +kfree (nna_ar[nr_node_ids]); + nna_ar[nr_node_ids] =NULL ; +} + +static void init_node_nr_active (struct wq_node_nr_active *nna) +{ +atomic_set (&nna->nr,0 ); +} + +/* +* Each node's nr_active counter will be accessed mostly from its own node and +* should be allocated in the node. +*/ +static int alloc_node_nr_active (struct wq_node_nr_active **nna_ar) +{ +struct wq_node_nr_active *nna; +int node; + +for_each_node (node) { + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL , node); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[node] = nna; + } + +/* [nr_node_ids] is used as the fallback */ + nna =kzalloc_node (sizeof (*nna),GFP_KERNEL ,NUMA_NO_NODE ); +if (!nna) + goto err_free; +init_node_nr_active (nna); + nna_ar[nr_node_ids] = nna; + +return 0 ; + +err_free: +free_node_nr_active (nna_ar); +return -ENOMEM ; +} +static void rcu_free_wq (struct rcu_head *rcu) {struct workqueue_struct *wq = container_of(rcu, struct workqueue_struct, rcu); +if (wq->flags &WQ_UNBOUND ) +free_node_nr_active (wq->node_nr_active); +wq_free_lockdep (wq);free_percpu (wq->cpu_pwq);free_workqueue_attrs (wq->unbound_attrs);@@ -4785,7 +4898,8 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, {va_list args;struct workqueue_struct *wq; -int len; +size_t wq_size; +int name_len;/* * Unbound && max_active == 1 used to imply ordered, which is no longer @@ -4801,7 +4915,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, flags |=WQ_UNBOUND ;/* allocate wq and format name */ - wq =kzalloc (sizeof (*wq),GFP_KERNEL ); +if (flags &WQ_UNBOUND ) + wq_size =struct_size (wq, node_nr_active, nr_node_ids +1 ); +else + wq_size =sizeof (*wq); + + wq =kzalloc (wq_size,GFP_KERNEL );if (!wq)return NULL ;@@ -4812,11 +4931,12 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, }va_start (args, max_active); - len =vsnprintf (wq->name,sizeof (wq->name), fmt, args); + name_len =vsnprintf (wq->name,sizeof (wq->name), fmt, args);va_end (args); -if (len >=WQ_NAME_LEN ) -pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , wq->name); +if (name_len >=WQ_NAME_LEN ) +pr_warn_once ("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n" , + wq->name); max_active = max_active ?:WQ_DFL_ACTIVE ; max_active =wq_clamp_max_active (max_active, flags, wq->name);@@ -4835,8 +4955,13 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,wq_init_lockdep (wq);INIT_LIST_HEAD (&wq->list); +if (flags &WQ_UNBOUND ) { +if (alloc_node_nr_active (wq->node_nr_active) <0 ) + goto err_unreg_lockdep; + } +if (alloc_and_link_pwqs (wq) <0 ) - goto err_unreg_lockdep; + goto err_free_node_nr_active;if (wq_online &&init_rescuer (wq) <0 ) goto err_destroy;@@ -4861,6 +4986,9 @@ struct workqueue_struct *alloc_workqueue(const char *fmt,return wq; +err_free_node_nr_active: +if (wq->flags &WQ_UNBOUND ) +free_node_nr_active (wq->node_nr_active); err_unreg_lockdep:wq_unregister_lockdep (wq);wq_free_lockdep (wq);Signature · 1 line
2.43.0
I just bisected a crash that I see when booting several different architectures in QEMU in -next to this change as commit 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues"). For example, with arm64 virtconfig (I also see it with ARCH=arm multi_v7_defconfig and ARCH=riscv defconfig):
$ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux- virtconfig Image.gz
$ qemu-system-aarch64 \
-display none \
-nodefaults \
-cpu max,pauth-impdef=true \
-machine virt,gic-version=max,virtualization=true \
-append 'console=ttyAMA0 earlycon' \
-kernel arch/arm64/boot/Image.gz \
-initrd rootfs.cpio \
-m 512m \
-serial mon:stdio
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
[ 0.000000] Linux version 6.7.0-09946-g5797b1c18919 ([email protected]) (aarch64-linux-gcc (GCC) 13.2.0, GNU ld (GNU Binutils) 2.41) #1 SMP PREEMPT Tue Jan 30 10:10:57 MST 2024
...
[ 0.000000] Unable to handle kernel paging request at virtual address ffff000021c0b380
[ 0.000000] Mem abort info:
[ 0.000000] ESR = 0x0000000096000006
[ 0.000000] EC = 0x25: DABT (current EL), IL = 32 bits
[ 0.000000] SET = 0, FnV = 0
[ 0.000000] EA = 0, S1PTW = 0
[ 0.000000] FSC = 0x06: level 2 translation fault
[ 0.000000] Data abort info:
[ 0.000000] ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[ 0.000000] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 0.000000] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 0.000000] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000413b1000
[ 0.000000] [ffff000021c0b380] pgd=180000005fff7003, p4d=180000005fff7003, pud=180000005fff6003, pmd=0000000000000000
[ 0.000000] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.7.0-09946-g5797b1c18919 #1
[ 0.000000] Hardware name: linux,dummy-virt (DT)
[ 0.000000] pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 0.000000] pc : wq_update_node_max_active+0x48/0x1d8
[ 0.000000] lr : apply_wqattrs_commit+0x160/0x180
[ 0.000000] sp : ffffda8d476b3be0
[ 0.000000] x29: ffffda8d476b3be0 x28: ffff000001c0d600 x27: 0000000000000000
[ 0.000000] x26: ffff000001c0d6c0 x25: 0000000000000001 x24: 0000000000000200
[ 0.000000] x23: 00000000ffffffff x22: ffffda8d476b9c40 x21: 0000000000000008
[ 0.000000] x20: ffffda8d476b9a40 x19: ffff000001c0b360 x18: ffff00001feebed0
[ 0.000000] x17: 0000000000c65c70 x16: ffff00001feebb28 x15: fffffc0000070488
[ 0.000000] x14: 0000000000000000 x13: 0000000000000000 x12: ffff00001feebb28
[ 0.000000] x11: 0000000000000001 x10: ffff000001c0b388 x9 : 0000000000000000
[ 0.000000] x8 : 0000000000000000 x7 : 0000000000000000 x6 : ffff000001c0d600
[ 0.000000] x5 : ffff000001c0d600 x4 : ffff000001c0e880 x3 : ffff000001c0d600
[ 0.000000] x2 : ffff000001c0b388 x1 : ffffda8d476b9000 x0 : 0000000003ffffff
[ 0.000000] Call trace:
[ 0.000000] wq_update_node_max_active+0x48/0x1d8
[ 0.000000] apply_wqattrs_commit+0x160/0x180
[ 0.000000] apply_workqueue_attrs_locked+0x50/0x84
[ 0.000000] alloc_workqueue+0x588/0x6c8
[ 0.000000] workqueue_init_early+0x480/0x554
[ 0.000000] start_kernel+0x240/0x5e8
[ 0.000000] __primary_switched+0xb8/0xc0
[ 0.000000] Code: f9418033 d000a081 9100a262 f90037e2 (f8607840)
[ 0.000000] ---[ end trace 0000000000000000 ]---
[ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
[ 0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---If there is any more information I can provide or patches I can test, I am more than happy to do so.
Cheers,
Nathan
# bad: [41d66f96d0f15a0a2ad6fa2208f6bac1a66cbd52] Add linux-next specific files for 20240130
# good: [41bccc98fb7931d63d03f326a746ac4d429c1dd3] Linux 6.8-rc2
git bisect start '41d66f96d0f15a0a2ad6fa2208f6bac1a66cbd52' '41bccc98fb7931d63d03f326a746ac4d429c1dd3'
# good: [f3f89885646036e16b325aea597fc9f375f1a56a] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good f3f89885646036e16b325aea597fc9f375f1a56a
# good: [8042d32dd6c3730b0b4c8c9c811e204ed9f5f829] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect good 8042d32dd6c3730b0b4c8c9c811e204ed9f5f829
# bad: [f0edba72fe5ae932877f49796aaef8adf1a2eb8c] Merge branch 'togreg' of git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git
git bisect bad f0edba72fe5ae932877f49796aaef8adf1a2eb8c
# bad: [549632942a27cc3987473f8a8629200bc2ab0734] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/westeri/thunderbolt.git
git bisect bad 549632942a27cc3987473f8a8629200bc2ab0734
# good: [9eb48e8d465d67362dac65963979e65cb2ad7634] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/dennis/percpu.git
git bisect good 9eb48e8d465d67362dac65963979e65cb2ad7634
# good: [bc83a87759cabbf6f3366568e44bda088b315204] dt-bindings: usb: dwc3: Add snps,host-vbus-glitches-quirk avoid vbus glitch git bisect good bc83a87759cabbf6f3366568e44bda088b315204
# bad: [2c0ea2f8eb6140767fae8d01a30ce45dcf4ead85] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git
git bisect bad 2c0ea2f8eb6140767fae8d01a30ce45dcf4ead85
# good: [a045a272d887575da17ad86d6573e82871b50c27] workqueue: Move pwq->max_active to wq->max_active
git bisect good a045a272d887575da17ad86d6573e82871b50c27
# good: [9f66cff212bb3c1cd25996aaa0dfd0c9e9d8baab] workqueue: RCU protect wq->dfl_pwq and implement accessors for it git bisect good 9f66cff212bb3c1cd25996aaa0dfd0c9e9d8baab
# good: [91ccc6e7233bb10a9c176aa4cc70d6f432a441a5] workqueue: Introduce struct wq_node_nr_active
git bisect good 91ccc6e7233bb10a9c176aa4cc70d6f432a441a5
# bad: [07daa99b7fd7adfffa22180184e39ec124e73013] tools/workqueue/wq_dump.py: Add node_nr/max_active dump
git bisect bad 07daa99b7fd7adfffa22180184e39ec124e73013
# bad: [5797b1c18919cd9c289ded7954383e499f729ce0] workqueue: Implement system-wide nr_active enforcement for unbound workqueues git bisect bad 5797b1c18919cd9c289ded7954383e499f729ce0
# first bad commit: [5797b1c18919cd9c289ded7954383e499f729ce0] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Tejun Heo
maintainerRe: [PATCH v4 08/10] workqueue: Introduce struct wq_node_nr_active
Hello,
Thanks for the report. Can you please see whether the following patch fixes the problem?
http://lkml.kernel.org/r/[email protected]
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 9/10 workqueue: Implement system-wide nr_active enforcement for unbound workqueues
In reply to Tejun HeoA pool_workqueue (pwq) represents the connection between a workqueue and a worker_pool. One of the roles that a pwq plays is enforcement of the max_active concurrency limit. Before 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues"), there was one pwq per each CPU for per-cpu workqueues and per each NUMA node for unbound workqueues, which was a natural result of per-cpu workqueues being served by per-cpu pools and unbound by per-NUMA pools.
In terms of max_active enforcement, this was, while not perfect, workable. For per-cpu workqueues, it was fine. For unbound, it wasn't great in that NUMA machines would get max_active that's multiplied by the number of nodes but didn't cause huge problems because NUMA machines are relatively rare and the node count is usually pretty low.
However, cache layouts are more complex now and sharing a worker pool across a whole node didn't really work well for unbound workqueues. Thus, a series of commits culminating on 8639ecebc9b1 ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") implemented more flexible affinity mechanism for unbound workqueues which enables using e.g. last-level-cache aligned pools. In the process, 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") made unbound workqueues use per-cpu pwqs like per-cpu workqueues.
While the change was necessary to enable more flexible affinity scopes, this came with the side effect of blowing up the effective max_active for unbound workqueues. Before, the effective max_active for unbound workqueues was multiplied by the number of nodes. After, by the number of CPUs.
636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") claims that this should generally be okay. It is okay for users which self-regulates concurrency level which are the vast majority; however, there are enough use cases which actually depend on max_active to prevent the level of concurrency from going bonkers including several IO handling workqueues that can issue a work item for each in-flight IO. With targeted benchmarks, the misbehavior can easily be exposed as reported in
http://lkml.kernel.org/r/dbu6wiwu3sdhmhikb2w6lns7b27gbobfavhjj57kwi2quafgwl@htjcc5oikcr3.
Unfortunately, there is no way to express what these use cases need using per-cpu max_active. A CPU may issue most of in-flight IOs, so we don't want to set max_active too low but as soon as we increase max_active a bit, we can end up with unreasonable number of in-flight work items when many CPUs issue IOs at the same time. ie. The acceptable lowest max_active is higher than the acceptable highest max_active.
Ideally, max_active for an unbound workqueue should be system-wide so that the users can regulate the total level of concurrency regardless of node and cache layout. The reasons workqueue hasn't implemented that yet are:
- One max_active enforcement decouples from pool boundaires, chaining execution after a work item finishes requires inter-pool operations which would require lock dancing, which is nasty.
- Sharing a single nr_active count across the whole system can be pretty expensive on NUMA machines.
- Per-pwq enforcement had been more or less okay while we were using per-node pools.
It looks like we no longer can avoid decoupling max_active enforcement from pool boundaries. This patch implements system-wide nr_active mechanism with the following design characteristics:
- To avoid sharing a single counter across multiple nodes, the configured max_active is split across nodes according to the proportion of each workqueue's online effective CPUs per node. e.g. A node with twice more online effective CPUs will get twice higher portion of max_active.
- Workqueue used to be able to process a chain of interdependent work items which is as long as max_active. We can't do this anymore as max_active is distributed across the nodes. Instead, a new parameter min_active is introduced which determines the minimum level of concurrency within a node regardless of how max_active distribution comes out to be.
It is set to the smaller of max_active and WQ_DFL_MIN_ACTIVE which is 8. This can lead to higher effective max_weight than configured and also deadlocks if a workqueue was depending on being able to handle chains of interdependent work items that are longer than 8.
I believe these should be fine given that the number of CPUs in each NUMA node is usually higher than 8 and work item chain longer than 8 is pretty unlikely. However, if these assumptions turn out to be wrong, we'll need to add an interface to adjust min_active.
- Each unbound wq has an array of struct wq_node_nr_active which tracks per-node nr_active. When its pwq wants to run a work item, it has to obtain the matching node's nr_active. If over the node's max_active, the pwq is queued on wq_node_nr_active->pending_pwqs. As work items finish, the completion path round-robins the pending pwqs activating the first inactive work item of each, which involves some pool lock dancing and kicking other pools. It's not the simplest code but doesn't look too bad.
v3: - wq_node_max_active() used to calculate per-node max_active on the fly
based on system-wide CPU online states. Lai pointed out that this can lead to skewed distributions for workqueues with restricted cpumasks. Update the max_active distribution to use per-workqueue effective online CPU counts instead of system-wide and cache the calculation results in node_nr_active->max.
v2: - wq->min/max_active now uses WRITE/READ_ONCE() as suggested by Lai.
--- include/linux/workqueue.h | 35 ++++- kernel/workqueue.c | 321 ++++++++++++++++++++++++++++++++++---- 2 files changed, 323 insertions(+), 33 deletions(-)
include/linux/workqueue.h +32 -3
@@ -391,6 +391,13 @@ enum {WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_UNBOUND_MAX_ACTIVE =WQ_MAX_ACTIVE ,WQ_DFL_ACTIVE =WQ_MAX_ACTIVE /2 , + +/* +* Per-node default cap on min_active. Unless explicitly set, min_active +* is set to min(max_active, WQ_DFL_MIN_ACTIVE). For more details, see +* workqueue_struct->min_active definition. +*/ + WQ_DFL_MIN_ACTIVE = 8, };/*
@@ -433,11 +440,33 @@ extern struct workqueue_struct *system_freezable_power_efficient_wq;* alloc_workqueue - allocate a workqueue * @fmt: printf format for the name of the workqueue * @flags: WQ_* flags -* @max_active: max in-flight work items per CPU, 0 for default +* @max_active: max in-flight work items, 0 for default * remaining args: args for @fmt * -* Allocate a workqueue with the specified parameters. For detailed -* information on WQ_* flags, please refer to +* For a per-cpu workqueue, @max_active limits the number of in-flight work +* items for each CPU. e.g. @max_active of 1 indicates that each CPU can be +* executing at most one work item for the workqueue. +* +* For unbound workqueues, @max_active limits the number of in-flight work items +* for the whole system. e.g. @max_active of 16 indicates that that there can be +* at most 16 work items executing for the workqueue in the whole system. +* +* As sharing the same active counter for an unbound workqueue across multiple +* NUMA nodes can be expensive, @max_active is distributed to each NUMA node +* according to the proportion of the number of online CPUs and enforced +* independently. +* +* Depending on online CPU distribution, a node may end up with per-node +* max_active which is significantly lower than @max_active, which can lead to +* deadlocks if the per-node concurrency limit is lower than the maximum number +* of interdependent work items for the workqueue. +* +* To guarantee forward progress regardless of online CPU distribution, the +* concurrency limit on every node is guaranteed to be equal to or greater than +* min_active which is set to min(@max_active, %WQ_DFL_MIN_ACTIVE). This means +* that the sum of per-node max_active's may be larger than @max_active. +* +* For detailed information on %WQ_* flags, please refer to * Documentation/core-api/workqueue.rst. * * RETURNS:
kernel/workqueue.c +291 -30
516 lines not on this page · raw message
Signature · 1 line
2.43.0
Lai Jiangshan
reviewerRe: [PATCH 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Hello, Tejun
kernel/workqueue.c void workqueue_set_max_active(struct workqueue_struct *wq, int max_active) · line 5379
Quoted 7 lines · Tejun Heo
kernel/workqueue.c +3
-0@@ -5121,6 +5374,9 @@ void workqueue_set_max_active(struct workqueue_struct *wq, int max_active) wq->flags &= ~__WQ_ORDERED; wq->saved_max_active = max_active; +if (wq->flags &WQ_UNBOUND ) + wq->saved_min_active =min (wq->saved_min_active, max_active); +
wq_update_node_max_active() must be also called here.
Thanks
Lai
Quoted 3 lines
kernel/workqueue.c +0
-0wq_adjust_max_active (wq);mutex_unlock (&wq->mutex);
Tejun Heo
maintainerPATCH v4 9/10 workqueue: Implement system-wide nr_active enforcement for unbound workqueues
From 5797b1c18919cd9c289ded7954383e499f729ce0 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Mon, 29 Jan 2024 08:11:25 -1000
A pool_workqueue (pwq) represents the connection between a workqueue and a worker_pool. One of the roles that a pwq plays is enforcement of the max_active concurrency limit. Before 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues"), there was one pwq per each CPU for per-cpu workqueues and per each NUMA node for unbound workqueues, which was a natural result of per-cpu workqueues being served by per-cpu pools and unbound by per-NUMA pools.
In terms of max_active enforcement, this was, while not perfect, workable. For per-cpu workqueues, it was fine. For unbound, it wasn't great in that NUMA machines would get max_active that's multiplied by the number of nodes but didn't cause huge problems because NUMA machines are relatively rare and the node count is usually pretty low.
However, cache layouts are more complex now and sharing a worker pool across a whole node didn't really work well for unbound workqueues. Thus, a series of commits culminating on 8639ecebc9b1 ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") implemented more flexible affinity mechanism for unbound workqueues which enables using e.g. last-level-cache aligned pools. In the process, 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") made unbound workqueues use per-cpu pwqs like per-cpu workqueues.
While the change was necessary to enable more flexible affinity scopes, this came with the side effect of blowing up the effective max_active for unbound workqueues. Before, the effective max_active for unbound workqueues was multiplied by the number of nodes. After, by the number of CPUs.
636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") claims that this should generally be okay. It is okay for users which self-regulates concurrency level which are the vast majority; however, there are enough use cases which actually depend on max_active to prevent the level of concurrency from going bonkers including several IO handling workqueues that can issue a work item for each in-flight IO. With targeted benchmarks, the misbehavior can easily be exposed as reported in
http://lkml.kernel.org/r/dbu6wiwu3sdhmhikb2w6lns7b27gbobfavhjj57kwi2quafgwl@htjcc5oikcr3.
Unfortunately, there is no way to express what these use cases need using per-cpu max_active. A CPU may issue most of in-flight IOs, so we don't want to set max_active too low but as soon as we increase max_active a bit, we can end up with unreasonable number of in-flight work items when many CPUs issue IOs at the same time. ie. The acceptable lowest max_active is higher than the acceptable highest max_active.
Ideally, max_active for an unbound workqueue should be system-wide so that the users can regulate the total level of concurrency regardless of node and cache layout. The reasons workqueue hasn't implemented that yet are:
- One max_active enforcement decouples from pool boundaires, chaining execution after a work item finishes requires inter-pool operations which would require lock dancing, which is nasty.
- Sharing a single nr_active count across the whole system can be pretty expensive on NUMA machines.
- Per-pwq enforcement had been more or less okay while we were using per-node pools.
It looks like we no longer can avoid decoupling max_active enforcement from pool boundaries. This patch implements system-wide nr_active mechanism with the following design characteristics:
- To avoid sharing a single counter across multiple nodes, the configured max_active is split across nodes according to the proportion of each workqueue's online effective CPUs per node. e.g. A node with twice more online effective CPUs will get twice higher portion of max_active.
- Workqueue used to be able to process a chain of interdependent work items which is as long as max_active. We can't do this anymore as max_active is distributed across the nodes. Instead, a new parameter min_active is introduced which determines the minimum level of concurrency within a node regardless of how max_active distribution comes out to be.
It is set to the smaller of max_active and WQ_DFL_MIN_ACTIVE which is 8. This can lead to higher effective max_weight than configured and also deadlocks if a workqueue was depending on being able to handle chains of interdependent work items that are longer than 8.
I believe these should be fine given that the number of CPUs in each NUMA node is usually higher than 8 and work item chain longer than 8 is pretty unlikely. However, if these assumptions turn out to be wrong, we'll need to add an interface to adjust min_active.
- Each unbound wq has an array of struct wq_node_nr_active which tracks per-node nr_active. When its pwq wants to run a work item, it has to obtain the matching node's nr_active. If over the node's max_active, the pwq is queued on wq_node_nr_active->pending_pwqs. As work items finish, the completion path round-robins the pending pwqs activating the first inactive work item of each, which involves some pool lock dancing and kicking other pools. It's not the simplest code but doesn't look too bad.
v4: - wq_adjust_max_active() updated to invoke wq_update_node_max_active().
- wq_adjust_max_active() is now protected by wq->mutex instead of wq_pool_mutex.
v3: - wq_node_max_active() used to calculate per-node max_active on the fly
based on system-wide CPU online states. Lai pointed out that this can lead to skewed distributions for workqueues with restricted cpumasks. Update the max_active distribution to use per-workqueue effective online CPU counts instead of system-wide and cache the calculation results in node_nr_active->max.
v2: - wq->min/max_active now uses WRITE/READ_ONCE() as suggested by Lai.
--- include/linux/workqueue.h | 35 +++- kernel/workqueue.c | 341 ++++++++++++++++++++++++++++++++++---- 2 files changed, 341 insertions(+), 35 deletions(-)
include/linux/workqueue.h +32 -3
@@ -398,6 +398,13 @@ enum wq_consts {WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_UNBOUND_MAX_ACTIVE =WQ_MAX_ACTIVE ,WQ_DFL_ACTIVE =WQ_MAX_ACTIVE /2 , + +/* +* Per-node default cap on min_active. Unless explicitly set, min_active +* is set to min(max_active, WQ_DFL_MIN_ACTIVE). For more details, see +* workqueue_struct->min_active definition. +*/ + WQ_DFL_MIN_ACTIVE = 8, };/*
@@ -440,11 +447,33 @@ extern struct workqueue_struct *system_freezable_power_efficient_wq;* alloc_workqueue - allocate a workqueue * @fmt: printf format for the name of the workqueue * @flags: WQ_* flags -* @max_active: max in-flight work items per CPU, 0 for default +* @max_active: max in-flight work items, 0 for default * remaining args: args for @fmt * -* Allocate a workqueue with the specified parameters. For detailed -* information on WQ_* flags, please refer to +* For a per-cpu workqueue, @max_active limits the number of in-flight work +* items for each CPU. e.g. @max_active of 1 indicates that each CPU can be +* executing at most one work item for the workqueue. +* +* For unbound workqueues, @max_active limits the number of in-flight work items +* for the whole system. e.g. @max_active of 16 indicates that that there can be +* at most 16 work items executing for the workqueue in the whole system. +* +* As sharing the same active counter for an unbound workqueue across multiple +* NUMA nodes can be expensive, @max_active is distributed to each NUMA node +* according to the proportion of the number of online CPUs and enforced +* independently. +* +* Depending on online CPU distribution, a node may end up with per-node +* max_active which is significantly lower than @max_active, which can lead to +* deadlocks if the per-node concurrency limit is lower than the maximum number +* of interdependent work items for the workqueue. +* +* To guarantee forward progress regardless of online CPU distribution, the +* concurrency limit on every node is guaranteed to be equal to or greater than +* min_active which is set to min(@max_active, %WQ_DFL_MIN_ACTIVE). This means +* that the sum of per-node max_active's may be larger than @max_active. +* +* For detailed information on %WQ_* flags, please refer to * Documentation/core-api/workqueue.rst. * * RETURNS:
kernel/workqueue.c +309 -32
541 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Dear All,
Quoted 106 lines · Tejun Heo
From: Tejun Heo <[email protected]>
Date: Mon, 29 Jan 2024 08:11:25 -1000A
pool_workqueue(pwq) represents the connection between a workqueue and aworker_pool. One of the roles that a pwq plays is enforcement of themax_activeconcurrency limit. Before636b927eba5b("workqueue: Make unbound workqueues to use per-cpupool_workqueues"), there was one pwq per each CPU for per-cpu workqueues and per each NUMA node for unbound workqueues, which was a natural result of per-cpu workqueues being served by per-cpu pools and unbound by per-NUMA pools.In terms of
max_activeenforcement, this was, while not perfect, workable. For per-cpu workqueues, it was fine. For unbound, it wasn't great in that NUMA machines would getmax_activethat's multiplied by the number of nodes but didn't cause huge problems because NUMA machines are relatively rare and the node count is usually pretty low.However, cache layouts are more complex now and sharing a worker pool across a whole node didn't really work well for unbound workqueues. Thus, a series of commits culminating on
8639ecebc9b1("workqueue: Make unbound workqueues to use per-cpupool_workqueues") implemented more flexible affinity mechanism for unbound workqueues which enables using e.g. last-level-cache aligned pools. In the process,636b927eba5b("workqueue: Make unbound workqueues to use per-cpupool_workqueues") made unbound workqueues use per-cpu pwqs like per-cpu workqueues.While the change was necessary to enable more flexible affinity scopes, this came with the side effect of blowing up the effective
max_activefor unbound workqueues. Before, the effectivemax_activefor unbound workqueues was multiplied by the number of nodes. After, by the number of CPUs.
636b927eba5b("workqueue: Make unbound workqueues to use per-cpupool_workqueues") claims that this should generally be okay. It is okay for users which self-regulates concurrency level which are the vast majority; however, there are enough use cases which actually depend onmax_activeto prevent the level of concurrency from going bonkers including several IO handling workqueues that can issue a work item for each in-flight IO. With targeted benchmarks, the misbehavior can easily be exposed as reported in
http://lkml.kernel.org/r/dbu6wiwu3sdhmhikb2w6lns7b27gbobfavhjj57kwi2quafgwl@htjcc5oikcr3.Unfortunately, there is no way to express what these use cases need using per-cpu
max_active. A CPU may issue most of in-flight IOs, so we don't want to setmax_activetoo low but as soon as we increasemax_activea bit, we can end up with unreasonable number of in-flight work items when many CPUs issue IOs at the same time. ie. The acceptable lowestmax_activeis higher than the acceptable highestmax_active.Ideally,
max_activefor an unbound workqueue should be system-wide so that the users can regulate the total level of concurrency regardless of node and cache layout. The reasons workqueue hasn't implemented that yet are:- One
max_activeenforcement decouples from pool boundaires, chaining execution after a work item finishes requires inter-pool operations which would require lock dancing, which is nasty.- Sharing a single
nr_activecount across the whole system can be pretty expensive on NUMA machines.- Per-pwq enforcement had been more or less okay while we were using per-node pools.
It looks like we no longer can avoid decoupling
max_activeenforcement from pool boundaries. This patch implements system-widenr_activemechanism with the following design characteristics:- To avoid sharing a single counter across multiple nodes, the configured
max_activeis split across nodes according to the proportion of each workqueue's online effective CPUs per node. e.g. A node with twice more online effective CPUs will get twice higher portion ofmax_active.- Workqueue used to be able to process a chain of interdependent work items which is as long as
max_active. We can't do this anymore asmax_activeis distributed across the nodes. Instead, a new parametermin_activeis introduced which determines the minimum level of concurrency within a node regardless of howmax_activedistribution comes out to be.It is set to the smaller of
max_activeandWQ_DFL_MIN_ACTIVEwhich is 8. This can lead to higher effectivemax_weightthan configured and also deadlocks if a workqueue was depending on being able to handle chains of interdependent work items that are longer than 8.I believe these should be fine given that the number of CPUs in each NUMA node is usually higher than 8 and work item chain longer than 8 is pretty unlikely. However, if these assumptions turn out to be wrong, we'll need to add an interface to adjust
min_active.- Each unbound wq has an array of struct
wq_node_nr_activewhich tracks per-nodenr_active. When its pwq wants to run a work item, it has to obtain the matching node'snr_active. If over the node'smax_active, the pwq is queued onwq_node_nr_active->pending_pwqs. As work items finish, the completion path round-robins the pending pwqs activating the first inactive work item of each, which involves some pool lock dancing and kicking other pools. It's not the simplest code but doesn't look too bad.v4: -
wq_adjust_max_active()updated to invokewq_update_node_max_active().-
wq_adjust_max_active()is now protected bywq->mutexinstead ofwq_pool_mutex.v3: -
wq_node_max_active()used to calculate per-nodemax_activeon the fly
based on system-wide CPU online states. Lai pointed out that this can lead to skewed distributions for workqueues with restricted cpumasks. Update themax_activedistribution to use per-workqueue effective online CPU counts instead of system-wide and cache the calculation results innode_nr_active->max.
Quoted 7 lines
v2: - wq->min/max_active now uses WRITE/READ_ONCE() as suggested by Lai.
Signed-off-by Tejun Heo <[email protected]>Reported-by Naohiro Aota <[email protected]>Link http://lkml.kernel.org/r/dbu6wiwu3sdhmhikb2w6lns7b27gbobfavhjj57kwi2quafgwl@htjcc5oikcr3 open hereFixes 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues")Reviewed-by Lai Jiangshan <[email protected]>
This patch landed in linux-next from 20240130 as commit 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues"). Unfortunately it causes the following regression on ARM64 RK3568 SoC based Odroid-M1 board:
Unable to handle kernel paging request at virtual address ffff0002100296e0
Mem abort info:
ESR = 0x0000000096000005
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x05: level 1 translation fault
Data abort info:
ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
CM = 0, WnR = 0, TnD = 0, TagAccess = 0
GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000000255a000 [ffff0002100296e0] pgd=18000001ffff7003, p4d=18000001ffff7003, pud=0000000000000000
Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP Modules linked in: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc2-next-20240130+ #14392 Hardware name: Hardkernel ODROID-M1 (DT) pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : wq_update_node_max_active+0x50/0x1fc lr : wq_update_node_max_active+0x1f0/0x1fc ... Call trace: wq_update_node_max_active+0x50/0x1fc apply_wqattrs_commit+0xf0/0x114 apply_workqueue_attrs_locked+0x58/0xa0 alloc_workqueue+0x5ac/0x774 workqueue_init_early+0x460/0x540 start_kernel+0x258/0x684 __primary_switched+0xb8/0xc0 Code: 9100a273 35000d01 53067f00 d0016dc1 (f8607a60) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Attempted to kill the idle task! ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
This wasn't trivial to bisect, because next-20240130 suffers from other regressions:
https://lore.kernel.org/all/[email protected]/
but reverting this change and the ones mentioned in that thread fixes all the issues observed on top of today's linux-next release. Let me know if there is anything I can do to help fixing this issue.
Quoted 6 lines
--- include/linux/workqueue.h | 35 +++- kernel/workqueue.c | 341 ++++++++++++++++++++++++++++++++++---- 2 files changed, 341 insertions(+), 35 deletions(-)...
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Tejun Heo
maintainerRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Hello,
Thanks for the report. Can you please test whether the following patch fixes the problem?
----- 8< -----
From: Tejun Heo <[email protected]>
Subject: workqueue: Fix crash due to premature NUMA topology access on some archs
System workqueues are allocated early during boot from workqueue_init_early(). While allocating unbound workqueues, wq_update_node_max_active() is invoked from apply_workqueue_attrs() and accesses NUMA topology information - cpumask_of_node() and cpu_to_node().
At this point, topology information is not initialized yet and on arm and some other archs, it leads to an oops like the following:
Unable to handle kernel paging request at virtual address ffff0002100296e0
Mem abort info:
ESR = 0x0000000096000005
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x05: level 1 translation fault
Data abort info:
ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
CM = 0, WnR = 0, TnD = 0, TagAccess = 0
GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000000255a000
[ffff0002100296e0] pgd=18000001ffff7003, p4d=18000001ffff7003,
pud=0000000000000000
Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc2-next-20240130+ #14392
Hardware name: Hardkernel ODROID-M1 (DT)
pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : wq_update_node_max_active+0x50/0x1fc
lr : wq_update_node_max_active+0x1f0/0x1fc
...
Call trace:
wq_update_node_max_active+0x50/0x1fc
apply_wqattrs_commit+0xf0/0x114
apply_workqueue_attrs_locked+0x58/0xa0
alloc_workqueue+0x5ac/0x774
workqueue_init_early+0x460/0x540
start_kernel+0x258/0x684
__primary_switched+0xb8/0xc0
Code: 9100a273 35000d01 53067f00 d0016dc1 (f8607a60)
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---Fix it by initializing wq->node_nr_active[].max to WQ_DFL_MIN_ACTIVE on allocation and making wq_update_node_max_active() noop until workqueue_init_topology(). Note that workqueue_init_topology() invokes wq_update_node_max_active() on all unbound workqueues, so the end result is still the same.
--- kernel/workqueue.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/workqueue.c +8 -0
@@ -386,6 +386,8 @@ static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { [WQ_AFFN_SYSTEM] = "system", }; +static bool wq_topo_initialized = false; +/* * Per-cpu work items which run for longer than the following threshold are * automatically considered CPU intensive and excluded from concurrency
@@ -1510,6 +1512,9 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); +if (!wq_topo_initialized) +return ; +if (!cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ;
@@ -4356,6 +4361,7 @@ static void free_node_nr_active(struct wq_node_nr_active **nna_ar)static void init_node_nr_active (struct wq_node_nr_active *nna) { + nna->max =WQ_DFL_MIN_ACTIVE ;atomic_set (&nna->nr,0 );raw_spin_lock_init (&nna->lock);INIT_LIST_HEAD (&nna->pending_pwqs);
@@ -7400,6 +7406,8 @@ void __init workqueue_init_topology(void)init_pod_type (&wq_pod_types[WQ_AFFN_CACHE ], cpus_share_cache);init_pod_type (&wq_pod_types[WQ_AFFN_NUMA ], cpus_share_numa); + wq_topo_initialized = true; +mutex_lock (&wq_pool_mutex);/*
Re: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Hi Tejun,
Quoted 4 lines · Tejun Heo
Hello,
Thanks for the report. Can you please test whether the following patch fixes the problem?
I just tested this change on top of 5797b1c18919 but it does not appear to resolve the issue for any of the three configurations that I tested.
Cheers,
Nathan
Quoted 102 lines
----- 8< -----
From: Tejun Heo <[email protected]>
Subject: workqueue: Fix crash due to premature NUMA topology access on some archsSystem workqueues are allocated early during boot from
workqueue_init_early(). While allocating unbound workqueues,wq_update_node_max_active()is invoked fromapply_workqueue_attrs()and accesses NUMA topology information -cpumask_of_node()andcpu_to_node().At this point, topology information is not initialized yet and on arm and some other archs, it leads to an oops like the following:
Unable to handle kernel paging request at virtual address ffff0002100296e0 Mem abort info: ESR = 0x0000000096000005 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x05: level 1 translation fault Data abort info: ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000000255a000 [ffff0002100296e0] pgd=18000001ffff7003, p4d=18000001ffff7003, pud=0000000000000000 Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP Modules linked in: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc2-next-20240130+ #14392 Hardware name: Hardkernel ODROID-M1 (DT) pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : wq_update_node_max_active+0x50/0x1fc lr : wq_update_node_max_active+0x1f0/0x1fc ... Call trace: wq_update_node_max_active+0x50/0x1fc apply_wqattrs_commit+0xf0/0x114 apply_workqueue_attrs_locked+0x58/0xa0 alloc_workqueue+0x5ac/0x774 workqueue_init_early+0x460/0x540 start_kernel+0x258/0x684 __primary_switched+0xb8/0xc0 Code: 9100a273 35000d01 53067f00 d0016dc1 (f8607a60) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Attempted to kill the idle task! ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---Fix it by initializing wq->node_nr_active[].max to
WQ_DFL_MIN_ACTIVEon allocation and makingwq_update_node_max_active()noop untilworkqueue_init_topology(). Note thatworkqueue_init_topology()invokeswq_update_node_max_active()on all unbound workqueues, so the end result is still the same.Signed-off-by Tejun Heo <[email protected]>Reported-by Marek Szyprowski <[email protected]>Reported-by Nathan Chancellor <[email protected]>Fixes 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")--- kernel/workqueue.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/workqueue.c +8
-0@@ -386,6 +386,8 @@ static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { [WQ_AFFN_SYSTEM] = "system", }; +static bool wq_topo_initialized = false; +/* * Per-cpu work items which run for longer than the following threshold are * automatically considered CPU intensive and excluded from concurrency @@ -1510,6 +1512,9 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); +if (!wq_topo_initialized) +return ; +if (!cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ;@@ -4356,6 +4361,7 @@ static void free_node_nr_active(struct wq_node_nr_active **nna_ar)static void init_node_nr_active (struct wq_node_nr_active *nna) { + nna->max =WQ_DFL_MIN_ACTIVE ;atomic_set (&nna->nr,0 );raw_spin_lock_init (&nna->lock);INIT_LIST_HEAD (&nna->pending_pwqs);@@ -7400,6 +7406,8 @@ void __init workqueue_init_topology(void)init_pod_type (&wq_pod_types[WQ_AFFN_CACHE ], cpus_share_cache);init_pod_type (&wq_pod_types[WQ_AFFN_NUMA ], cpus_share_numa); + wq_topo_initialized = true; +mutex_lock (&wq_pool_mutex);/*
Tejun Heo
maintainerRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Quoted 10 lines · Nathan Chancellor
Hi Tejun,
Quoted 4 lines · Tejun Heo
Hello,
Thanks for the report. Can you please test whether the following patch fixes the problem?
I just tested this change on top of
5797b1c18919but it does not appear to resolve the issue for any of the three configurations that I tested.
Bummer. Can you map the faulting address to the source line?
Thanks.
Signature · 1 line
tejun
Re: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Nathan Chancellor
Hi Tejun,
Quoted 2 lines · Tejun Heo
Hello,
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Thanks for the report. Can you please test whether the following patch fixes the problem?
I just tested this change on top of
5797b1c18919but it does not appear to resolve the issue for any of the three configurations that I tested.Bummer. Can you map the faulting address to the source line?
Sure, here is the arm64 stacktrace run through scripts/decode_stacktrace.sh, the line numbers correspond to your tree at 5797b1c18919.
[ 0.000000] Unable to handle kernel paging request at virtual address ffff000021c0b380 [ 0.000000] Mem abort info: [ 0.000000] ESR = 0x0000000096000006 [ 0.000000] EC = 0x25: DABT (current EL), IL = 32 bits [ 0.000000] SET = 0, FnV = 0 [ 0.000000] EA = 0, S1PTW = 0 [ 0.000000] FSC = 0x06: level 2 translation fault [ 0.000000] Data abort info: [ 0.000000] ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000 [ 0.000000] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 [ 0.000000] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 0.000000] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000413b1000 [ 0.000000] [ffff000021c0b380] pgd=180000005fff7003, p4d=180000005fff7003, pud=180000005fff6003, pmd=0000000000000000 [ 0.000000] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.7.0-09946-g5797b1c18919 #1 [ 0.000000] Hardware name: linux,dummy-virt (DT) [ 0.000000] pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.000000] pc : wq_update_node_max_active (include/asm-generic/bitops/generic-non-atomic.h:128 include/linux/cpumask.h:504 kernel/workqueue.c:1513) [ 0.000000] lr : apply_wqattrs_commit (kernel/workqueue.c:4838) [ 0.000000] sp : ffff8000814b3be0 [ 0.000000] x29: ffff8000814b3be0 x28: ffff000001c0d600 x27: 0000000000000000 [ 0.000000] x26: ffff000001c0d6c0 x25: 0000000000000001 x24: 0000000000000200 [ 0.000000] x23: 00000000ffffffff x22: ffff8000814b9c40 x21: 0000000000000008 [ 0.000000] x20: ffff8000814b9a40 x19: ffff000001c0b360 x18: ffff00001feebed0 [ 0.000000] x17: 0000000000c65c70 x16: ffff00001feebb28 x15: fffffc0000070488 [ 0.000000] x14: 0000000000000000 x13: 0000000000000000 x12: ffff00001feebb28 [ 0.000000] x11: 0000000000000001 x10: ffff000001c0b388 x9 : 0000000000000000 [ 0.000000] x8 : 0000000000000000 x7 : 0000000000000000 x6 : ffff000001c0d600 [ 0.000000] x5 : ffff000001c0d600 x4 : ffff000001c0e880 x3 : ffff000001c0d600 [ 0.000000] x2 : ffff000001c0b388 x1 : ffff8000814b9000 x0 : 0000000003ffffff [ 0.000000] Call trace: [ 0.000000] wq_update_node_max_active (include/asm-generic/bitops/generic-non-atomic.h:128 include/linux/cpumask.h:504 kernel/workqueue.c:1513) [ 0.000000] apply_wqattrs_commit (kernel/workqueue.c:4838) [ 0.000000] apply_workqueue_attrs_locked (kernel/workqueue.c:4745 kernel/workqueue.c:4864) [ 0.000000] alloc_workqueue (kernel/workqueue.c:4894 kernel/workqueue.c:5015 kernel/workqueue.c:5224) [ 0.000000] workqueue_init_early (kernel/workqueue.c:7210) [ 0.000000] start_kernel (init/main.c:965) [ 0.000000] __primary_switched (arch/arm64/kernel/head.S:524) [ 0.000000] Code: f9418033 d000a081 9100a262 f90037e2 (f8607840) All code ======== 0:* 33 80 41 f9 81 a0 xor -0x5f7e06bf(%rax),%eax <-- trapping instruction 6: 00 d0 add %dl,%al 8: 62 a2 00 91 e2 (bad) d: 37 (bad) e: 00 f9 add %bh,%cl 10: 40 78 60 rex js 0x73 13: f8 clc
Code starting with the faulting instruction
===========================================
0: 40 78 60 rex js 0x63 3: f8 clc [ 0.000000] ---[ end trace 0000000000000000 ]--- [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task! [ 0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
Cheers,
Nathan
Tejun Heo
maintainerRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Quoted 10 lines · Nathan Chancellor
Quoted 9 lines · Tejun Heo
Quoted 8 lines · Nathan Chancellor
Hi Tejun,
Quoted 4 lines · Tejun Heo
Hello,
Thanks for the report. Can you please test whether the following patch fixes the problem?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I just tested this change on top of
5797b1c18919but it does not appear to resolve the issue for any of the three configurations that I tested.Bummer. Can you map the faulting address to the source line?
Sure, here is the arm64 stacktrace run through
scripts/decode_stacktrace.sh, the line numbers correspond to your tree at5797b1c18919.
Ah, I see. How about the following?
Thanks.
kernel/workqueue.c +1 -1
@@ -1510,7 +1510,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); -if (!cpumask_test_cpu (off_cpu, effective)) +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ; total_cpus =cpumask_weight_and (effective, cpu_online_mask);
Signature · 1 line
tejun
Nathan Chancellor
Tested-byRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Quoted 16 lines · Tejun Heo
Quoted 15 lines · Nathan Chancellor
Quoted 13 lines · Tejun Heo
Quoted 10 lines · Nathan Chancellor
Hi Tejun,
Quoted 4 lines · Tejun Heo
Hello,
Thanks for the report. Can you please test whether the following patch fixes the problem?
I just tested this change on top of
5797b1c18919but it does not appear to resolve the issue for any of the three configurations that I tested.Bummer. Can you map the faulting address to the source line?
Quoted 7 lines
Quoted 3 lines
Sure, here is the arm64 stacktrace run through
scripts/decode_stacktrace.sh, the line numbers correspond to your tree at5797b1c18919.Ah, I see. How about the following?
Thanks.
That works for three easy to test configurations that were broken, thanks!
Quoted 16 lines
kernel/workqueue.c +1
-1@@ -1510,7 +1510,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); -if (!cpumask_test_cpu (off_cpu, effective)) +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ; total_cpus =cpumask_weight_and (effective, cpu_online_mask);Signature · 2 lines
-- tejun
Re: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
From 15930da42f8981dc42c19038042947b475b19f47 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 30 Jan 2024 18:55:55 -1000
Subject: [PATCH] workqueue: Don't call cpumask_test_cpu() with -1 CPU in
wq_update_node_max_active()
For wq_update_node_max_active(), @off_cpu of -1 indicates that no CPU is going down. The function was incorrectly calling cpumask_test_cpu() with -1 CPU leading to oopses like the following on some archs:
Unable to handle kernel paging request at virtual address ffff0002100296e0
..
pc : wq_update_node_max_active+0x50/0x1fc
lr : wq_update_node_max_active+0x1f0/0x1fc
...
Call trace:
wq_update_node_max_active+0x50/0x1fc
apply_wqattrs_commit+0xf0/0x114
apply_workqueue_attrs_locked+0x58/0xa0
alloc_workqueue+0x5ac/0x774
workqueue_init_early+0x460/0x540
start_kernel+0x258/0x684
__primary_switched+0xb8/0xc0
Code: 9100a273 35000d01 53067f00 d0016dc1 (f8607a60)
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---Fix it.
---
Applied to wq/for-6.9.
Thanks.
kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/workqueue.c +1 -1
@@ -1510,7 +1510,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); -if (!cpumask_test_cpu (off_cpu, effective)) +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ; total_cpus =cpumask_weight_and (effective, cpu_online_mask);
Signature · 1 line
2.43.0
Marek Szyprowski
Tested-byRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
Quoted 27 lines · Tejun Heo
From: Tejun Heo <[email protected]>
Date: Tue, 30 Jan 2024 18:55:55 -1000
Subject: [PATCH] workqueue: Don't callcpumask_test_cpu()with -1 CPU in
wq_update_node_max_active()For
wq_update_node_max_active(), @off_cpu of -1 indicates that no CPU is going down. The function was incorrectly callingcpumask_test_cpu()with -1 CPU leading to oopses like the following on some archs:Unable to handle kernel paging request at virtual address ffff0002100296e0 .. pc : wq_update_node_max_active+0x50/0x1fc lr : wq_update_node_max_active+0x1f0/0x1fc ... Call trace: wq_update_node_max_active+0x50/0x1fc apply_wqattrs_commit+0xf0/0x114 apply_workqueue_attrs_locked+0x58/0xa0 alloc_workqueue+0x5ac/0x774 workqueue_init_early+0x460/0x540 start_kernel+0x258/0x684 __primary_switched+0xb8/0xc0 Code: 9100a273 35000d01 53067f00 d0016dc1 (f8607a60) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Attempted to kill the idle task! ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
Quoted 8 lines
Fix it.
Signed-off-by Tejun Heo <[email protected]>Reported-by Marek Szyprowski <[email protected]>Reported-by Nathan Chancellor <[email protected]>Tested-by Nathan Chancellor <[email protected]>Fixes 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")
Quoted 13 lines
---
Applied to wq/for-6.9.Thanks.
kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/workqueue.c +0
-0
kernel/workqueue.c static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu) · line 1516
Quoted 8 lines
kernel/workqueue.c +1
-1@@ -1510,7 +1510,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); -if (!cpumask_test_cpu (off_cpu, effective)) +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ; total_cpus =cpumask_weight_and (effective, cpu_online_mask);
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Mark Brown
Tested-byRe: [PATCH v4 09/10] workqueue: Implement system-wide nr_active enforcement for unbound workqueues
In reply to Tejun HeoQuoted 9 lines · Tejun Heo
Ah, I see. How about the following?
Thanks.
kernel/workqueue.c +0
-0
kernel/workqueue.c static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu) · line 1514
Quoted 6 lines
kernel/workqueue.c +1
-1@@ -1510,7 +1510,7 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); -if (!cpumask_test_cpu (off_cpu, effective)) +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ;
This commit was also breaking at91sam9g20ek (and still is in today's -next), the fix here gets that booting again:
Tejun Heo
maintainerPATCH 1/1 workqueue: Avoid premature init of wq->node_nr_active[].max
In reply to Tejun HeoFrom 28596c7850a72ee82dc33758f81b71b4e8954c73 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 30 Jan 2024 19:06:43 -1000
System workqueues are allocated early during boot from workqueue_init_early(). While allocating unbound workqueues, wq_update_node_max_active() is invoked from apply_workqueue_attrs() and accesses NUMA topology to initialize wq->node_nr_active[].max.
However, topology information may not be set up at this point. wq_update_node_max_active() is explicitly invoked from workqueue_init_topology() later when topology information is known to be available.
This doesn't seem to crash anything but it's doing useless work with dubious data. Let's skip the premature and duplicate node_max_active updates by initializing the field to WQ_DFL_MIN_ACTIVE on allocation and making wq_update_node_max_active() noop until workqueue_init_topology().
---
Hello,
This doesn't seem to be causing immediate problems but let's nip it in the bud. Applied to wq/for-6.9.
Thanks.
kernel/workqueue.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/workqueue.c +8 -0
@@ -386,6 +386,8 @@ static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { [WQ_AFFN_SYSTEM] = "system", }; +static bool wq_topo_initialized__read_mostly = false; +/* * Per-cpu work items which run for longer than the following threshold are * automatically considered CPU intensive and excluded from concurrency
@@ -1510,6 +1512,9 @@ static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)lockdep_assert_held (&wq->mutex); +if (!wq_topo_initialized) +return ; +if (off_cpu >=0 && !cpumask_test_cpu (off_cpu, effective)) off_cpu =-1 ;
@@ -4356,6 +4361,7 @@ static void free_node_nr_active(struct wq_node_nr_active **nna_ar)static void init_node_nr_active (struct wq_node_nr_active *nna) { + nna->max =WQ_DFL_MIN_ACTIVE ;atomic_set (&nna->nr,0 );raw_spin_lock_init (&nna->lock);INIT_LIST_HEAD (&nna->pending_pwqs);
@@ -7400,6 +7406,8 @@ void __init workqueue_init_topology(void)init_pod_type (&wq_pod_types[WQ_AFFN_CACHE ], cpus_share_cache);init_pod_type (&wq_pod_types[WQ_AFFN_NUMA ], cpus_share_numa); + wq_topo_initialized = true; +mutex_lock (&wq_pool_mutex);/*
Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 10/10 tools/workqueue/wq_dump.py: Add node_nr/max_active dump
In reply to Tejun HeoPrint out per-node nr/max_active numbers to improve visibility into node_nr_active operations.
--- tools/workqueue/wq_dump.py | 41 +++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-)
tools/workqueue/wq_dump.py +40 -1
@@ -50,6 +50,7 @@ import drgn from drgn.helpers.linux.list import list_for_each_entry,list_empty from drgn.helpers.linux.percpu import per_cpu_ptr from drgn.helpers.linux.cpumask import for_each_cpu,for_each_possible_cpu +from drgn.helpers.linux.nodemask import for_each_node from drgn.helpers.linux.idr import idr_for_each import argparse
@@ -107,7 +108,6 @@ WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA'] WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM'] WQ_NAME_LEN = prog['WQ_NAME_LEN'].value_() - cpumask_str_len = len(cpumask_str(wq_unbound_cpumask)) print('Affinity Scopes')
@@ -205,3 +205,42 @@ print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ {"unbound_cpus":{ucpus_len}} pid {" print(f' {wq.rescuer.task.pid.value_():6}', end='') print(f' {cpumask_str(wq.rescuer.task.cpus_ptr):{rcpus_len}}', end='') print('') + +print('') +print('Unbound workqueue -> node_nr/max_active') +print('=======================================') + +if 'node_to_cpumask_map' in prog: + __cpu_online_mask = prog['__cpu_online_mask'] + node_to_cpumask_map = prog['node_to_cpumask_map'] + nr_node_ids = prog['nr_node_ids'].value_() + + print(f'online_cpus={cpumask_str(__cpu_online_mask.address_of_())}') + for node in for_each_node(): + print(f'NODE[{node:02}]={cpumask_str(node_to_cpumask_map[node])}') + print('') + + print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ min max', end='') + first = True + for node in for_each_node(): + if first: + print(f' NODE {node}', end='') + first = False + else: + print(f' {node:7}', end='') + print(f' {"dfl":>7} ]') + print('') + + for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'): + if not (wq.flags & WQ_UNBOUND): + continue + + print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} ', end='') + print(f'{wq.min_active.value_():3} {wq.max_active.value_():3}', end='') + for node in for_each_node(): + nna = wq.node_nr_active[node] + print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}', end='') + nna = wq.node_nr_active[nr_node_ids] + print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}') +else: + printf(f'node_to_cpumask_map not present, is NUMA enabled?')
Signature · 1 line
2.43.0
Hello, Tejun
Quoted 13 lines · Tejun Heo
This patchset includes the following patches:
0001-workqueue-Move-pwq-max_active-to-wq-max_active.patch 0002-workqueue-Factor-out-pwq_is_empty.patch 0003-workqueue-Replace-pwq_activate_inactive_work-with-__.patch 0004-workqueue-Move-nr_active-handling-into-helpers.patch 0005-workqueue-Make-wq_adjust_max_active-round-robin-pwqs.patch 0006-workqueue-RCU-protect-wq-dfl_pwq-and-implement-acces.patch 0007-workqueue-Move-pwq_dec_nr_in_flight-to-the-end-of-wo.patch 0008-workqueue-Introduce-struct-wq_node_nr_active.patch 0009-workqueue-Implement-system-wide-nr_active-enforcemen.patch 0010-tools-workqueue-wq_dump.py-Add-node_nr-max_active-du.patch
I just left a small piece of comments on patch 8 and patch 9. After they are resolved, for patch 1-9:
Thanks
Lai
Tejun Heo
maintainerQuoted 18 lines · Lai Jiangshan
Hello, Tejun
Quoted 13 lines · Tejun Heo
This patchset includes the following patches:
0001-workqueue-Move-pwq-max_active-to-wq-max_active.patch 0002-workqueue-Factor-out-pwq_is_empty.patch 0003-workqueue-Replace-pwq_activate_inactive_work-with-__.patch 0004-workqueue-Move-nr_active-handling-into-helpers.patch 0005-workqueue-Make-wq_adjust_max_active-round-robin-pwqs.patch 0006-workqueue-RCU-protect-wq-dfl_pwq-and-implement-acces.patch 0007-workqueue-Move-pwq_dec_nr_in_flight-to-the-end-of-wo.patch 0008-workqueue-Introduce-struct-wq_node_nr_active.patch 0009-workqueue-Implement-system-wide-nr_active-enforcemen.patch 0010-tools-workqueue-wq_dump.py-Add-node_nr-max_active-du.patch
Quoted 4 lines
I just left a small piece of comments on patch 8 and patch 9. After they are resolved, for patch 1-9:
Reviewed-by Lai Jiangshan <[email protected]>
Updated 8 and 9 and applied the series to wq/for-6.9. If there are further issues, let's address with followup patches.
Thanks a lot for all the reviews. Hopefully, this should address the max_active issue for the long term.
Signature · 1 line
tejun