Skip to contentSkip to conversation

Refactor the workqueue allocations

RFC PATCH In review Breno Leitao 2 participants 9 messages Jul 14 – Jul 29 Workqueue

lore
Share & tools
lore
#0 lore raw

cover Refactor the workqueue allocations

POC for Tejun's idea to unify the per-cpu and unbound workqueues, as suggested in http://lkml.kernel.org/r/[email protected]

This series only unifies the allocation part (alloc_pwq()). It makes alloc_pwq() a single allocator that returns a pwq backed by either a static per-cpu pool (get_percpu_pool) or an unbound pool (get_unbound_pool), reusing the existing static per-cpu pools. The install/link path is unchanged and all patches are behavior-neutral. This sets things up to unify the install side later.

Is this the right direction?

Questions to follow up:

1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well, so the unification can happen later (apply_workqueue_attrs_locked())?

2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node accounting (wq_node_nr_active) and lose percpu's per-cpu max_active (pwq->nr_active).

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over unified internals (single install path, per-cpu accounting special-cased) with WQ_UNBOUND staying for now -- or something more radical (the flags gone entirely, percpu purely an affinity value)?

Tests:

1) I've tested this on x86 and arm64, with regular tests
2) I've hacked up a workqueue test suite, which has a bunch of tests, and this is what I am using to test these changes:
https://github.com/leitao/wqtest

Thanks,
-breno

---
Breno Leitao (3):
workqueue: introduce alloc_pwq()
workqueue: allocate percpu pwqs through alloc_pwq()
workqueue: factor out alloc_and_link_percpu_pwqs()

 kernel/workqueue.c | 96 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 60 insertions(+), 36 deletions(-)

---
base-commit: b9810cd75b9fb56a3425d391cba3f608502bd474
change-id: 20260709-tejun1-c2aaf36f54a4

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
#1 lore raw

RFC PATCH 1/3 workqueue: introduce alloc_pwq()

Factor the static per-cpu pool lookup out of alloc_and_link_pwqs() into get_percpu_pool(), and add alloc_pwq() -- a common pwq allocator that picks the backing pool by workqueue type: a percpu workqueue uses the static per-cpu pool for @cpu (get_percpu_pool()), an unbound workqueue uses a pool matching @attrs (get_unbound_pool()). The paired put_pwq_pool() releases only unbound pools; static per-cpu pools are permanent.

alloc_pwq() replaces alloc_unbound_pwq() and gives the workqueue core a single entry point that can produce a pwq pointing at either kind of pool -- the building block for unifying percpu and unbound workqueues.

No functional change: the current alloc_pwq() callers only operate on unbound workqueues, so its percpu branch is not exercised yet.

---
 kernel/workqueue.c | 61 +++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 19 deletions(-)
kernel/workqueue.c +42 -19
index 94f37ea762365..c32e173af2335 100644
@@ -5354,8 +5354,38 @@ static void link_pwq(struct pool_workqueue *pwq)
 	list_add_tail_rcu(&pwq->pwqs_node, &wq->pwqs);
 }
 
-/* obtain a pool matching @attr and create a pwq associating the pool and @wq */
-static struct pool_workqueue *alloc_unbound_pwq(struct workqueue_struct *wq,
+/* Return the static per-cpu worker_pool that backs @wq on @cpu. */
+static struct worker_pool *get_percpu_pool(struct workqueue_struct *wq, int cpu)
+{
+	struct worker_pool __percpu *pools;
+	bool highpri;
+
+	WARN_ON(wq->flags & WQ_UNBOUND);
+
+	highpri = wq->flags & WQ_HIGHPRI;
+	if (wq->flags & WQ_BH)
+		pools = bh_worker_pools;
+	else
+		pools = cpu_worker_pools;
+
+	return &per_cpu_ptr(pools, cpu)[highpri];
+}
+
+/* release the pool obtained for @wq's pwq; only unbound pools are refcounted */
+static void put_pwq_pool(struct workqueue_struct *wq, struct worker_pool *pool)
+{
+	if (!(wq->flags & WQ_UNBOUND))
+		return;
+
+	put_unbound_pool(pool);
+}
+
+/*
+ * Obtain the pool backing @wq on @cpu and create a pwq associating it with @wq.
+ * A percpu @wq uses the static per-cpu pool for @cpu; an unbound @wq uses a
+ * pool matching @attrs.
+ */
+static struct pool_workqueue *alloc_pwq(struct workqueue_struct *wq, int cpu,
 					const struct workqueue_attrs *attrs)
 {
 	struct worker_pool *pool;
@@ -5363,13 +5393,16 @@ static struct pool_workqueue *alloc_unbound_pwq(struct workqueue_struct *wq,
 
 	lockdep_assert_held(&wq_pool_mutex);
 
-	pool = get_unbound_pool(attrs);
+	if (wq->flags & WQ_UNBOUND)
+		pool = get_unbound_pool(attrs);
+	else
+		pool = get_percpu_pool(wq, cpu);
 	if (!pool)
 		return NULL;
 
 	pwq = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL, pool->node);
 	if (!pwq) {
-		put_unbound_pool(pool);
+		put_pwq_pool(wq, pool);
 		return NULL;
 	}
 
@@ -5478,7 +5511,7 @@ apply_wqattrs_prepare(struct workqueue_struct *wq,
 	copy_workqueue_attrs(new_attrs, attrs);
 	wqattrs_actualize_cpumask(new_attrs, unbound_cpumask);
 	cpumask_copy(new_attrs->__pod_cpumask, new_attrs->cpumask);
-	ctx->dfl_pwq = alloc_unbound_pwq(wq, new_attrs);
+	ctx->dfl_pwq = alloc_pwq(wq, -1, new_attrs);
 	if (!ctx->dfl_pwq)
 		goto out_free;
 
@@ -5488,7 +5521,7 @@ apply_wqattrs_prepare(struct workqueue_struct *wq,
 			ctx->pwq_tbl[cpu] = ctx->dfl_pwq;
 		} else {
 			wq_calc_pod_cpumask(new_attrs, cpu);
-			ctx->pwq_tbl[cpu] = alloc_unbound_pwq(wq, new_attrs);
+			ctx->pwq_tbl[cpu] = alloc_pwq(wq, cpu, new_attrs);
 			if (!ctx->pwq_tbl[cpu])
 				goto out_free;
 		}
@@ -5632,7 +5665,7 @@ static void unbound_wq_update_pwq(struct workqueue_struct *wq, int cpu)
 		return;
 
 	/* create a new pwq */
-	pwq = alloc_unbound_pwq(wq, target_attrs);
+	pwq = alloc_pwq(wq, cpu, target_attrs);
 	if (!pwq) {
 		pr_warn("workqueue: allocation failed while updating CPU pod affinity of \"%s\"\n",
 			wq->name);
@@ -5668,19 +5701,9 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 		goto enomem;
 
 	if (!(wq->flags & WQ_UNBOUND)) {
-		struct worker_pool __percpu *pools;
-
-		if (wq->flags & WQ_BH)
-			pools = bh_worker_pools;
-		else
-			pools = cpu_worker_pools;
-
 		for_each_possible_cpu(cpu) {
-			struct pool_workqueue **pwq_p;
-			struct worker_pool *pool;
-
-			pool = &(per_cpu_ptr(pools, cpu)[highpri]);
-			pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+			struct worker_pool *pool = get_percpu_pool(wq, cpu);
 
 			*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
 						       pool->node);
Signature · 1 line
2.53.0-Meta
#2 lore raw

RFC PATCH 2/3 workqueue: allocate percpu pwqs through alloc_pwq()

In reply to Breno Leitao

Route the per-cpu pwq allocation loop in alloc_and_link_pwqs() through the common alloc_pwq() instead of open-coding get_percpu_pool() + pwq allocation + init_pwq(). alloc_pwq() is now the single entry point for both percpu and unbound pwq allocation, and its percpu branch is exercised for every percpu workqueue created.

No functional change: alloc_pwq(wq, cpu, NULL) on a percpu workqueue selects the static per-cpu pool for @cpu, exactly as the open-coded path did.

---
 kernel/workqueue.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/workqueue.c +1 -5
index c32e173af2335..50baf4fe5fcf5 100644
@@ -5703,15 +5703,11 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 	if (!(wq->flags & WQ_UNBOUND)) {
 		for_each_possible_cpu(cpu) {
 			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
-			struct worker_pool *pool = get_percpu_pool(wq, cpu);
 
-			*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
-						       pool->node);
+			*pwq_p = alloc_pwq(wq, cpu, NULL);
 			if (!*pwq_p)
 				goto enomem;
 
-			init_pwq(*pwq_p, wq, pool);
-
 			mutex_lock(&wq->mutex);
 			link_pwq(*pwq_p);
 			mutex_unlock(&wq->mutex);
Signature · 1 line
2.53.0-Meta
#3 lore raw

RFC PATCH 3/3 workqueue: factor out alloc_and_link_percpu_pwqs()

In reply to Breno Leitao

Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a helper. The inner allocation-failure path now returns -ENOMEM and the caller jumps to the existing enomem cleanup, equivalent to the previous goto.

Now that the per-cpu branch returns a value through the helper, the per-cpu, ordered and unbound cases can share a single error-handling tail. Turn the separate per-cpu block and the ordered/unbound if/else into one if/else-if/else chain, dropping the early return and the duplicated goto.

No functional change.

---
 kernel/workqueue.c | 35 ++++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)
kernel/workqueue.c +20 -15
index 50baf4fe5fcf5..411e4bd105576 100644
@@ -5689,6 +5689,24 @@ static void unbound_wq_update_pwq(struct workqueue_struct *wq, int cpu)
 	put_pwq_unlocked(old_pwq);
 }
 
+static int alloc_and_link_percpu_pwqs(struct workqueue_struct *wq)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+
+		*pwq_p = alloc_pwq(wq, cpu, NULL);
+		if (!*pwq_p)
+			return -ENOMEM;
+
+		mutex_lock(&wq->mutex);
+		link_pwq(*pwq_p);
+		mutex_unlock(&wq->mutex);
+	}
+	return 0;
+}
+
 static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 {
 	bool highpri = wq->flags & WQ_HIGHPRI;
@@ -5701,21 +5719,8 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 		goto enomem;
 
 	if (!(wq->flags & WQ_UNBOUND)) {
-		for_each_possible_cpu(cpu) {
-			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
-
-			*pwq_p = alloc_pwq(wq, cpu, NULL);
-			if (!*pwq_p)
-				goto enomem;
-
-			mutex_lock(&wq->mutex);
-			link_pwq(*pwq_p);
-			mutex_unlock(&wq->mutex);
-		}
-		return 0;
-	}
-
-	if (wq->flags & __WQ_ORDERED) {
+		ret = alloc_and_link_percpu_pwqs(wq);
+	} else if (wq->flags & __WQ_ORDERED) {
 		struct pool_workqueue *dfl_pwq;
 
 		ret = apply_workqueue_attrs_locked(wq, ordered_wq_attrs[highpri]);
Signature · 1 line
2.53.0-Meta
maintainer
#4 lore raw
In reply to Breno Leitao

Hello,

On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
...

Quoted 3 lines

1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well, so the unification can happen later (apply_workqueue_attrs_locked())?

I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the shape more be like unbound workqueue subsuming percpu workqueue?

Quoted 4 lines

2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node accounting (wq_node_nr_active) and lose percpu's per-cpu max_active (pwq->nr_active).

I think it probably would be better to introduce a separate affinity scope than modifying WQ_AFFN_CPU. Something which indicates that concurrency management is in effect and max_active is per-cpu.

Quoted 4 lines

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over unified internals (single install path, per-cpu accounting special-cased) with WQ_UNBOUND staying for now -- or something more radical (the flags gone entirely, percpu purely an affinity value)?

Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense to me.

Thanks.

Signature · 1 line
tejun
maintainer
#5 lore raw

On Thu, Jul 16, 2026 at 09:22:50AM -1000, Tejun Heo wrote:
...

Quoted 7 lines
Quoted 4 lines

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over unified internals (single install path, per-cpu accounting special-cased) with WQ_UNBOUND staying for now -- or something more radical (the flags gone entirely, percpu purely an affinity value)?

Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense to me.

Oh, and please note that if a workqueue specifies WQ_PERCPU, it should be confined to strict cpu affinity mode.

Thanks.

Signature · 1 line
tejun
#6 lore raw
In reply to Tejun Heo
Quoted 20 lines · Tejun Heo

Hello,

On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
...

Quoted 3 lines

1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well, so the unification can happen later (apply_workqueue_attrs_locked())?

I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the shape more be like unbound workqueue subsuming percpu workqueue?

Quoted 4 lines

2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node accounting (wq_node_nr_active) and lose percpu's per-cpu max_active (pwq->nr_active).

I think it probably would be better to introduce a separate affinity scope than modifying WQ_AFFN_CPU. Something which indicates that concurrency management is in effect and max_active is per-cpu.

Quoted 7 lines
Quoted 4 lines

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over unified internals (single install path, per-cpu accounting special-cased) with WQ_UNBOUND staying for now -- or something more radical (the flags gone entirely, percpu purely an affinity value)?

Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense to me.

Oh, now I see what you mean, I was heading the wrong way. Thanks for the clarification, that makes total sense.

So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than reusing WQ_AFFN_CPU. Something like:

    enum wq_affn_scope {
            ...

+ WQ_AFFN_PERCPU, /* one pod per CPU, backed by the per-cpu pool */

and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can point a pwq at a per-cpu pool, so one mechanism serves both. Then move all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate WQ_PERCPU ?

I have this working as a prototype: WQ_PERCPU selects the scope and forces strict affinity, and it boots with every percpu wq created through the new path.

A few things I'd like your read on:

1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu accounting falls out of the existing !WQ_UNBOUND checks. do you have any preference here, or should percpu become purely an affn_scope value with accounting decoupled from the flag?

2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq context) for now?

3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions (unbound_attrs allocation, and the CPU-hotplug fixups in workqueue_online_cpu()/workqueue_offline_cpu() that gate on unbound_attrs) that now have to learn about the percpu scope.

Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would you rather percpu keep a lighter install path (closer to the current WQ_PERCPU direct path), if that's feasible?

Thanks for the guidance,
--breno

maintainer
#7 lore raw

Hello, Breno.

Sorry about the delay.

Quoted 5 lines · Breno Leitao

So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than reusing WQ_AFFN_CPU. Something like:

    enum wq_affn_scope {
            ...
Quoted 7 lines

+ WQ_AFFN_PERCPU, /* one pod per CPU, backed by the per-cpu pool */

and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can point a pwq at a per-cpu pool, so one mechanism serves both. Then move all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate WQ_PERCPU ?

I don't think we'd deprecate WQ_PERCPU. It'd remain the way to specify on creation that the workqueue has to be WQ_AFFN_PERCPU.

Also, maybe WQ_AFFN_PERCPU can be more descriptive - WQ_AFFN_CPU_CM for concurrency-managed CPU scope? Or maybe this shouldn't be packaged into WQ_AFFN but rather become its own mode field.

Quoted 11 lines

I have this working as a prototype: WQ_PERCPU selects the scope and forces strict affinity, and it boots with every percpu wq created through the new path.

A few things I'd like your read on:

1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu accounting falls out of the existing !WQ_UNBOUND checks. do you have any preference here, or should percpu become purely an affn_scope value with accounting decoupled from the flag?

For concurrency management to work, it would need separate accounting (of max_active, right?). WQ_PERCPU would indicate that the wq must stay per-cpu for correctness, and we'd also want to allow concurrency management to workqueues which want to be percpu for performance reasons but can be switched into other affinity scopes for isolation, so it should move together with whether the backend needs concurrency management or not instead of WQ_PERCPU expressed at creation time.

It kinda sucks that max_active's meaning is different across the boundary tho. Maybe percpu max_active should be separate into its own field, idk.

Quoted 2 lines

2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq context) for now?

We can't switch workqueues on / off WQ_BH, but it'd also look a bit silly if this becomes its own path. Code-shape-wise, I suspect it'd be cleaner if this also becomes one of the pwq backends like the other two cases but that's just a gut feel.

Quoted 8 lines

3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions (unbound_attrs allocation, and the CPU-hotplug fixups in workqueue_online_cpu()/workqueue_offline_cpu() that gate on unbound_attrs) that now have to learn about the percpu scope.

Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would you rather percpu keep a lighter install path (closer to the current WQ_PERCPU direct path), if that's feasible?

Again, without thinking too deeploy about it, I think the code would look better if we unify everything we can.

Thanks.

Signature · 1 line
tejun
#8 lore raw

Hello Tejun,

Quoted 3 lines · Tejun Heo

Again, without thinking too deeploy about it, I think the code would look better if we unify everything we can.

I believe I have what I need to proceed. I'm planning a two-phase approach:

First, refactor the code to improve modularity where needed keeping as-is behavior, then introduce the substantive changes once the initial refactoring is done.

The goal is to make the code more managable and easy to digest.

Please let me know if you prefer otherwise.
--breno