Skip to contentSkip to conversation

blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

PATCH Superseded Aaron Tomlin 4 participants 7 messages Oct 16 – Oct 21

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu()

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

---
 block/blk-mq.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
block/blk-mq.c +17 -2
index 025a799f35b9..1a113679f15b 100644
@@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;
+
+		guard(rcu)();
+
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+	return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2254,7 +2269,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 
 	/* Switch to unbound if no allowable CPUs in this hctx */
 	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 
 	if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2296,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 		 */
 		hctx->next_cpu = next_cpu;
 		hctx->next_cpu_batch = 1;
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 	}
 
 	hctx->next_cpu = next_cpu;
Signature · 1 line
2.55.0
#1 lore raw
Quoted 72 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu()

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

Quoted 6 lines

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

Looks reasonable and related to my patch as well

https://lore.kernel.org/all/[email protected]

and I was able to create a blktests for it:

https://lore.kernel.org/all/[email protected]

Would it possible to extend those tests or add a new one so we get this also tested? And these two patches then should/could merge.

#2 lore raw
Quoted 82 lines · Daniel Wagner
Quoted 72 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu()

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

Quoted 6 lines

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

Looks reasonable and related to my patch as well

Quoted 8 lines

https://lore.kernel.org/all/[email protected]

and I was able to create a blktests for it:

https://lore.kernel.org/all/[email protected]

Would it possible to extend those tests or add a new one so we get this also tested? And these two patches then should/could merge.

Hi Daniel,

Thank you for the feedback.

Having test coverage in blktests makes complete sense. Between extending your NVMe test and adding a new one, I think creating a new, generic test under tests/block/ is the cleaner path.

While your nvme/070 test targets NVMe fabrics connect and passthrough request allocation, this fix is in core blk-mq (blk_mq_hctx_next_cpu) and affects deferred workqueue execution across all block devices. Using a new test based on null_blk would allow us to:

1. Target infrastructure for generic blk-mq workqueue placement testing

2. Easily configure a shallow queue depth or inject tag exhaustion/busy conditions to reliably trigger asynchronous queue dispatch (&hctx->run_work)

3. Trace workqueue:workqueue_execute_start to assert that no kblockd worker executes on the isolated core

I'll put together a test case for tests/block/.

Kind regards,

Signature · 1 line
Aaron Tomlin
#3 lore raw
Quoted 7 lines

While your nvme/070 test targets NVMe fabrics connect and passthrough request allocation, this fix is in core blk-mq (blk_mq_hctx_next_cpu) and affects deferred workqueue execution across all block devices. Using a new test based on null_blk would allow us to:

1. Target infrastructure for generic blk-mq workqueue placement testing

Quoted 8 lines

2. Easily configure a shallow queue depth or inject tag exhaustion/busy conditions to reliably trigger asynchronous queue dispatch (&hctx->run_work)

3. Trace workqueue:workqueue_execute_start to assert that no kblockd worker executes on the isolated core

I'll put together a test case for tests/block/.

Sounds good to me! Thanks.

maintainer
#4 lore raw
In reply to Aaron Tomlin
Quoted 12 lines · Aaron Tomlin
block/blk-mq.c +4 -0
index 025a799f35b9..1a113679f15b 100644
@@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;

block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2257

Quoted 10 lines
block/blk-mq.c +10 -0
+
+		guard(rcu)();
+
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+	return WORK_CPU_UNBOUND;
+}

Not opposed to this, but let's please think about making the common case the fast path. CPU isolation is not the common path. That might mean annotating the hctx itself.

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 19 lines · Jens Axboe
Quoted 18 lines · Aaron Tomlin
block/blk-mq.c +10 -0
index 025a799f35b9..1a113679f15b 100644
@@ -2241,6 +2241,21 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;
+
+		guard(rcu)();
+
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
Quoted 9 lines

block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2257

Quoted 4 lines
block/blk-mq.c +4 -0
+			return cpu;
+	}
+	return WORK_CPU_UNBOUND;
+}

Not opposed to this, but let's please think about making the common case the fast path. CPU isolation is not the common path. That might mean annotating the hctx itself.

Hi Jens,

Agreed on keeping the fast path penalty-free.

For multi-queue devices, blk_mq_hctx_empty_cpumask(hctx) already serves as that annotation (via hctx->next_cpu >= nr_cpu_ids) since isolated CPUs are stripped during mapping, so normal multi-queue devices never hit that path.

The main concern is single-queue devices (nr_hw_queues == 1), where the hctx is shared between housekeeping and isolated cores. To avoid calling cpu_is_isolated() on every deferred dispatch in the common case, we can split the check and gate it behind a static key:
static_branch_unlikely(&housekeeping_overridden)

I will prepare a v4.

Kind regards,

Signature · 1 line
Aaron Tomlin
#6 lore raw
In reply to Aaron Tomlin
Quoted 33 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu()

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

Quoted 8 lines

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")

The current code follows behavior of schedule_work() and other related wq APIs. If you think it is wrong, care to fix schedule_work() too, otherwise it confuses people, IMHO.

Thanks,
Ming