Skip to contentSkip to conversation

blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

PATCH In review Aaron Tomlin 2 participants 4 messages Oct 22 – Oct 23

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().

Additionally, for single-queue devices, gate the check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that common configurations without CPU isolation incur zero overhead on the fast path.

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.

4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v3:

- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)

- Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)

- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur

- Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state

- Link to v3: https://lore.kernel.org/all/[email protected]/

Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

---
 block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
block/blk-mq.c +28 -3
index 025a799f35b9..f8f862a0e525 100644
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	guard(rcu)();
+
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;
+
+		guard(preempt)();
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+
+	return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2252,9 +2269,17 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 	bool tried = false;
 	int next_cpu = hctx->next_cpu;
 
-	/* Switch to unbound if no allowable CPUs in this hctx */
-	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
+	if (hctx->queue->nr_hw_queues == 1) {
+#ifdef CONFIG_CPU_ISOLATION
+		if (static_branch_unlikely(&housekeeping_overridden))
+			return blk_mq_hctx_unbound_cpu();
+#endif
 		return WORK_CPU_UNBOUND;
+	}
+
+	/* Switch to unbound if no allowable CPUs in this hctx */
+	if (blk_mq_hctx_empty_cpumask(hctx))
+		return blk_mq_hctx_unbound_cpu();
 
 	if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2306,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 		 */
 		hctx->next_cpu = next_cpu;
 		hctx->next_cpu_batch = 1;
-		return WORK_CPU_UNBOUND;
+		return blk_mq_hctx_unbound_cpu();
 	}
 
 	hctx->next_cpu = next_cpu;
Signature · 1 line
2.55.0
#1 lore raw

(Adding Marco)

Quoted 125 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().

Additionally, for single-queue devices, gate the check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that common configurations without CPU isolation incur zero overhead on the fast path.

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.

4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v3:

- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)

- Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)

- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur

- Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state

- Link to v3: https://lore.kernel.org/all/[email protected]/

Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

---
 block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
block/blk-mq.c +6 -0
index 025a799f35b9..f8f862a0e525 100644
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	guard(rcu)();
+
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;

block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259

Quoted 10 lines
block/blk-mq.c +10 -0
+
+		guard(preempt)();
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+
+	return WORK_CPU_UNBOUND;
+}

Just for future record, we are working on a WQ_PREFER_PERCPU workqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#2 lore raw
Quoted 144 lines · Frederic Weisbecker

(Adding Marco)

Quoted 125 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().

Additionally, for single-queue devices, gate the check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that common configurations without CPU isolation incur zero overhead on the fast path.

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.

4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v3:

- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)

- Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)

- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur

- Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state

- Link to v3: https://lore.kernel.org/all/[email protected]/

Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

---
 block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
block/blk-mq.c +6 -0
index 025a799f35b9..f8f862a0e525 100644
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	guard(rcu)();
+
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;

block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259

Quoted 10 lines
block/blk-mq.c +10 -0
+
+		guard(preempt)();
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+
+	return WORK_CPU_UNBOUND;
+}

Just for future record, we are working on a WQ_PREFER_PERCPU workqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.

Quoted 5 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs

Hi Frederic,

Noted and thank you for sharing.

Sean and I observed this behaviour on a kernel with commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") applied.

We will certainly keep WQ_PREFER_PERCPU in mind as a cleaner long-term path.

Kind regards,

Signature · 1 line
Aaron Tomlin
#3 lore raw
Quoted 153 lines · Aaron Tomlin
Quoted 144 lines · Frederic Weisbecker

(Adding Marco)

Quoted 125 lines · Aaron Tomlin

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.

Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().

Additionally, for single-queue devices, gate the check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that common configurations without CPU isolation incur zero overhead on the fast path.

This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.

2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.

3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.

4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.

Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc Ming Lei <[email protected]>
Cc Jens Axboe <[email protected]>

---
Changes since v3:

- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)

- Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)

- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur

- Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state

- Link to v3: https://lore.kernel.org/all/[email protected]/

Changes since v2:

- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions

- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings

- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores

- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.

- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

---
 block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
block/blk-mq.c +6 -0
index 025a799f35b9..f8f862a0e525 100644
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }
 
+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+	guard(rcu)();
+
+	if (cpu_is_isolated(raw_smp_processor_id())) {
+		int cpu;

block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259

Quoted 10 lines
block/blk-mq.c +10 -0
+
+		guard(preempt)();
+		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
+						 cpu_online_mask);
+		if (cpu < nr_cpu_ids)
+			return cpu;
+	}
+
+	return WORK_CPU_UNBOUND;
+}

Just for future record, we are working on a WQ_PREFER_PERCPU workqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.

Quoted 5 lines

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs

Hi Frederic,

Quoted 7 lines

Noted and thank you for sharing.

Sean and I observed this behaviour on a kernel with commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") applied.

We will certainly keep WQ_PREFER_PERCPU in mind as a cleaner long-term path.

Hi Jens,

Would this patch be suitable for inclusion while we await the introduction of WQ_PREFER_PERCPU?

Kind regards,

Signature · 1 line
Aaron Tomlin