blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty
Subsystems (2)
PATCH v2 1/1 blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty
Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.
However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.
Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU via housekeeping_any_cpu(HK_TYPE_TIMER) when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().
This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.
2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing.
---
Changes since v1:
- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores
- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.
- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
block/blk-mq.c +9 -2
@@ -2241,6 +2241,13 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +if (cpu_is_isolated (raw_smp_processor_id ())) +return housekeeping_any_cpu (HK_TYPE_TIMER ); +return WORK_CPU_UNBOUND ; +} +/* * It'd be great if the workqueue API had a way to pass * in a mask and had some smarts for more clever placement.
@@ -2254,7 +2261,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)/* Switch to unbound if no allowable CPUs in this hctx */ if (hctx->queue->nr_hw_queues ==1 ||blk_mq_hctx_empty_cpumask (hctx)) -return WORK_CPU_UNBOUND ; +return blk_mq_hctx_unbound_cpu ();if (--hctx->next_cpu_batch <=0 ) { select_cpu:
@@ -2281,7 +2288,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)*/ hctx->next_cpu = next_cpu; hctx->next_cpu_batch =1 ; -return WORK_CPU_UNBOUND ; +return blk_mq_hctx_unbound_cpu (); } hctx->next_cpu = next_cpu;
Signature · 1 line
2.55.0
Quoted 106 lines · Aaron Tomlin
Commit
a46c27026da1("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs fromhctx->cpumaskto prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e.blk_mq_hctx_empty_cpumask(hctx)evaluates to true),blk_mq_hctx_next_cpu()returnsWORK_CPU_UNBOUNDwith the intention of offloading execution.However,
kblockd_workqueueis allocated as a per-CPU workqueue (WQ_PERCPU). WhenWORK_CPU_UNBOUNDis queued on a per-CPU workqueue, the workqueue core resolves the target CPU viaraw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returningBLK_STS_RESOURCE, or plugging),&hctx->run_workis queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an activeSCHED_FIFOworkload will starve thisSCHED_NORMALworker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter onnohz_fullcores, thereby defeating the original objective of commita46c27026da1.Introduce
blk_mq_hctx_unbound_cpu()to select an online housekeeping CPU viahousekeeping_any_cpu(HK_TYPE_TIMER)when executing on an isolated CPU, and otherwise fall back toWORK_CPU_UNBOUND. Use this helper across the unbound fallback paths inblk_mq_hctx_next_cpu().This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU,
WORK_CPU_UNBOUNDis preserved, maintaining optimal local execution with zero CPU bouncing.Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")Cc Ming Lei <[email protected]>Cc Jens Axboe <[email protected]>Signed-off-by Aaron Tomlin <[email protected]>---
Changes since v1:- Introduce the
blk_mq_hctx_unbound_cpu()helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preservingWORK_CPU_UNBOUNDand local cache locality for housekeeping cores- Apply
blk_mq_hctx_unbound_cpu()across all unbound fallback paths inblk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues== 1), empty hardware queue cpumasks, and the offline CPU fallback path.- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion,
BLK_STS_RESOURCEbackpressure, and plugging) and theSCHED_FIFOworker starvation hazard on isolated cores- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)block/blk-mq.c +9
-2@@ -2241,6 +2241,13 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +if (cpu_is_isolated (raw_smp_processor_id ())) +return housekeeping_any_cpu (HK_TYPE_TIMER ); +return WORK_CPU_UNBOUND ; +} +/* * It'd be great if the workqueue API had a way to pass * in a mask and had some smarts for more clever placement. @@ -2254,7 +2261,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)/* Switch to unbound if no allowable CPUs in this hctx */ if (hctx->queue->nr_hw_queues ==1 ||blk_mq_hctx_empty_cpumask (hctx)) -return WORK_CPU_UNBOUND ; +return blk_mq_hctx_unbound_cpu ();if (--hctx->next_cpu_batch <=0 ) { select_cpu:@@ -2281,7 +2288,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)*/ hctx->next_cpu = next_cpu; hctx->next_cpu_batch =1 ; -return WORK_CPU_UNBOUND ; +return blk_mq_hctx_unbound_cpu (); } hctx->next_cpu = next_cpu;Signature · 1 line
2.55.0
Apologies.
The proper approach is to query housekeeping_cpumask(HK_TYPE_TIMER) directly and select an online core via cpumask_any_and_distribute(). This avoids invoking smp_processor_id() altogether, and round-robins delayed work cleanly across available housekeeping cores. Furthermore, because CPU hotplug explicitly enforces that at least one housekeeping core must remain online on a running system, the intersection is guaranteed never to be empty:
static inline int blk_mq_hctx_unbound_cpu(void)
{
if (cpu_is_isolated(raw_smp_processor_id()))
return cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_TIMER),
cpu_online_mask);
return WORK_CPU_UNBOUND;
}Kind regards,
Signature · 1 line
Aaron Tomlin