blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty
Subsystems (2)
PATCH v4 1/1 blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty
Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution.
However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter on nohz_full cores, thereby defeating the original objective of commit a46c27026da1.
Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the unbound fallback paths in blk_mq_hctx_next_cpu().
Additionally, for single-queue devices, gate the check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that common configurations without CPU isolation incur zero overhead on the fast path.
This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.
2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU, WORK_CPU_UNBOUND is preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.
4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.
---
Changes since v3:
- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)
- Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)
- Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur
- Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions
- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings
- Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND
- Link to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores
- Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path.
- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-)
block/blk-mq.c +28 -3
@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +guard (rcu)(); + +if (cpu_is_isolated (raw_smp_processor_id ())) { +int cpu; + +guard (preempt)(); + cpu =cpumask_any_and_distribute (housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu_online_mask); +if (cpu < nr_cpu_ids) +return cpu; + } + +return WORK_CPU_UNBOUND ; +} +/* * It'd be great if the workqueue API had a way to pass * in a mask and had some smarts for more clever placement.
@@ -2252,9 +2269,17 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)bool tried = false;int next_cpu = hctx->next_cpu; -/* Switch to unbound if no allowable CPUs in this hctx */ -if (hctx->queue->nr_hw_queues ==1 ||blk_mq_hctx_empty_cpumask (hctx)) +if (hctx->queue->nr_hw_queues ==1 ) { +#ifdef CONFIG_CPU_ISOLATION +if (static_branch_unlikely (&housekeeping_overridden)) +return blk_mq_hctx_unbound_cpu (); +#endif return WORK_CPU_UNBOUND ; + } + +/* Switch to unbound if no allowable CPUs in this hctx */ +if (blk_mq_hctx_empty_cpumask (hctx)) +return blk_mq_hctx_unbound_cpu ();if (--hctx->next_cpu_batch <=0 ) { select_cpu:
@@ -2281,7 +2306,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)*/ hctx->next_cpu = next_cpu; hctx->next_cpu_batch =1 ; -return WORK_CPU_UNBOUND ; +return blk_mq_hctx_unbound_cpu (); } hctx->next_cpu = next_cpu;
Signature · 1 line
2.55.0
(Adding Marco)
Quoted 125 lines · Aaron Tomlin
Commit
a46c27026da1("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs fromhctx->cpumaskto prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e.blk_mq_hctx_empty_cpumask(hctx)evaluates to true),blk_mq_hctx_next_cpu()returnsWORK_CPU_UNBOUNDwith the intention of offloading execution.However,
kblockd_workqueueis allocated as a per-CPU workqueue (WQ_PERCPU). WhenWORK_CPU_UNBOUNDis queued on a per-CPU workqueue, the workqueue core resolves the target CPU viaraw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returningBLK_STS_RESOURCE, or plugging),&hctx->run_workis queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an activeSCHED_FIFOworkload will starve thisSCHED_NORMALworker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter onnohz_fullcores, thereby defeating the original objective of commita46c27026da1.Introduce
blk_mq_hctx_unbound_cpu()to select an online housekeeping CPU fromhousekeeping_cpumask(HK_TYPE_DOMAIN)viacpumask_any_and_distribute()when executing on an isolated CPU, and otherwise fall back toWORK_CPU_UNBOUND. Use this helper across the unbound fallback paths inblk_mq_hctx_next_cpu().Additionally, for single-queue devices, gate the check behind the static key
housekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure that common configurations without CPU isolation incur zero overhead on the fast path.This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU,
WORK_CPU_UNBOUNDis preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.
Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")Cc Ming Lei <[email protected]>Cc Jens Axboe <[email protected]>Signed-off-by Aaron Tomlin <[email protected]>---
Changes since v3:- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)
- Gate the single-queue (
nr_hw_queues== 1) check behind the static keyhousekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)- Wrap the evaluation of
cpu_is_isolated()inside guard(rcu)() inblk_mq_hctx_unbound_cpu()to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur- Disable preemption via guard(preempt)() around
cpumask_any_and_distribute()to prevent a preemption warning underCONFIG_PREEMPT_RCUand avoid data races when accessing per-CPU distribution state- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Query
housekeeping_cpumask(HK_TYPE_DOMAIN)instead ofHK_TYPE_TIMERto align withcpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings
- Validate that
cpumask_any_and_distribute()returns a valid cpu ID before returning, falling back cleanly toWORK_CPU_UNBOUND- Link to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Introduce the
blk_mq_hctx_unbound_cpu()helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preservingWORK_CPU_UNBOUNDand local cache locality for housekeeping cores- Apply
blk_mq_hctx_unbound_cpu()across all unbound fallback paths inblk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues== 1), empty hardware queue cpumasks, and the offline CPU fallback path.- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion,
BLK_STS_RESOURCEbackpressure, and plugging) and theSCHED_FIFOworker starvation hazard on isolated cores- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-)block/blk-mq.c +6
-0@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +guard (rcu)(); + +if (cpu_is_isolated (raw_smp_processor_id ())) { +int cpu;
block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259
Quoted 10 lines
block/blk-mq.c +10
-0+ +guard (preempt)(); + cpu =cpumask_any_and_distribute (housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu_online_mask); +if (cpu < nr_cpu_ids) +return cpu; + } + +return WORK_CPU_UNBOUND ; +}
Just for future record, we are working on a WQ_PREFER_PERCPU workqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.
Thanks.
Signature · 2 lines
Frederic Weisbecker SUSE Labs
Quoted 144 lines · Frederic Weisbecker
(Adding Marco)
Quoted 125 lines · Aaron Tomlin
Commit
a46c27026da1("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs fromhctx->cpumaskto prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e.blk_mq_hctx_empty_cpumask(hctx)evaluates to true),blk_mq_hctx_next_cpu()returnsWORK_CPU_UNBOUNDwith the intention of offloading execution.However,
kblockd_workqueueis allocated as a per-CPU workqueue (WQ_PERCPU). WhenWORK_CPU_UNBOUNDis queued on a per-CPU workqueue, the workqueue core resolves the target CPU viaraw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returningBLK_STS_RESOURCE, or plugging),&hctx->run_workis queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an activeSCHED_FIFOworkload will starve thisSCHED_NORMALworker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter onnohz_fullcores, thereby defeating the original objective of commita46c27026da1.Introduce
blk_mq_hctx_unbound_cpu()to select an online housekeeping CPU fromhousekeeping_cpumask(HK_TYPE_DOMAIN)viacpumask_any_and_distribute()when executing on an isolated CPU, and otherwise fall back toWORK_CPU_UNBOUND. Use this helper across the unbound fallback paths inblk_mq_hctx_next_cpu().Additionally, for single-queue devices, gate the check behind the static key
housekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure that common configurations without CPU isolation incur zero overhead on the fast path.This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU,
WORK_CPU_UNBOUNDis preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.
Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")Cc Ming Lei <[email protected]>Cc Jens Axboe <[email protected]>Signed-off-by Aaron Tomlin <[email protected]>---
Changes since v3:- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)
- Gate the single-queue (
nr_hw_queues== 1) check behind the static keyhousekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)- Wrap the evaluation of
cpu_is_isolated()inside guard(rcu)() inblk_mq_hctx_unbound_cpu()to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur- Disable preemption via guard(preempt)() around
cpumask_any_and_distribute()to prevent a preemption warning underCONFIG_PREEMPT_RCUand avoid data races when accessing per-CPU distribution state- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Query
housekeeping_cpumask(HK_TYPE_DOMAIN)instead ofHK_TYPE_TIMERto align withcpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings
- Validate that
cpumask_any_and_distribute()returns a valid cpu ID before returning, falling back cleanly toWORK_CPU_UNBOUND- Link to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Introduce the
blk_mq_hctx_unbound_cpu()helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preservingWORK_CPU_UNBOUNDand local cache locality for housekeeping cores- Apply
blk_mq_hctx_unbound_cpu()across all unbound fallback paths inblk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues== 1), empty hardware queue cpumasks, and the offline CPU fallback path.- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion,
BLK_STS_RESOURCEbackpressure, and plugging) and theSCHED_FIFOworker starvation hazard on isolated cores- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-)block/blk-mq.c +6
-0@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +guard (rcu)(); + +if (cpu_is_isolated (raw_smp_processor_id ())) { +int cpu;block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259
Quoted 10 lines
block/blk-mq.c +10
-0+ +guard (preempt)(); + cpu =cpumask_any_and_distribute (housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu_online_mask); +if (cpu < nr_cpu_ids) +return cpu; + } + +return WORK_CPU_UNBOUND ; +}Just for future record, we are working on a
WQ_PREFER_PERCPUworkqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.
Quoted 5 lines
Thanks.
Signature · 2 lines
Frederic Weisbecker SUSE Labs
Hi Frederic,
Noted and thank you for sharing.
Sean and I observed this behaviour on a kernel with commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") applied.
We will certainly keep WQ_PREFER_PERCPU in mind as a cleaner long-term path.
Kind regards,
Signature · 1 line
Aaron Tomlin
Quoted 153 lines · Aaron Tomlin
Quoted 144 lines · Frederic Weisbecker
(Adding Marco)
Quoted 125 lines · Aaron Tomlin
Commit
a46c27026da1("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs fromhctx->cpumaskto prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e.blk_mq_hctx_empty_cpumask(hctx)evaluates to true),blk_mq_hctx_next_cpu()returnsWORK_CPU_UNBOUNDwith the intention of offloading execution.However,
kblockd_workqueueis allocated as a per-CPU workqueue (WQ_PERCPU). WhenWORK_CPU_UNBOUNDis queued on a per-CPU workqueue, the workqueue core resolves the target CPU viaraw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (e.g. upon hardware queue congestion, tag/resource exhaustion returningBLK_STS_RESOURCE, or plugging),&hctx->run_workis queued directly onto the local isolated CPU's worker pool. This causes the local per-CPU high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU. In real-time environments, an activeSCHED_FIFOworkload will starve thisSCHED_NORMALworker, causing hardware dispatch to stall, in addition to introducing unwanted latency jitter onnohz_fullcores, thereby defeating the original objective of commita46c27026da1.Introduce
blk_mq_hctx_unbound_cpu()to select an online housekeeping CPU fromhousekeeping_cpumask(HK_TYPE_DOMAIN)viacpumask_any_and_distribute()when executing on an isolated CPU, and otherwise fall back toWORK_CPU_UNBOUND. Use this helper across the unbound fallback paths inblk_mq_hctx_next_cpu().Additionally, for single-queue devices, gate the check behind the static key
housekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure that common configurations without CPU isolation incur zero overhead on the fast path.This ensures that:
1. When executing on an isolated CPU, deferred queue work is cleanly offloaded to an online housekeeping CPU across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths.2. The per-CPU high-priority kworker on the isolated CPU is not scheduled, avoiding priority inversion and latency jitter.
3. When executing on a non-isolated CPU,
WORK_CPU_UNBOUNDis preserved, maintaining optimal local execution with zero CPU bouncing while avoiding timer pinning hazards.4. On systems without CPU isolation, single-queue devices maintain the zero-overhead fast path via a static key.
Fixes a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")Cc Ming Lei <[email protected]>Cc Jens Axboe <[email protected]>Signed-off-by Aaron Tomlin <[email protected]>---
Changes since v3:- Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner)
- Gate the single-queue (
nr_hw_queues== 1) check behind the static keyhousekeeping_overriddenunderCONFIG_CPU_ISOLATIONto ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe)- Wrap the evaluation of
cpu_is_isolated()inside guard(rcu)() inblk_mq_hctx_unbound_cpu()to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur- Disable preemption via guard(preempt)() around
cpumask_any_and_distribute()to prevent a preemption warning underCONFIG_PREEMPT_RCUand avoid data races when accessing per-CPU distribution state- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Query
housekeeping_cpumask(HK_TYPE_DOMAIN)instead ofHK_TYPE_TIMERto align withcpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions- Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings
- Validate that
cpumask_any_and_distribute()returns a valid cpu ID before returning, falling back cleanly toWORK_CPU_UNBOUND- Link to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Introduce the
blk_mq_hctx_unbound_cpu()helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preservingWORK_CPU_UNBOUNDand local cache locality for housekeeping cores- Apply
blk_mq_hctx_unbound_cpu()across all unbound fallback paths inblk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues== 1), empty hardware queue cpumasks, and the offline CPU fallback path.- Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion,
BLK_STS_RESOURCEbackpressure, and plugging) and theSCHED_FIFOworker starvation hazard on isolated cores- Link to v1: https://lore.kernel.org/lkml/[email protected]/
--- block/blk-mq.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-)block/blk-mq.c +6
-0@@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu (void ) +{ +guard (rcu)(); + +if (cpu_is_isolated (raw_smp_processor_id ())) { +int cpu;block/blk-mq.c static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) · line 2259
Quoted 10 lines
block/blk-mq.c +10
-0+ +guard (preempt)(); + cpu =cpumask_any_and_distribute (housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu_online_mask); +if (cpu < nr_cpu_ids) +return cpu; + } + +return WORK_CPU_UNBOUND ; +}Just for future record, we are working on a
WQ_PREFER_PERCPUworkqueue flag that should automatically handle this sort of things. It's not ready for now so these hacks are still necessary but keep in mind this is a good candidare for future simplication.Quoted 5 lines
Thanks.
Signature · 2 lines
Frederic Weisbecker SUSE LabsHi Frederic,
Quoted 7 lines
Noted and thank you for sharing.
Sean and I observed this behaviour on a kernel with commit
a46c27026da1("blk-mq: don't schedule block kworker on isolated CPUs") applied.We will certainly keep
WQ_PREFER_PERCPUin mind as a cleaner long-term path.
Hi Jens,
Would this patch be suitable for inclusion while we await the introduction of WQ_PREFER_PERCPU?
Kind regards,
Signature · 1 line
Aaron Tomlin